Performance analysis of large language models on
a specialized medical examination: allergy and clinical
immunology

Betül Dumanoğlu; Pamir Çerçi; Özge Can Bostan; Ümit Şahiner

doi:10.5114/pja.2025.156158

eISSN: 2391-6052
ISSN: 2353-3854

Alergologia Polska - Polish Journal of Allergology

Current issue Archive Manuscripts accepted About the journal Supplements Special issues Editorial board Abstracting and indexing Subscription Contact Instructions for authors Publication charge Ethical standards and procedures

Editorial System

Submit your Manuscript

Editorial Policies

Sarajevo Declaration on Integrity and Visibility of Scholarly Publications

Open access

4/2025
vol. 12

Send email

Copy url:

Original paper

Performance analysis of large language models on a specialized medical examination: allergy and clinical immunology

Betül Dumanoğlu

¹

,

Pamir Çerçi

²

,

Özge Can Bostan

³

,

Ümit M. Şahiner

⁴

Department of Allergy and Immunology, Agri Training and Research Hospital, Agri, Turkey
Department of Allergy and Immunology, Eskisehir City Hospital, Eskisehir, Turkey
Department of Allergy and Immunology, Canakkale Mehmet Akif Ersoy State Hospital, Canakkale, Turkey
Department of Pediatric Allergy and Asthma, Faculty of Medicine, Hacettepe University, Ankara, Turkey

Alergologia Polska – Polish Journal of Allergology 2025; 12, 4: 271–277

DOI: https://doi.org/10.5114/pja.2025.156158

Online publish date: 2025/11/13

Article file

- Performance analysis.pdf [0.24 MB]

Get citation

PlumX metrics:

1. Abbas A, Rehman MS, Rehman SS, et al. Comparing the performance of popular large language models on the National Board of Medical Examiners Sample Questions. Cureus 2024; 16: e55991.

2. Morreel S, Verhoeven V, Mathysen D. Microsoft Bing outperforms five other generative artificial intelligence chatbots in the Antwerp University multiple choice medical license exam. PLoS Dig Health 2024; 3: e0000349.

3. Shieh A, Tran B, He G, et al. Assessing ChatGPT 4.0’s test performance and clinical diagnostic accuracy on USMLE STEP 2 CK and clinical case reports. Sci Rep 2024; 14: 9330.

4. Gilson A, Safranek CW, Huang T, et al. How does ChatGPT perform on the united states medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ 2023; 9: e45312.

5. Garabet R, Mackey BP, Cross J, Weingarten M. ChatGPT-4 performance on USMLE Step 1 questions and its implications for medical education: a comparative study across systems and disciplines. Med Sci Educ 2023; 34: 145-52.

6. Nakao T, Miki S, Nakamura Y, et al. Capability of GPT-4V(ision) in the Japanese National Medical Licensing Examination: evaluation study. JMIR Med Educ 2024; 10: e54393.

7. Bielówka M, Kufel J, Rojek M, et al. Evaluating ChatGPT-3.5 in allergology: performance in the Polish Specialist Examination. Pol J Allergol 2024; 11: 42-7.

8. Engelhart MD, Furst EJ, Krathwohl DR. Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook Cognitive Domain. Longmans, 1956.

9. Lucas HC, Upperman JS, Robinson JR. A systematic review of large language models and their implications in medical education. Med Educ 2024; 58: 1276-85.

10. Fuchs A, Trachsel T, Weiger R, et al. ChatGPT’s performance in dentistry and allergy-immunology assessments: a comparative study. Swiss Dent J 2023; 134: 1-17.

11. Klimczak M, Rymarczyk B, Gawlik R. Exploring the link between gut health, diet, and food allergy. Pol J Allergol 2024; 11: 264-70.

12. Nittner-Marszalska M, Niedoszytko M, Cichocka-Jarosz E, et al. Position of the Anaphylaxis, Insect Venom Allergy and Mastocytosis Section of the Polish Society of Allergology regarding the principles of diagnosing hypersensitivity to Hymenoptera insects venom. Pol J Allergol 2024; 11: 187-94.

13. Watari T, Takagi S, Sakaguchi K, et al. Performance comparison of ChatGPT-4 and Japanese medical residents in the general medicine in-training examination: comparison study. JMIR Med Educ 2023; 9: e52202.

Copyright: © Polish Society of Allergology This is an Open Access article distributed under the terms of the Creative Commons Attribution-Noncommercial-No Derivatives 4.0 International (CC BY-NC-SA 4.0). License (http://creativecommons.org/licenses/by-nc-sa/4.0/), allowing third parties to copy and redistribute the material in any medium or format and to remix, transform, and build upon the material, provided the original work is properly cited and states its license.

Performance analysis of large language models on a specialized medical examination: allergy and clinical immunology

Betül Dumanoğlu 1 , Pamir Çerçi 2 , Özge Can Bostan 3 , Ümit M. Şahiner 4

Betül Dumanoğlu

¹

,

Pamir Çerçi

²

,

Özge Can Bostan

³

,

Ümit M. Şahiner

⁴