The development of large language models (LLMs) has introduced new opportunities in medical education, including the generation of clinical scenarios and examination questions. With their increasing use, concerns arise regarding the quality and reliability of AI-generated educational content. Notably, 52% of academic teachers report using LLMs to create exam questions [1].
Examination questions are essential not only for assessment but also for education, requiring accuracy, clarity, appropriate difficulty, and alignment with current guidelines. The growing demand for new question sets, driven by workforce shortages, combined with the time-consuming nature of question development, has increased interest in automated solutions [2].
While many studies have assessed AI performance in answering exam questions, fewer have evaluated its ability to generate them. This task is more complex and depends on factors such as prompt quality, source material, and model characteristics. AI-generated questions may lack sufficient difficulty or contain inaccuracies, and existing dermatology studies are mostly qualitative, with limited comparison to expert-created content [3–5].
This study compared dermatology examination questions generated by ChatGPT-4 Turbo with those prepared by dermatology specialists, focusing on clarity, correctness, difficulty, and the ability to distinguish AI-generated (AIQ) from human-authored questions (HQ).
A comparative analysis was performed between examination questions generated by ChatGPT-4 Turbo and those prepared by certified dermatology specialists. A 40-item single-choice exam (USMLE-style, five options per question) was developed, including 20 AI-generated and 20 expert-derived questions. AI questions were created using a standardised English prompt based on Polish Dermatological Society guidelines and subsequently translated into Polish. Expert questions were selected from previous final dermatology exams. Both sets covered four topics: hidradenitis suppurativa [6], chronic urticaria [7], plaque psoriasis [8], and atopic dermatitis [9]. No prior review of questions was performed.
The final randomised exam was administered in Polish to 48 participants: 26 medical students, 12 residents, and 10 specialists. Participants had 40 min to complete the test and additionally rated each question for clarity, correctness, difficulty, and source (AI vs. human).
The translated version of the exam and the prompt are included as Supplement 1.
Statistical analysis included the Kruskal–Wallis test with Dunn’s post-hoc test, Wilcoxon signed-rank test, χ2 test, and Student’s t-test, depending on the variable analysed.
The final analysis included 26 students, 12 residents, and 9 specialists (mean age: 27 ±6 years). One incomplete test submitted by a specialist was excluded from the analysis. Specialists and residents achieved higher scores than students (p < 0.001), with no difference between residents and specialists. The average correct response rate among students was 59.25% (64% HQ, 54.5% AIQ), among residents 80.75% (83% HQ, 79% AIQ), and among specialists 72.25% (76.5% HQ, 68% AIQ). Overall accuracy was higher for HQ than AIQ in the whole group (p < 0.001) and among students (p < 0.01) and specialists (p = 0.014), but not residents (Figure 1).
Figure 1
Comparison of results obtained according to type of test questions (A), subgroup of respondents (B)
On average, 89.8% of AIQ and 95.3% of HQ were rated as clearly stated. The clarity index for AIQ and HQ was highest among students. Students and specialists rated HQ as more clearly formulated than AIQ (p = 0.001 and p = 0.013 respectively). On average, 88.83% of AIQ and 94.2% of HQ were rated as correctly arranged. For students, the percentage of AIQ identified as correct was 93.2% but for specialists 77.1%. Both students and specialists rated HQ as written more correctly than AIQ (p =0.012 and p =0.001, respectively) (Table 1).
Table 1
Subjective ratings for AIQ and HQ: clarity, correctness, difficulty. P-values < 0.05 are marked in bold. Small effect size is marked as *, moderate as ** and large as ***
The average difficulty was 2.03 for AIQ and 1.9 for HQ. All groups considered AIQ as more difficult than HQ.
The average AIQ detection rate was 40%, while for HQ it was 64.1%. No significant differences in identification accuracy were noted between responder groups for HQ, whereas for AIQ there were differences between students and specialists (p = 0.023, d = 0.087) and residents and specialists (p = 0.007, d = 0.097).
The mean Difficulty Index (DI) was 0.728 for HQ and 0.647 for AIQ with no significant difference between AIQ and HQ (p = 0.15). A total of 12 AIQ and 5 HQ fell within the optimal difficulty range (0.3–0.7). Seven AIQ and 14 HQ were categorised as easy (DI > 0.7), and one AIQ and one HQ as difficult (DI < 0.3) (Figure 2).
Figure 2
Discriminatory power in HQ and AIQ: Difficulty Index (A), Discrimination Index (B). The limits of the optimal ranges are marked with dashed lines
The Discrimination Index was similar for both sets (AIQ 0.32 vs. HQ 0.35; p = 0.696). Nine AIQ and 7 HQ received between 0 and 0.3 rating/score and 9 AIQ and 9 HQ had good discrimination (between 0.3 and 0.6). Two questions each received rating/score above 0.6, which means the discrimination power of these was very good (Figure 2).
A major limitation of AI-generated questions is their relatively low cognitive complexity and difficulty, as noted in previous studies [10]. In our study, AIQ were only slightly more difficult overall and were solved with high accuracy, particularly by residents, suggesting limited usefulness for specialty-level examinations.
Another important concern is quality. Prior reports indicate frequent errors and the need for substantial revisions in AI-generated content [10, 11].
Similarly, in our study AIQ were rated as less clear and less correct than HQ, particularly by specialists, likely reflecting higher expectations and greater sensitivity to inaccuracies, consistent with earlier findings [12]. Less experienced participants tended to rate both AIQ and HQ more favourably.
Despite this, participants were generally unable to reliably distinguish between AI- and human-generated questions, which aligns with previous studies [13–15]. Interestingly, AIQ were perceived as slightly more difficult, although objective indices indicated mostly easy-to-moderate difficulty levels. The Difficulty and Discrimination Indices were comparable between AIQ and HQ, suggesting similar ability to differentiate between participants despite differences in perceived quality.
Importantly, HQ in this study were derived from real examination databases rather than created for research purposes, reflecting routine educational practice. Overall, HQ were rated as clearer and more accurate, while AIQ demonstrated acceptable but inferior quality.
These findings suggest that LLMs may be useful for generating student-level examination content but are not yet suitable for specialty-level assessments without expert review. The presence of factual and structural errors highlights the necessity of human oversight.
