An Exploratory Analysis of ChatGPT Compared to Human Performance With the Anesthesiology Oral Board Examination: Initial Insights and Implications
Authors' conclusion
ChatGPT generated SOE answers with comparable module ratings to anesthesiology fellows, as graded by 8 ABA oral board examiners. However, the ChatGPT answers were deemed subjectively inferior due to the length of responses and lack of focus.
Study context
- Population
- Anesthesiology fellows and ChatGPT answering 2 sample American Board of Anesthesiology (ABA) Standardized Oral Examination (SOEs), scored by 8 ABA applied examiners8 ABA examiners
- Comparison
- Fellows vs ChatGPT
- Fellowsn = 4reference
- Outcome
- Examiner scores
Methodological detailsexaminers scored randomized voice-replicated responses
- Examiners scored randomized voice-replicated responses
Main finding
As reported in the abstract
- Fellows vs ChatGPT: P = 0.03
- Fellows vs ChatGPT: P = 0.17
Further findings
Examiner identification of AI-generated answers
Modules identified as ChatGPT ChatGPT 23 of 24
Modules thought AI-generated Fellows 10 of 24
Examiner comments
- ChatGPT content relevant but lengthy, at times not focused on scenario priorities; no comments on fact hallucinations
Original publication
An Exploratory Analysis of ChatGPT Compared to Human Performance With the Anesthesiology Oral Board Examination: Initial Insights and Implications. Anesth Analg. Published 13 September 2024. doi:10.
Composed by AI from the published abstract and checked against it. How Briefs are made →