Large Language Models Provide Accurate but Potentially Unsafe Answers to Multimodal Critical Care Medicine Board Review Questions
Authors' conclusion
ChatGPT-4o demonstrates performance slightly above pooled clinician benchmarks on critical care board-style questions but has substantial limitations in multimodal question interpretation.
Study context
- Population
- Multimodal board-style questions (multiple-choice with clinical images) from the Society of Critical Care Medicine item bank
- Participants
- 183 board-style questions
- Exposure
- ChatGPT-4o vs pooled clinician responses
- Outcome
- Accuracy (questions answered correctly)
Methodological detailsvalidated item bank, simulated board examination environment, expert clinician panel review, Responses reviewed by 14 clinical reviewers (physicians, advanced practice providers, and pharmacists)
- Validated item bank
- Simulated board examination environment
- Expert clinician panel review
- Responses reviewed by 14 clinical reviewers (physicians, advanced practice providers, and pharmacists)
Main finding
As reported in the abstract
Questions answered correctly
P = 0.03
Observational data: associations, not effects.
Further findings
ChatGPT-4o performance by component
- Question comprehension correct
- 87.4%
- Image interpretation
- 61.7%
- Reasoning
- 68.3%
- Supporting information
- 66.1%
ChatGPT-4o performance by domain
- Pulmonary disease
- 91.7%
- Surgery and trauma
- 87.5%
- Neurologic disorders
- 81.8%
- Critical care ultrasound
- 51.1%
Safety
- Responses associated with potential for clinical harm
- 33.3%
- Often due to incorrect image interpretation and treatment recommendations
Original publication
Large Language Models Provide Accurate but Potentially Unsafe Answers to Multimodal Critical Care Medicine Board Review Questions. Crit Care Med. Published 18 June 2026. doi:10.
Composed by AI from the published abstract and checked against it. How Briefs are made →