Suitability of GPT-4o as an evaluator of cardiopulmonary resuscitation skills examinations
Authors' conclusion
GPT-4o demonstrated a level of accuracy that was similar to that of senior experts in examining CPR skills examination videos.
Study context
- Population
- Examinees' cardiopulmonary resuscitation (CPR) skills test videos (patient assessment, chest compressions, rescue breathing, repeated operations)
- Participants
- 103 examinees
- Comparison
- GPT-4o scoring vs junior and senior expert scoring of CPR skills test videos
- Junior expertsn = 3reference
- Senior expertsn = 3reference
- Outcome
- Accuracy of GPT-4o: agreement with expert scores (intraclass correlation coefficient for first three sections; Fleiss' Kappa for repeated operations)
Methodological detailsvideo-based assessment, comparison with expert raters
- Video-based assessment
- Comparison with expert raters
Performance
As reported in the abstract
Mean accuracy (agreement) scores, GPT-4o vs experts, by section
| Test or model | Patient assessment | Chest compressions | Rescue breathing | Repeated operations |
|---|---|---|---|---|
| Junior experts | 0.65 | 0.58 | 0.60 | 0.31 |
| Senior experts | 0.75 | 0.65 | 0.72 | 0.41 |
Patient assessment
Junior experts0.65
Senior experts0.75
Chest compressions
Junior experts0.58
Senior experts0.65
Rescue breathing
Junior experts0.60
Senior experts0.72
Repeated operations
Junior experts0.31
Senior experts0.41
Further findings
| Outcome | Junior experts | Senior experts |
|---|---|---|
| Expert-rated reliability of GPT-4o (5-point Likert scale)Median | 4.00 (IQR 3.66–4.33) | 4.33 (IQR 4.00–4.67) |
| Mean (SD) | 3.95 [0.55] | 4.29 [0.50] |
Original publication
Suitability of GPT-4o as an evaluator of cardiopulmonary resuscitation skills examinations. Resuscitation. Published 28 September 2024. doi:10.
Composed by AI from the published abstract and checked against it. How Briefs are made →