Human versus artificial intelligence: evaluating ChatGPT’s performance in conducting published systematic reviews with meta-analysis in chronic pain research
Authors' conclusion
ChatGPT demonstrates modest to moderate accuracy in screening and study selection tasks, but performs well in data pooling and meta-analytic calculations.
Study context
- Population
- Data from a published meta-analysis on emotional functioning after spinal cord stimulation
- Comparison
- ChatGPT-4o vs Human
- Outcome
- Accuracy, sensitivity, specificity, PPV and NPV for screening and full-text review tasks
Performance
As reported in the abstract
Title and abstract screening (vs human gold standard)
| Test or model | Accuracy | Sensitivity | Specificity |
|---|---|---|---|
| ChatGPT | 70.4% | 54.9% | 80.1% |
ChatGPT
Accuracy70.4%
Sensitivity54.9%
Specificity80.1%
Further findings
| Outcome | Values |
|---|---|
| Data pooling (5 forest plots): pooled MDs, 95% CIs, I2 and tau-squared | ChatGPT accuracy for most outcomes 100% · tau-squared discrepancies, range 0.01-0.05 |
Data pooling (5 forest plots): pooled MDs, 95% CIs, I2 and tau-squared
- Forest plots showed no significant discrepancies
Parts of this Brief are withheld: the automatic checks could not confirm them against the abstract.
Original publication
Human versus artificial intelligence: evaluating ChatGPT’s performance in conducting published systematic reviews with meta-analysis in chronic pain research. Reg Anesth Pain Med. Published 16 February 2025. doi:10.
Composed by AI from the published abstract and checked against it. How Briefs are made →