Perioperative medicine & outcomes
Development and Validation of a Machine Learning Model to Identify Patients Before Surgery at High Risk for Postoperative Adverse Events
Performance · Discrimination of the automated model for 30-day mortality and MACCEResult — go to the resultDifference in AUROC 0.048Authors' conclusion
This study found that an automated machine learning model was accurate in identifying patients undergoing surgery who were at high risk of adverse outcomes using only preoperative variables within the electronic health record, with superior performance compared with the NSQIP calculator.
Study context
- Population
- Patients undergoing surgery at 20 community and tertiary care hospitals in the University of Pittsburgh Medical Center (UPMC) health network206,353 prospectively evaluated
- Participants
- 1,477,561 patients
- Comparison
- UPMC model vs National Surgical Quality Improvement Program (NSQIP)
- Outcome
- Postoperative mortality and major adverse cardiac and cerebrovascular events (MACCEs)
- Follow-up
- 30 days
Methodological detailsdevelopment and validation of a machine learning prediction model, retrospective development and testing, prospective validation in clinical care, gradient-boosted decision tree, multicentre (20 hospitals), Data analyzed September through December 2021
- Development and validation of a machine learning prediction model
- Retrospective development and testing
- Prospective validation in clinical care
- Gradient-boosted decision tree
- Multicentre (20 hospitals)
- Data analyzed September through December 2021
Performance
As reported in the abstract
Discrimination of the automated model for 30-day mortality and MACCE
Difference in AUROC
0.048
By group
UPMC model
- AUROC for mortality vs NSQIP (902 patients)
- 0.945 [95% CI, 0.914-0.977]
- Specificity
- 0.87 [95% CI, 0.83-0.89]
- Accuracy
- 0.85 [95% CI, 0.82-0.87]
NSQIP
- AUROC for mortality (902 patients)
- 0.897 [95% CI, 0.854-0.941]
- Specificity
- 0.68 [95% CI, 0.65-0.69]
- Accuracy
- 0.69 [95% CI, 0.66, 0.72]
AUROC for mortality vs NSQIP (902 patients)
UPMC model0.945 [95% CI, 0.914-0.977]
NSQIP— not stated in the abstract
Specificity
UPMC model0.87 [95% CI, 0.83-0.89]
NSQIP0.68 [95% CI, 0.65-0.69]
Accuracy
UPMC model0.85 [95% CI, 0.82-0.87]
NSQIP0.69 [95% CI, 0.66, 0.72]
AUROC for mortality (902 patients)
UPMC model— not stated in the abstract
NSQIP0.897 [95% CI, 0.854-0.941]
- AUROC for mortality, training set
- 0.972 (95% CI, 0.971-0.973)
- AUROC for mortality, test set
- 0.946 (95% CI, 0.943-0.948)
- AUROC for MACCE and mortality, training set
- 0.923 (95% CI, 0.922-0.924)
- AUROC for MACCE and mortality, test set
- 0.899 (95% CI, 0.896-0.902)
- Prospective evaluation: AUROC for mortality
- 0.956 (95% CI, 0.953-0.959)
- Prospective evaluation: sensitivity
- 2148 of 2517 patients (85.3%)
- Prospective evaluation: specificity
- 186 286 of 203 836 patients (91.4%)
- Prospective evaluation: NPV
- 186 286 of 186 655 patients (99.8%)
Further findings
| Outcome | UPMC model | NSQIP |
|---|---|---|
| Mortality prediction, UPMC model vs NSQIP | ||
| AUROC | 0.945 (95% CI, 0.914-0.977) | 0.897 (95% CI, 0.854-0.941) |
| Specificity | 0.87 (95% CI, 0.83-0.89) | 0.68 (95% CI, 0.65-0.69) |
| Accuracy | 0.85 (95% CI, 0.82-0.87) | 0.69 (95% CI, 0.66, 0.72) |
- Mortality prediction, UPMC model vs NSQIP: 902 patients
Original publication
Development and Validation of a Machine Learning Model to Identify Patients Before Surgery at High Risk for Postoperative Adverse Events. JAMA Netw Open. Published 3 July 2023. doi:10.
Composed by AI from the published abstract and checked against it. How Briefs are made →