Table 3

Performance metrics for the various classification systems

NLDNLDweighted LCRFLCextended RF_extended
GICU
 AUROC0.7913 (0.0098)0.8197 (0.0098)0.8788 (0.0087)0.8692 (0.0093) 0.8822 (0.0091) 0.8721 (0.0094)
 Accuracy0.7222 (0.0248)0.7829 (0.0339)0.8397 (0.0492)0.8389 (0.0496)0.8318 (0.0475) 0.8426 (0.0505)
 F10.7473 (0.0109)0.7709 (0.0153)0.8109 (0.0099)0.8102 (0.0115)0.8050 (0.0119) 0.8129 (0.0109)
 Specificity0.7000 (0.0000)0.7000 (0.0000)0.7000 (0.0000)0.7000 (0.0000)0.7000 (0.0000)0.7000 (0.0000)
 pAUROC0.1469 (0.0061)0.1471 (0.0076)0.1961 (0.0068)0.1876 (0.0078) 0.1989 (0.0068) 0.1888 (0.0079)
 Brier0.2677 (0.0060)0.2265 (0.0083)0.1465 (0.0052)0.1502 (0.0056) 0.1439 (0.0059) 0.1482 (0.0049)
 Sensitivity0.7426 (0.0166)0.8098 (0.0263)0.8870 (0.0171)0.8860 (0.0196)0.8767 (0.0196) 0.8909 (0.0185)
MIMIC
 AUROC0.7442 (0.0059)0.8248 (0.0056)0.8549 (0.0124)0.8605 (0.0122)0.8726 (0.0108) 0.8859 (0.0110)
 Accuracy0.6783 (0.0125)0.8007 (0.0358)0.8366 (0.0513)0.8387 (0.0517)0.8494 (0.0533) 0.8531 (0.0545)
 F10.6908 (0.0120)0.7830 (0.0103)0.8084 (0.0171)0.8097 (0.0158)0.8175 (0.0123) 0.8201 (0.0133)
 Specificity0.7000 (0.0000)0.7000 (0.0000)0.7000 (0.0000)0.7000 (0.0000)0.7000 (0.0000)0.7000 (0.0000)
 pAUROC0.1238 (0.0030)0.1429 (0.0043)0.1677 (0.0100)0.1729 (0.0099)0.1837 (0.0092) 0.1955 (0.0091)
 Brier0.2510 (0.0029)0.1986 (0.0046)0.1470 (0.0065)0.1472 (0.0069)0.1394 (0.0056) 0.1388 (0.0064)
 Sensitivity0.6713 (0.0126)0.8337 (0.0174)0.8827 (0.0282)0.8860 (0.0265)0.9001 (0.0207) 0.9049 (0.0210)
  • All scores are averaged over 100 train-test data splits and given as: mean (SD). All metrics other than AUROC and Brier score are evaluated at a specificity of 0.7, using linear interpolation to estimate this operating point in receiver-operator-characteristic-space. NLDweighted are the NLD criteria, weighted by feature importances from the LC. LCextended and RFextended are the machine learning classifiers with extended feature sets.

  • Best scores for each metric are shown in bold.

  • GICU, general intensive care unit; LC, logistic classifier; NLD, nurse-led discharge; RF, random forest.