Table 3

Performance comparison between the system and three DR experts

DimensionReadernAccuracyF1 scoreSensitivitySpecificityAUROC (95% CI)
TNFPFNTP
Referable retinopathyExpert 118145620.9640.9320.9250.978NA
Expert 218146610.9600.9240.9100.978NA
Expert 318322650.9840.970.9700.989NA
Experts’ averageNANANANA0.9700.9420.9350.982NA
DLA17690670.9640.9371.0000.9510.9958 (0.9916 to 1.0000)
Referable maculopathyExpert 117137710.9600.9340.9100.983NA
Expert 216774740.9560.9310.9490.960NA
Expert 316684740.9520.9250.9490.954NA
Experts’ averageNANANANA0.9560.930.9360.966NA
DLA163114740.9400.9080.9490.9370.9877 (0.9756 to 0.9999)
**Referable DRExpert 116437780.9600.940.9180.982NA
Expert 216166790.9520.9290.9290.964NA
Expert 316074810.9560.9360.9530.958NA
Experts’ averageNANANANA0.9560.9350.9330.968NA
DLA16344810.9680.9530.9530.9760.9909 (0.9809 to 1.0000)
  • NA, not applicable.

  • *By integrating the prediction of referable retinopathy and maculopathy on an image, referable DR decisions were given by the system when any referable lesion is detected.

  • AUROC, area under the receiver operating characteristic curve; DLA, deep learning algorithm; DR, diabetic retinopathy; FN, false negative; FP, false positive; TN, true negative; TP, true positive.