Table 2

Performance of the five classifiers of the system

ClassifiersData setnAccuracyF1 scoreSensitivitySpecificityAUROC (95% CI)
TNFPFNTP
Image qualityTraining5818180174132 1450.9520.9710.9490.970.9882 (0.9874 to 0.9891)
Validation8046423141650.9440.9660.9470.9260.9812 (0.9780 to 0.9844)
Test109510341864470.9350.9610.9390.9140.9768 (0.9737 to 0.9798)
External validation4706243142923 8760.9450.9660.9440.9510.9751 (0.9732 to 0.9770)
RetinopathyTraining30 118544932150.9840.9210.9970.9820.9992 (0.9991 to 0.9994)
Validation3916107153580.9720.8540.960.9730.9956 (0.9941 to 0.9970)
Test6031178166400.9720.8680.9760.9710.9962 (0.9951 to 0.9972)
External validation21 6097846528470.9660.870.9780.9650.9944 (0.9936 to 0.9952)
Maculopathy gradabilityTraining5068148126927 4010.9580.9750.9560.9720.9934 (0.9928 to 0.9940)
Validation6174317435620.9510.970.9530.9350.9896 (0.9873 to 0.9918)
Test9704930255440.9490.9690.9480.9520.9890 (0.9871 to 0.9910)
External validation4374451170418 7760.9150.9460.9170.9070.9639 (0.9617 to 0.9660)
MaculopathyTraining24 30147017037290.9780.9210.9560.9810.9962 (0.9957 to 0.9967)
Validation3186104294170.9640.8620.9350.9680.9906 (0.9864 to 0.9948)
Test4900130617550.9670.8880.9250.9740.9928 (0.9912 to 0.9944)
External validation16 98757215027710.9650.8850.9490.9670.9904 (0.9888 to 0.9919)
PhotocoagulationTraining32 526105012550.9970.961.0000.9971.0000 (0.9999 to 1.0000)
Validation42522751120.9930.8750.9570.9940.9924 (0.9794 to 1.0000)
Test65893382350.9940.920.9670.9950.9979 (0.9958 to 1.0000)
External validation24 277467295320.980.6820.9480.9810.9904 (0.9869 to 0.9940)
Referable DR*
ImagesTraining23 88836214748900.9830.9510.9710.9850.9980 (0.9977 to 0.9983)
Validation313889285440.9690.9030.9510.9720.9932 (0.9899 to 0.9965)
Test4838114559530.9720.9190.9450.9770.9952 (0.9940 to 0.9964)
External validation16 66757511738590.9670.9180.9710.9670.9931 (0.9920 to 0.9942)
EyesTraining14 76441111031360.9720.9230.9660.9730.9961 (0.9955 to 0.9967)
Validation198687153420.9580.870.9580.9580.9906 (0.9850 to 0.9961)
Test2949117366080.9590.8880.9440.9620.9923 (0.9901 to 0.9946)
External validation94296245918760.9430.8460.970.9380.9884 (0.9863 to 0.9905)
PatientsTraining84152917421410.9670.9210.9670.9670.9956 (0.9949 to 0.9964)
Validation113864102370.9490.8650.960.9470.9894 (0.9837 to 0.9951)
Test166980254070.9520.8860.9420.9540.9914 (0.9884 to 0.9943)
External validation46834923712190.9180.8220.9710.9050.9848 (0.9819 to 0.9877)
  • *By integrating the prediction of referable retinopathy and maculopathy on an image, referable DR decisions were given by the system when any referable lesion was detected, and the accuracies were based on the image, eye and patient levels.

  • AUROC, area under the receiver operating characteristic curve; DR, diabetic retinopathy; FN, false negative; FP, false positive; TN, true negative; TP, true positive.