Article Text

Download PDFPDF

Predicting the risk of stroke among patients with type 2 diabetes: a systematic review and meta-analysis of C-statistics
  1. Mohammad Ziaul Islam Chowdhury1,
  2. Fahmida Yeasmin2,
  3. Doreen M Rabi1,
  4. Paul E Ronksley3,
  5. Tanvir C Turin4
  1. 1 Department of Community Health Sciences, University of Calgary Cumming School of Medicine, Calgary, Alberta, Canada
  2. 2 Department of Mathematics and Statistics, University of Calgary, Calgary, Alberta, Canada
  3. 3 Department of Community Health Sciences, University of Calgary, Calgary, Alberta, Canada
  4. 4 Department of Family Medicine, University of Calgary, Calgary, Alberta, Canada
  1. Correspondence to Dr Tanvir C Turin; chowdhut{at}ucalgary.ca

Abstract

Objective Stroke is a major cause of disability and death worldwide. People with diabetes are at a twofold to fivefold increased risk for stroke compared with people without diabetes. This study systematically reviews the literature on available stroke prediction models specifically developed or validated in patients with diabetes and assesses their predictive performance through meta-analysis.

Design Systematic review and meta-analysis.

Data sources A detailed search was performed in MEDLINE, PubMed and EMBASE (from inception to 22 April 2019) to identify studies describing stroke prediction models.

Eligibility criteria All studies that developed stroke prediction models in populations with diabetes were included.

Data extraction and synthesis Two reviewers independently identified eligible articles and extracted data. Random effects meta-analysis was used to obtain a pooled C-statistic.

Results Our search retrieved 26 202 relevant papers and finally yielded 38 stroke prediction models, of which 34 were specifically developed for patients with diabetes and 4 were developed in general populations but validated in patients with diabetes. Among the models developed in those with diabetes, 9 reported their outcome as stroke, 23 reported their outcome as composite cardiovascular disease (CVD) where stroke was a component of the outcome and 2 did not report stroke initially as their outcome but later were validated for stroke as the outcome in other studies. C-statistics varied from 0.60 to 0.92 with a median C-statistic of 0.71 (for stroke as the outcome) and 0.70 (for stroke as part of a composite CVD outcome). Seventeen models were externally validated in diabetes populations with a pooled C-statistic of 0.68.

Conclusions Overall, the performance of these diabetes-specific stroke prediction models was not satisfactory. Research is needed to identify and incorporate new risk factors into the model to improve models’ predictive ability and further external validation of the existing models in diverse population to improve generalisability.

  • stroke
  • risk
  • prediction model
  • systematic review
  • meta-analysis

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

View Full Text

Statistics from Altmetric.com

Strengths and limitations of this study

  • The breadth of the comprehensive systematic literature search is a strength of this study.

  • To our knowledge, this is the first study where a meta-analysis and study quality assessment was performed on stroke prediction models in patients with diabetes.

  • We were only able to use C-statistics to compare the model performance, which might be insensitive to identify differences in models’ ability to accurately risk-stratify patients into clinically meaningful risk groups.

Introduction

Stroke, also known as a cerebrovascular accident, is the third leading cause of disability and accounted for over 6 million deaths worldwide in 2015.1 2 Diabetes mellitus, characterised by chronic hyperglycaemia due to an absolute or relative deficiency in insulin, is a major risk factor for stroke. People with diabetes are at a twofold- to fivefold increased risk for stroke compared with people without diabetes.3–7 Large clinical trials performed in people with diabetes supports the need for targeted cardiovascular risk reduction strategies to prevent the onset, recurrence and progression of acute stroke.8

Risk prediction models are statistical tools to estimate the probability that an individual with specific risk factors (eg, diabetes mellitus) will develop a future condition, such as stroke, within a certain time period (eg, 5 years).9 Such tools for the estimation of stroke risk are frequently used to assist in decisions about clinical management for both individuals and populations. Accurate risk prediction of stroke is thus necessary to provide patients with accurate information on the expected benefit from a therapy or intervention. The importance of well-performing prediction models is increasingly being recognised and health researchers continue to develop parsimonious risk prediction models under different scenarios to meet this demand. Model performance statistics, such as C-statistic or AUC (area under the receiver operating characteristic curve) are indicators frequently used to identify models with the best predictive ability. These metrics can be compared and assessed through a formal systematic review and meta-analysis. Performing a systematic review and meta-analysis can also provide a comprehensive quantitative summary of the predictive ability of these models and evaluate their predictive performance within the available literature.

Risk factors for stroke include lifestyle-related factors,10 11 predisposing medical conditions,10 12 specific genetic diseases,13 14 as well as sociodemographic factors.11 12 Over the past decade, a number of prediction models (or risk scores) have been developed incorporating these risk factors to predict a person’s risk of developing stroke.15 Prediction of stroke is important for a number of reasons: to detect or screen high-risk subjects to prevent developing stroke through early interventions, to facilitate patient–doctor communication based on more objective information and to help patients to make an informed choice regarding their treatment. While multiple stroke prediction models have been proposed in patients with diabetes, little is known about which is the most accurate one. There has also been a lack of consistency in estimating risk across these different models. With this in mind, we aimed to systematically identify all prediction models for stroke that have been applied to patients with diabetes. We characterised the study populations in which these models were derived and validated. We also assessed the predictive performance and generalisability of these stroke prediction models so that the selection of models for clinical implementation can be informed.

Methods

Data sources and searches

Similar to previously employed methodology,16 we searched MEDLINE, EMBASE and PubMed (from database inception to 22 April 2019) for studies predicting the risk of stroke among patients with diabetes. We also searched the reference lists of all identified relevant publications. The search strategy focused on three key elements: diabetes, risk prediction with specific names of known risk scores and stroke. Only studies published in English were considered. The detailed search strategy is given in online supplemental table S1.

Study selection

Eligible articles were identified by two reviewers independently using a two-step process. First, an initial screen of titles and abstracts was performed. Abstract were retained if they reported data from an original study and reported on the development and/or validation of a stroke risk prediction model for patients with type 2 diabetes. We defined a stroke risk prediction model as one combining two or more independent variables to obtain estimates of the predicted risk for developing stroke. We considered any clinical-based or laboratory-based definition of stroke. Selected abstracts were further screened based on a full-text review. We used broad inclusion criteria to provide an extensive systematic review of the topic. There were no restrictions on study design (eg, cohort study, case–control study), geographic region or age ranges. Studies that developed prediction models for stroke in populations with type 2 diabetes and in the general population were included; however, models that were developed in the general population but did not validate their model in a type 2 diabetes population or models developed on a type 1 diabetes population were excluded. A study was included if the outcome of the prediction model was any type of stroke or stroke that was part of a composite cardiovascular disease (CVD) outcome, but excluded if the outcome was any other cardiovascular conditions (eg, coronary heart disease (CHD), coronary artery disease (CAD), heart failure). We excluded studies that did not predict stroke. Studies on recurrent stroke or other vascular conditions (eg, patients with hypertension) were also excluded. Studies that focused only on the added predictive value of new risk factors to an existing prediction model without reporting the performance of the existing model were excluded. Studies on score-based tools, such as risk charts were also excluded. Agreement between reviewers at the full-text stage was quantified using the kappa statistic. Any disagreement between reviewers was solved through consensus.

Data extraction

Data were extracted from the finally selected studies using a standardised form by two reviewers. Information collected from each study included, outcome of the prediction model, location where the model was developed, predictors included in the model, age and gender of the study participants, number of events, duration of follow-up, modelling method used, measures of discrimination and calibration of the prediction model and the external validation of the prediction model. For the external validation studies, a different data extraction sheet was used. The collected information included specifics of the validation population, number of events, type of outcome, statistical tests and measures of discrimination, and calibration of the prediction model. Study quality was assessed using the CHARMS (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies) Checklist.17 The following items were evaluated for each study: Was inclusion/exclusion criteria for study participants specified?; Was there non-biased selection of study participants?; Did the authors discuss or consider missing values/information?; Was there blinded assessment of the outcome?; Was duration of follow-up adequate?; Were modelling assumptions satisfied?; Was the model externally validated? and Was the potential clinical utility of the model discussed in light of study limitations?

Data analysis

The selection process for this systematic review and meta-analysis is summarised using the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram.18 Discrimination is defined as any assessment of the ability of the model to differentiate between subjects who will develop stroke from those who will not. The discrimination of a prediction model is often assessed using the concordance or C-statistic (also known as AUC). Calibration is defined as any report of the agreement between predicted probabilities and observed probabilities. Calibration is assessed using goodness-of-fit tests (eg, Hosmer-Lemeshow test), calibration slopes, tabular or graphical comparisons of predicted versus observed values within groupings of predicted risk or calibration plots. In studies that only provided a C-statistic but no measure of its variance, the SE and 95% CI of the AUC (C-statistic) was calculated using the formula:

Embedded Image

where Embedded Image the number of patients with stroke and Embedded Image the number of patients without stroke and the upper 95% CI Embedded Image , and lower 95% CI Embedded Image .19 The summary statistic from the individual studies was the C- statistic or AUC. We grouped studies based on the outcome of the risk prediction models developed in diabetes populations, whether stroke was the primary outcome of the model or stroke was a part of composite CVD outcome. Random effects meta-analysis was used to obtain the pooled weighted average C-statistic with 95% CIs for common groups of models using the DerSimonian and Laird method.20 Heterogeneity was assessed using the Cochran Q and the I2 statistic and was explored using meta-regression and stratified analyses according to model outcomes. Small study effects were examined using funnel plots and Begg’s test. The analyses were performed using Stata version 13.1 (Stata, College Station, Texas, USA) using the metan, metareg, metabias and metafunnel commands.

Patient and public involvement

There was no direct patient or public involvement in this review.

Results

The search retrieved 21 797 citations (after duplicate removal) with an additional 63 potentially relevant papers retrieved from our grey literature search. After title and abstract screening, 262 studies were selected for full-text screening. After examining the full-text papers, 56 studies remained (reasons for exclusion stated in figure 1), describing 38 models predicting stroke in patients with diabetes. Agreement between reviewers on the final articles eligible for inclusion in the systematic review was good (κ=0.83). Of these 38 models, 34 were specifically developed in patients with diabetes and 4 were developed in the general population but later externally validated in patients with diabetes. Among the models developed in patients with diabetes, nine models reported their outcome as stroke and presented a corresponding performance measure (C-statistic) for the models. Twenty-three models reported their outcome as a composite CVD outcome where stroke was one of the components and presented the model’s performance measure (C-statistic) for the composite CVD outcome. Among the models developed in the general population, one model reported its outcome as stroke and three models reported a composite CVD outcome, which included stroke. Of these 38 prediction models, 17 were validated by 33 studies (some studies validated more than one model in the same study), of which 10 models had multiple validations, 7 models had a single validation and 21 models were not validated. Among the models with multiple validations, eight models were developed in a diabetes population (validated by 31 studies) and two were developed in the general population (validated by four studies). United Kingdom Prospective Diabetes Study (UKPDS) Risk Engine for Stroke by Kothari et al 21 was the most validated risk score (validated by 12 studies). Figure 1, describes the systematic selection process of studies presenting a stroke prediction model applicable to patients with diabetes.

Figure 1

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) diagram for systematic review of studies presenting stroke prediction models developed or validated in individuals with diabetes.

Predicting the risk of stroke within models developed in populations with diabetes

Table 1 describes the study characteristics of the nine risk prediction models developed in diabetes populations and presented a corresponding performance measure. The number of participants ranged from 1748 to 26 140 in the model development. The outcome of most models was stroke regardless of type. Duration of follow-up (total/median/mean) ranged from 501 days to 10.5 years with six models having ≥5 years of follow-up (defined as long duration) and three models with <5 years of follow-up. Most of the prediction models were developed using Cox proportional hazards modelling techniques. The number of predictors included in the prediction models ranged from 4 to 29 with an average of 11 predictors per model. Several predictors were common to multiple models including age, sex, duration of diagnosed diabetes, systolic blood pressure and haemoglobin A1c (HbA1c). Only two models were externally validated after their development and four of them had never been validated in an external population. Calibration of the prediction model was reported by six studies (most commonly using the Hosmer-Lemeshow test). Discrimination was assessed using the C-statistic (or AUC) and reported by six models with values ranging from 0.64 to 0.80. The median C-statistic of the models was 0.71 with a large amount of unexplained heterogeneity in the discriminative performance of these models (I2=94.6%; Cochran Q-statistic p<0.001; figure 2). Stratifying pooled results by sample size (small vs large, p=0.19), follow-up time (short vs long, p=0.60), variables included in the model (few vs many, p=0.24) and geographic location (Asia vs others, p=0.60) did not explain the observed heterogeneity in the discriminative performance of these models. The discriminative ability of the model by Kiadaliri et al 22 was highest (C-statistic=0.80). The funnel plot and Begg’s test (p>0.999) suggested the absence of small study effects, with no correlation between studies of smaller cohorts reporting higher C-statistics (online supplemental figure S1).

Figure 2

Forest plot of C-statistics, with 95% CIs of risk prediction models when outcome was reported for stroke.

Table 1

Characteristics of prediction models when outcome and corresponding performance measure (C-statistic) were reported for stroke

A set of nine criteria was used to assess the quality of the studies and was summarised in table 2. All the studies specified inclusion/exclusion criteria. Non-biased selection of study participants was clear in all studies except the study by Palmer et al.23 Handling of missing values was reported in four (44%) studies, modelling assumptions was satisfied by two studies and model external validation was performed in two studies. Stevens et al was the only study to mention whether the outcome was assessed without knowledge of the candidate predictors.24 Duration of follow-up was long (≥5 years) in six models (67%). The clinical utility of the models was discussed in six (67%) studies and almost all studies reported their study limitations.

Table 2

Study quality assessment of prediction models when outcome and corresponding performance measure (C-statistic) were reported for stroke

Predicting risk of stroke (as part of a composite CVD outcome) in populations with diabetes

We identified 23 models developed in diabetes populations that reported their outcome as a composite of CVD. A summary of the characteristics of these prediction models is described in table 3. The number of participants considered in the model development ranged from 132 to 1 81 619 with an average age of >50 years. Duration of follow-up (mean, median, maximum) ranged from 11 months to 11.8 years with 14 models with ≥5 years and nine models with <5 years of follow-up. The number of predictors included in prediction models ranged from 4 to 18 with an average of 11 predictors per model. The most common predictors included in the models were age, sex, systolic blood pressure and HbA1c, smoking and high-density lipoprotein-cholesterol. Four models were externally validated after its development and 17 of them had never been validated in an external population. Calibration of the prediction models was reported by 13 studies while discrimination reported by almost all studies with C-statistics ranging from 0.60 to 0.92. The median C-statistic of the models was 0.70 with a large amount of unexplained heterogeneity in the discriminative performance of these models (I2=93.7%; Cochran Q-statistic p<0.001; figure 3). Sample size (small vs large, p=0.46), models’ external validation (externally validated vs not externally validated, p=0.71), variables included in the model (few vs many, p=0.21) and geographic location (Europe vs others, p=0.08) were not identified as significant sources of heterogeneity in the discriminative performance of these models. The discriminative ability of the model by Ofstad et al 25 was highest (C-statistic=0.92) when novel risk markers were added to their standard model. The funnel plot and Begg’s test (p=0.24) suggested the absence of small study effects, with no correlation between studies of smaller cohorts reporting higher C-statistics (online supplemental figure S2). Only four models were developed using logistic regression models while others were developed mostly using Cox proportional hazards models.

Figure 3

Forest plot of C-statistics, with 95% CIs of risk prediction models when stroke was reported as part of a composite cardiovascular disease outcome. AD-ON, Action in Diabetes and Vascular Disease:Preterax and Diamicron Modified Release Controlled Evaluation-Observational; EPIC-NL,European Prospective Investigation into Cancer and Nutrition-Netherlands; HWNNs,hybrid wavelet neural networks; MACE, major adverse cardiovascular event; NZDCS, New Zealand Diabetes Cohort Study; SMART, second manifestations ofarterial disease; SOMs, self-organising maps.

Table 3

Characteristics of prediction models when stroke is reported as a part of composite CV outcome and performance measure (C-statistic) is presented for the composite CV outcome

The study quality for this group of models is summarised in table 4. Similar to the models developed in diabetic populations that look at the outcome of stroke specifically, we found that study quality was similar.

Table 4

Study quality assessment of prediction models when stroke is reported as a part of composite CV outcome and performance measure (C-statistic) is presented for the composite CV outcome

Validation studies of stroke prediction models developed in populations with and without diabetes

Seventeen risk prediction models for stroke (developed both in patients with diabetes and in general populations) were validated in diabetes populations by 33 studies (table 5). Among the 17 validated models, 14 of them were externally validated in independent cohorts and 3 of them were internally validated in a test sample or separate data set from the same cohort. Three studies validated more than one risk model in the same cohort. Models with multiple validations (two or more) and reported C-statistics are provided in figure 4. Models that had only been validated once were excluded from meta-analysis. In addition, only those studies that provided enough information to estimate the variance of the provided C-statistic for meta-analysis were considered for analysis.

Figure 4

Forest plot of C-statistics, with 95% CIs, of stroke prediction models that are externally validated in two or more independent cohorts. JHS, Jackson Heart Study; MESA,Multiethnic Study of Atherosclerosis; UKPDS, United Kingdom ProspectiveDiabetes Study

Table 5

Characteristics of the validation studies of the stroke prediction models

UKPDS Risk Engine for Stroke by Kothari et al 21 was the most frequently externally validated model with a total of 12 studies reporting its performance in different diabetes cohorts. In 12 external validation studies, a total of 126 323 patients were included with considerable variations in sample sizes across the different studies. The pooled C-statistic for the model by Kothari et al 21 was 0.72 (95% CI, 0.68 to 0.75), with high heterogeneity identified (I2=95%; Cochran Q statistic p<0.001). Stratification by sample size (small vs large, p=0.69), geographic location (Asia vs others, p=0.09) and stroke type (fatal vs non-fatal, p=0.07) did not explain the observed heterogeneity in the discriminative performance of this model. UKPDS Risk Engine by Stevens et al 26 was the second most externally validated model with five validation studies including 2826 patients. One study did not report the number of participants and none of the studies reported C-statistics. As a result, a pooled C-statistic and heterogeneity was not possible to assess for this model. The UKPDS Outcomes Model by Clarke et al 27 was externally validated by four studies including 65 056 patients. The pooled C-statistic was 0.66 (95% CI, 0.61 to 0.71) with high heterogeneity between studies (I2=84.5%; Cochran Q statistic p=0.002). Similar to the UKPDS Risk Engine for Stroke,21 stratification across select study characteristics did not explain the observed heterogeneity. The Framingham risk score by Anderson et al 28 was externally validated in two studies including 8574 patients. The pooled C-statistic was 0.58 (95% CI, 0.54 to 0.61) with non-significant heterogeneity between studies (I2=56.1%; Cochran Q statistic p=0.102). The Framingham risk score by D’Agostino et al 29 was externally validated in two studies including 7604 patients. One study (Ataoglu et al 30) did not report the C-statistic for the model and one study (Kengne et al 31) reported two C-statistic values, one for major events and one for any event. The pooled C-statistic for these two values was 0.58 (95% CI, 0.55 to 0.60). Models by Mukamal et al.,32 Davis et al.,33 Kengne et al 34 and Zethelius et al 35 each were externally validated by two studies with pooled C-statistics of 0.67 (95% CI, 0.67 to 0.68), 0.75 (95% CI, 0.58 to 0.92), 0.67 (95% CI, 0.65 to 0.69) and 0.69 (95% CI, 0.63 to 0.75), respectively. Observed heterogeneity was high in models by Davis et al 33 and Zethelius et al 35 while low in models by Mukamal et al 32 and Kengne et al.34 The model by Basu et al 36 was externally validated by two studies in three different population yielding a pooled C-statistic of 0.71 (95% CI, 0.67 to 0.76) with moderate heterogeneity between studies (I2=56.8%; Cochran Q statistic p=0.099).

Separate models by D'Agostino et al (Framingham Stroke Risk Score),37 Yang et al (Hong Kong Diabetes Registry for Stroke),38 Kiadaliri et al,22 Stevens et al (UKPDS 66),24 Hippisley-Cox et al (QRISK2),39 Elley et al (New Zealand Diabetes Cohort Study)40 and Alrawahi et al 41 were each validated in one external or separate cohort with sample sizes ranging from 178 to 1 81 399 patients. For the studies that reported discrimination, C-statistics ranged from 0.67 to 0.79. In addition, calibration assessed by calibration plots and Hosmer-Lemeshow tests found good calibration in most studies.

The overall pooled C-statistic for all validation studies was 0.68 (95% CI, 0.67 to 0.70) with high heterogeneity between studies (I2=95.3%; Cochran Q statistic p<0.001). Models that were developed in diabetes population showed significantly higher C-statistics than models developed in general populations (meta-regression p=0.001). Models, where stroke was reported as the main outcome as opposed to part of a composite CVD outcome, did show borderline significantly higher C-statistics (meta-regression p=0.052), although the value of the C-statistic is still low. This observed difference in the two models makes sense as models that include stroke as part of a composite outcome are expected to be different from models where stroke is the only outcome. A summary describing the characteristics of the studies where prediction models were developed in general populations but validated in patients with diabetes is presented in table 5.

Discussion

This systematic review and meta-analysis provides an overview of all stroke prediction models that were specifically developed for, or validated in patients with diabetes to calculate future stroke risk. Thirty-four stroke prediction models were identified that were specifically designed for patients with diabetes and only 32% of these prediction models have been externally validated, with varying results. Overall, the pooled C-statistics were poor for most models. Four of the prediction models identified were originally developed in the general population but externally validated in diabetes populations. The most notable prediction model was the UKPDS Risk Engine for Stroke21 with 12 validation studies. Ten stroke prediction models had multiple validations, seven models had single validations and twenty-one had no validations at all. It is difficult to assess model performance for those with no validation or single validations. Additional validation studies on the performance of stroke prediction models in different diabetes populations are needed. Since stroke prediction models developed in the general population may not account for specific risk factors related to diabetes, using risk scores developed specifically in the diabetes population will help to estimate stroke risk among people with diabetes more accurately.

None of the models showed good discriminative performance consistently when externally validated. The model by Kothari et al 21 where the stroke was the primary outcome showed moderate discriminative performance (pooled C-statistic=0.72). Since this model was externally validated multiple times, the performance of this model can be considered as consistent. The discriminative ability of stroke prediction models where stroke was the primary outcome and models where stroke was a part of composite CVD outcome were modest, with C-statistics often less than 0.70.42 Meta-analyses of the C- statistic suggests that there is significant between-study heterogeneity in the models where stroke is reported as the primary outcome and in those where stroke is reported as part of composite CVD outcome. Further, the possible sources of heterogeneity are unexplained. Perhaps the difference in patient characteristics in the different cohorts could be a potential source of heterogeneity; however, geographic location, sample size, follow-up time, external validation and variables included in the models were not significant sources of heterogeneity in meta-regression.

The discrimination of the 17 models that were validated were generally comparable with those observed in the development cohorts. However, the performance of some models externally validated in multiple cohorts was heterogeneous and possible source for this heterogeneity remains unexplained. There was also variability in prediction model quality and the methodology used in developing them. Our study findings suggest that, from a large number of published models in patients with diabetes, very few well-validated models are available for stroke prediction. This is helpful to inform the determination of models for clinical uptake when risk stratification approaches for stroke are implemented.

No evidence of small-study effects was detected, in which smaller studies reported better discrimination of models for predicting stroke. Study quality assessment shows many of the models failed to meet some key criteria: consideration of missing values, modelling assumptions, model validation and blinded outcome assessment, which is a concern. Many studies lacked standard reporting. This, to some extend, may be due to lack of guidelines for standards of reporting for risk prediction studies during that time. Many authors reported different aspects of prediction models, and in varying ways created difficulty in collecting information. The publication of new guidelines such as Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD)43 has been introduced and may help improve reporting standards in subsequent studies in this area.

In prior reviews examining risk prediction models in adults with diabetes (Chamnan et al 44 van Dieren et al,45 and Chowdhury et al 16), all components of cardiovascular disease such as CHD, stroke, CAD, myocardial infarction, heart failure were considered as outcomes of the prediction model. Our review adds to knowledge on predicting risk of stroke in persons with diabetes in the following ways: (1) We only considered models where the primary outcome of the model was stroke or when stroke was part of a composite CVD outcome and corresponding C-statistic were provided; (2) We did not consider other components of CVD as outcomes of the model and therefore our estimates of model performance are more specific to stroke; (3) We have identified and included several recently derived models and conducted meta-analyses to explore reasons for variability in the discriminative performance across models and (4) We provide a detailed assessment of quality of studies among models developed in diabetes populations. Only one prior study16 in this area performed a meta-analysis of model performance statistics across multiple studies or assessed study quality.

One of the major strengths of our study is the breadth of the systematic search, which included three different databases and extensive use of reference lists of the identified studies. Therefore, it is unlikely that any stroke prediction model-related studies have been missed. To best of our knowledge, this is the first study, where a meta-analysis and study quality assessment was performed on stroke prediction models in patients with diabetes. Nonetheless, there are few limitations in our study, which need to be kept in mind. In this paper, we only considered studies that developed or validated stroke prediction models within patients with diabetes. While prediction models for stroke have been developed for patients with other potential risk factors (eg, patients with hypertension), we felt that an exploration of a broad range of risk factors was outside the scope of this review. Though the inclusion of all stroke prediction models (regardless of the underlying risk factor(s)) could potentially improve the generalisability of our findings, it could have also increased the between-study heterogeneity, making the pooled estimates more difficult to interpret. We also did not consider non-English publications. Although, the English language is generally perceived to be the universal language of science, selection of research findings in a particular language can introduce language bias and may lead to erroneous conclusions. With this in mind, readers should to be cautious when interpreting the findings of our results. Finally, we were only able to use C-statistics to compare the model performance, which might be insensitive to identify differences in the ability of models to accurately risk-stratify patients into clinically meaningful risk groups.46 In addition, meta-analysis of calibration measures (eg, E/O ratio) along with C-statistics could give a comprehensive summary of the performance of these models.

Our findings suggest that there is no significant difference between the discrimination of models where stroke was the primary outcome and stroke was part of composite CVD outcome. Models, particularly those that have never been validated or validated once need to undergo further external validation in which they will be used with or without recalibration or model updating to better understand the comparative performance of these models.

Conclusions

In conclusion, we have identified many models for predicting stroke in patients with diabetes and attempted to compare these models. Only a small number of models have undergone external validation and might provide generalisable predictions that would support their use in another clinical setting. It is difficult to choose one model over another as none of these models exhibited superior discriminative performance, and unfortunately, no single model appears to perform consistently well. It could be argued that risk prediction in patients with diabetes is not essential. Persons with diabetes are generally perceived to be at elevated risk of stroke and the current practice is to treat to common HbA1C, blood pressure and low-density lipoprotein targets based on diabetes status alone and not on calculated risk. This non-risk based approach may be leading to unnecessary overtreatment and the absence of high-quality validated risk prediction models which limits our ability to assess whether more targeted approaches are possible. Further research is warranted to identify new risk factors with high associated relative risk to improve the currently available prediction models.

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.
  30. 30.
  31. 31.
  32. 32.
  33. 33.
  34. 34.
  35. 35.
  36. 36.
  37. 37.
  38. 38.
  39. 39.
  40. 40.
  41. 41.
  42. 42.
  43. 43.
  44. 44.
  45. 45.
  46. 46.
  47. 47.
  48. 48.
  49. 49.
  50. 50.
  51. 51.
  52. 52.
  53. 53.
  54. 54.
  55. 55.
  56. 56.
  57. 57.
  58. 58.
  59. 59.
  60. 60.
  61. 61.
  62. 62.
  63. 63.
  64. 64.
  65. 65.
  66. 66.
  67. 67.
  68. 68.
  69. 69.
  70. 70.
  71. 71.
  72. 72.
  73. 73.
  74. 74.
  75. 75.
  76. 76.
  77. 77.
  78. 78.
  79. 79.
  80. 80.
  81. 81.
  82. 82.
  83. 83.
  84. 84.
  85. 85.
  86. 86.
View Abstract

Footnotes

  • Contributors All authors contributed to this work. MZIC and TCT contributed to the conception and design of the review. MZIC and FY read and screened abstracts and titles of potentially relevant studies. MZIC and FY read the retained papers and were responsible for extracting data and rating their quality independently. MZIC performed the data analysis. MZIC drafted the paper and PER, DMR and TCT critically reviewed it and suggested amendments prior to submission. All authors approved the final version of the manuscript and take responsibility for the integrity of the reported findings.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient consent for publication Not required.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data availability statement No data are available.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.