Design and data sources

This retrospective cohort study will use patient data collected from HES/surveillance clinic and other related databases/patient notes from three NHS trusts. Where required information is not available for extraction from the hospitals’ notes, it will be obtained from surveillance clinics if available (and vice versa) by the participating NHS trusts.

Patients and public involvement

Patients and the public have been involved in the design of this study (see online supplemental appendix 1 for details), and will be involved in the conduct, reporting and dissemination of this research.

Study population

The cohort will include type 1 and type 2 patients with diabetes, aged 12 years and over (as patients enter the screening programme from age 12), referred into HES or surveillance clinics with referable DR from DESP for close monitoring and treatment. Data will be collected from three NHS trusts for all patients within the catchment areas of Sandwell and Birmingham, Sunderland or Sussex NHS trusts. Records will be extracted for patients first entering the services between 1 January 2013 and 31 December 2016, with follow-up information extracted up to 31 December 2021. The Birmingham trust cares for an ethnically and socioeconomically diverse range of communities and was chosen to ensure equality, diversity and inclusion. The Sussex trust provides secondary care to a less diverse population and Sunderland is a primarily Caucasian population.

Patients with the specific outcome of retinopathy treatment or vision loss at referral or those referred for reasons other than retinopathy will be excluded. Patients objecting to their information being used (through a local or national opt out scheme) will also be excluded. Data flow and management stages are given in figure 2.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Data flow diagram. CAG, Confidentiality Advisory Group; NHS, National Health Service; PI, Principle investigator; R & D, Research and Development; SASH, Surrey and Sussex Healthcare NHS Trust; STS, South Tyneside and Sunderland NHS trust; SWBH, Sandwell and West Birmingham Hospitals NHS Trust.

Sample size

A minimum of 200 outcome events are required for external validation using current guidance for survival outcomes.24 Every trust receives approximately 200 referrals per year and we expect to have 4 years’ worth of data available for each trust. Therefore, we expect to have approximately 2400 patients from across the three trusts to ensure a minimum of 200 outcomes. Recruitment will stop once the target number is reached.

Using conservative estimates from our development data, we expect 15% of those referred to develop the outcome of interest within 3 years follow-up,22 providing at least 360 outcomes in the data that we will collect. For model updating, we will use the method of Riley et al25 to calculate the minimum sample size required, assuming an event rate of 0.05 per year, mean follow-up of 3.23 years, a default Nagelkerke R² of 0.15 and 19 candidate predictors considered in the model. A minimum of 1810 patients are required with 293 outcome events to target a shrinkage factor of 0.9 ensuring minimal overfitting to the data.

Data extraction

A data extraction sheet has been prepared using Microsoft Excel and will be piloted prior to use within each trust. Data will be extracted for all predictors at baseline (first HES appointment after referral) and visual acuity level will be collected at each follow-up appointment. Data will continue to be extracted until an outcome of interest occurs, death, lost to follow-up or the end of the study period (31 December 2021).

Personal information such as date of birth (for age calculation), date of diabetes diagnosis and date of death (for censoring or competing risks analysis) will be extracted. However, all the personal identifiable information will be removed after age, diabetes duration and follow-up durations have been calculated.

Outcomes

The primary outcome for this study is time to first treatment for DR or vision loss. Time will be calculated from referral to HES (baseline is first appointment) until the date of first treatment or vision loss. Time to death will also be recorded and treated as a competing risk if necessary.

Predictor variables

The previously developed DRPTVL-UK model included seven predictors measured at the time of or close to referral, namely (1) retinopathy stage, (2) HbA1c (mmol/mol), (3) eGFR (mL/min/1.73 m²), (4) total serum cholesterol (mmol/L), (5) systolic blood pressure (mm Hg) and drug use of (6) insulin for (7) statins.

Some candidate predictors like the Early Worsening, high non-attendance rate, pregnancy and visual acuity were not previously available in the IMRD database used to develop the model but will be collected from the three trusts. Their definitions are in online supplemental appendices 2 and 3.

DRPTVL-UK model

We previously developed and internally validated the DRPTVL-UK for the purpose of predicting the risk of vision loss and blindness or need for treatment in patients with referable DR. The model had good discrimination (C-statistic=0.74) and very little optimism in the internal validation. The model22 was developed in a primary care population and now needs external validation in the HES/surveillance clinic population, in which it is intended to be used.

The DRPTVL-UK model was developed using Cox regression and later refitted using a flexible parametric approach to obtain the baseline hazard function over time. The model can be used to predict the absolute risk of progression from referable DR to treatment or vision loss occurring within a 2-year period, based on an individual’s risk factor values. Thorough evaluation of the model’s external validity and net benefit is now required to establish whether the model is suitable for use in clinical practice in HES/surveillance clinics.

The DRPTVL-UK model can also be used to predict the time at which an individual reaches a particular risk threshold (to be agreed in a consensus meeting of clinical experts and patients planned after final analysis) which may be useful for determining appropriate follow-up intervals after referral to HES/surveillance clinics. This will be evaluated as a secondary objective.

External validation of the DRPTVL-UK model

The DRPTVL-UK model will be used to obtain the predicted probability of the outcome over time for every participant within each of the three trusts. Predictive performance of the model will be assessed using measures of discrimination (Harrell’s C-statistic and time-dependent C-statistic), calibration (calibration slope, ratio of observed to expected probabilities, calibration plots at multiple time points up to 2 years) and net benefit using decision curves. Performance measures will be calculated within each hospital and then pooled on an appropriate scale using random effects meta-analysis to account for clustering by hospital.26

As another potential use of the model would be to determine appropriate follow-up intervals based on the individual’s risk, it will also be crucial to ensure that the model performs well for predictions at all time points up to 2 years to ensure risk predictions are accurate over time. Therefore, we will also evaluate calibration performance at multiple time points. In addition to this, we will look at the predictions over time (predicted survival curves) and compare these to the observed survival curves for risk groups and other meaningful groupings, for example, DR grade.

In addition to external validation of the model in the whole sample, we will also validate the model within the subgroups of R2/M1 patients to see how well it performs in each.

Summarising baseline variables

Baseline variables will include predictors measured at referral or shortly before (see table 1 for full list of variables to be collected). Continuous variables will be described using means and SDs (prior to centring and standardisation), binary or categorical variables will be described using frequencies and proportions.

Model recalibration and updating

If necessary, we will recalibrate the model to the HES population (eg, by updating the baseline survival function or recalibrating the linear predictor).27 In addition, we will investigate whether updating the model to include additional predictors that were not available in the development dataset improves the predictive performance. Visual acuity, early worsening, pregnancy and frequent ‘did not attend’, were identified as candidate predictors based on expert opinion and evidence evaluation.21 To update the model with additional predictors, flexible parametric models (Royston-Parmar models) will be fitted using a multivariable fractional polynomial approach to consider non-linear functions for continuous variables while using backward elimination for the additional predictors considered.28 We will use a p>0.157 as a proxy for selection based on Akaike information criterion.29 All predictors from the original model will be forced to remain in the model regardless of statistical significance, therefore, only the four additional variables will be tested. The predictive performance of the updated model will be evaluated using internal–external cross-validation30 in which the model is developed using the data from two hospitals and externally validated in the third. This is then repeated a total of three times, each time reserving a different hospital for external validation. Predictive performance will be evaluated in each ‘external’ hospital using the same measures of discrimination and calibration as previously described and will be summarised across the hospitals using random-effects meta-analysis. Predictive performance of the updated model will be compared with the original model.

Clinical benefit

We will also evaluate the clinical benefit of the model using decision curve analysis in which the net benefit of using the model at different threshold probabilities is plotted and compared with strategies of intervention for all (following up everyone more frequently) or intervention for no-one (no-one followed up more frequently).31

Missing data

Missing data is a common problem in clinical data and needs to be appropriately accounted for in analyses. An audit using hospital notes from Sunderland Eye Infirmary showed physical examination variables of systolic and diastolic blood pressure nearest to referral were recorded in the clinical notes of 72% of patients; biochemical variables of HbA1c were recorded for 83% of patients, eGFR and cholesterol in 95.5% of patients, measured near to referral. In each case, we will consider why the values might be missing to understand whether a missing at random assumption is reasonable or whether missingness is likely to be informative. For variables missing for <40% of patients, missing data will be handled by multiple imputation using chained equations assuming data are missing at random. The missing at random assumption is an untestable one but data checks comparing characteristics of patients with missing values to those without will be performed to assess if there are any obvious problems with the assumption. To preserve any clustering that may be present, data will be imputed for each hospital separately. The imputation model will include all predictors as well as the outcome using the event indicator and estimate of the cumulative hazard function. Auxiliary variables will be considered to improve the missing at random assumption. The number of imputed datasets will be set at least equal to the percentage of observations of missing data for any of the variables of interest.32

Clinical consensus

Alongside the statistical analysis, clinical consensus workshops will take place to help determine clinically meaningful threshold probabilities for net benefit analysis and for use of the model in practice. This will include discussion and agreement on a suitable threshold for identifying higher risk patients and potential thresholds for determining the follow-up intervals, prior to decision curve analysis. After the results of the external validation including the decision curve analysis are available, they will be presented to the ophthalmic expert committee panel for discussion on how the model can be implemented.

The consensus process was first used in the USA in the early 1970s to address the National Institutes of Health development programme to seek agreement on the safety and efficacy of medical procedures, drugs and devices.33 Consensus development meetings were introduced to the UK health system to discuss healthcare policies and its implementation in clinical practice.34

The consensus process will be used in this expert group to reduce the range of potential options presented to facilitate joint decision making by the group on the most appropriate choice of the model implementation strategies. The consensus process will help us evaluate the list of options and combine them if an overlap is noted between different options. It can also accommodate the inclusion of further options, check for redundancy between included options and reach agreement through sharing information and knowledge of the participants.35 The consensus process described below also enhances the critical thinking of the key stakeholders and facilitates joint decision making of the diverse groups.36 Communication and cooperation between participants are key to reach successful agreement on the options discussed and to increase the chances of wider acceptance for implementation.37 Here, we aim to reach an agreement on participants’ opinions on the various options under consideration.

In this study, participants will be asked to rate the importance of each of the options based on a nine-point Likert scale that has been adopted in the COMET consensus style; (1–3=less important, 4–6=important and 7–9=critical) using a 70% threshold agreement to score the quality of evidence for outcomes in systematic reviews, and has been adopted in other core outcome development research groups using Delphi methods.38 Therefore, participants will be asked to vote on whether particular risk groupings should be considered for the final DRPTVL-UK model, excluded or require further discussion and refinement. For each option presented, the proportion of participants scoring 1–3, 4–6 and 7–9 on the nine-point Likert scale will be calculated for each item. ‘Consensus in’ will be defined as greater than 70% of participants scoring as 7–9. ‘Consensus out’ is based on an item being scored 1–3 by more than 70%. No consensus is based on an item where the level of importance was not decided due to uncertainty.39 We anticipate that the group joining the consensus process will include 10–15 participants, ensuring an appropriate balance of representation of the different stake holder groups.

Software

All statistical analyses will be done using Stata V.16 and R V.4.1 or later versions.

Ethics and dissemination

For the external validation, routine practice retrospective data including the lists of all NHS numbers of referrals for the years 2013–2016 will be kept on NHS trusts’ R & D secure computers in lockable rooms. Population characteristics like age at baseline/date of birth for age range, date of death for competing risk analysis will be extracted, but after calculation of age, gender distribution and lengths of follow-up durations, all the personal identifiable information will be removed to protect the patient privacy/confidentiality. Data extractors will be clinical nurse specialists employed by the NHS trusts’ R & D. No sensitive data will be extracted. Identifiable data will be removed from data extracted and only anonymised data will be sent to University of Birmingham (UoB) in encrypted fashion. This anonymised data will be stored on secure UoB computers with appropriate access controls, in lockable rooms.

Personal identifiable data will be deleted using deletion software at the time of study end by the contributing NHS trusts. Small number suppression (≤7, as per NHS Digital practice for Hospital Episode Statistics Admitted Patient Care data) will apply.40

For the consensus process, write ups will not mention any direct quotes, the NHS trust or any individual expert’s identity. This anonymised data will be stored on secure NHS/UoB computers for 10 years to allow for all possible publications.

The plans for dissemination include peer reviewed publication, presentation to professional/PPIE bodies and development of an electronic calculator application to allow risk-based prioritisation of their follow-up, after direct entry of clinical information. However, further research may be required to assess the clinical and economic impact of the final model. Social media will also be used to disseminate findings.