Abstract
Background Chronic kidney disease is a major health concern that, if left untreated, may progress to end-stage kidney failure (ESKF). Identifying individuals at an increased risk of kidney disease and who might benefit from a therapeutic or preventive intervention is an important challenge.
Aim To evaluate the performance of the QKidney® scores for predicting 5-year risk of developing moderate-severe kidney disease and ESKF in an independent UK cohort of patients from general practice records.
Design and setting Prospective cohort study to evaluate the performance of two risk scores for kidney disease in 364 practices from the UK, contributing to The Health Improvement Network (THIN) database.
Method Data were obtained from 1.6 million patients registered with a general practice surgery between 1 January 2002 and 1 July 2008, aged 35–74 years, with 43 186 incident cases of moderate-severe kidney disease and 2663 incident cases of ESKF. This is the first recorded evidence of moderate-severe chronic kidney and ESKF as recorded in general practice records.
Results The results from this independent and external validation of QKidney scores indicate that both scores showed good performance data for both moderate-severe kidney disease and ESKF, on a large cohort of general practice patients. Discrimination and calibration statistics were better for models including serum creatinine; however, there were considerable amounts of missing data for serum creatinine. QKidney scores both with and without serum creatinine were well calibrated.
Conclusion QKidney scores have been shown to be useful tools to predict the 5-year risk of moderate-severe kidney disease and ESKF in the UK.
INTRODUCTION
Chronic kidney disease (CKD) is a major health concern that, if left untreated, may progress to end-stage kidney failure (ESKF). While end-stage renal failure is relatively rare, the treatment, namely dialysis or transplantation, is expensive: in the UK, 2% of the total NHS budget is spent on renal replacement therapy.1 Furthermore, those with CKD are at a greater risk of dying from cardiovascular disease than progressing to ESKF.2–5 Identifying individuals who are at an increased risk of kidney disease, and who might benefit from a therapeutic or preventive intervention, is an important challenge. QKidney® scores are sex-specific multivariable risk scores that have recently been developed to predict the 5-year risk of moderate-severe CKD and the 5-year risk of developing ESKF in a primary care population.6
The risk scores were developed and validated on a large cohort of patients (2.4 million) from the QResearch® (http://www.qresearch.org) database; two-thirds of the cohort were randomly allocated for model development and one-third to model validation. The QResearch database is a large database comprising over 12 million anonymised health records from 602 practices throughout the UK, using the Egton Medical Information Systems (EMIS) computer system (used in 59% of general practices in England). QKidney scores were developed on 1.6 million patients aged 35–74 years, contributing 8 million person-years of observation, with 41 119 incident cases of moderate-severe CKD, and 2800 incident cases of ESKF between 1 January 2002 and 31 December 2008. QKidney scores were derived using a Cox proportional hazards model. Fractional polynomials were used to model non-linear risk relationships with continuous risk predictors, and the presence of interactions between risk predictors was tested.7 Multiple imputation was used to replace missing values for key risk predictors (body mass index, systolic blood pressure, smoking status, and serum creatinine), to reduce the biases that can occur in a ‘complete-case’ analysis.8,9 The final prediction models included 17 risk predictors for females and 15 risk predictors for males (Box 1). The open source code to calculate the QKidney scores is available from http://www.qkidney.org, released under the GNU Lesser General Public Licence (version 3).
Box 1. Summary of risk predictors in QKidney® scores
Predictors for males and females
Age (years)
Body mass index (kg/m2)
Self-assigned ethnicity (white/not recorded, Indian, Pakistani, Bangladeshi, other Asian, black African, black Caribbean, other included mixed)
Townsend deprivation score (output area level 2001 census data evaluated as a continuous variable)
Smoking status (non-smoker, ex-smoker, light smoker (<10 cigarettes/day), moderate smoker (10–19 cigarettes/day), heavy smoker (≥20 cigarettes/day)
Systolic blood pressure (mmHg)
Diagnosis of type 1 diabetes (yes/no)
Diagnosis of type 2 diabetes (yes/no)
Diagnosis of cardiovascular disease (yes/no)
Diagnosis of congestive cardiac failure (yes/no)
Diagnosis of peripheral vascular disease (yes/no)
Diagnosis of rheumatoid arthritis(yes/no)
Treated hypertension (yes/no)
Two or more prescriptions of non-steroidal anti-inflammatory drugs 6 months before study entry (yes/no)
Recorded family history of kidney disease including polycystic kidneys (yes/no)
Serum creatinine (µmol/litre)
Additional predictors for females
Diagnosis of systemic lupus erythematosus (yes/no)
Evidence of kidney stones based on diagnosis or operative procedure at baseline (yes/no)
This article describes the results from an independent evaluation assessing the performance of QKidney scores on a large dataset of general practice records in the UK. Due to concerns raised during the peer review process of using serum creatinine as a predictor of kidney disease, the authors of QKidney scores omitted it from the models.10 This study thus sought to examine the added value and performance of including serum creatinine in the current QKidney scores.
METHOD
Cohort selection
Study participants were patients registered between 1 January 2002 and 1 July 2008 and recorded on The Health Improvement Network (THIN) database (http://www.thin-uk.com). Patients were eligible if they were aged 35–74 years, had no previously recorded evidence of CKD, were permanent residents in the UK, and had no interrupted periods of registration with a practice.
How this fits in
Chronic kidney disease (CKD) is a major health concern that, if left untreated, may progress to end-stage kidney failure (ESKF). While end-stage renal failure is relatively rare, the treatment, namely dialysis or transplantation, is expensive: in the UK, 2% of the total NHS budget is spent on renal replacement therapy. A new risk score has recently been developed to predict the 5-year risk of moderate-severe CKD and the 5-year risk of developing ESKF in a primary care population. The risk score has good discrimination and calibration and could be integrated into clinical computer systems to identify individuals who are at increased risk of developing moderate-severe CKD and ESKF
Outcome measures
The two primary outcomes are recorded evidence of moderate-severe CKD and ESKF. Moderate-severe CKD was defined as the first occurrence of any of the following during follow-up: recorded kidney transplant, recorded kidney dialysis, recorded diagnosis of nephropathy, glomerular filtration rate <45 ml/min/1.73 m2, corresponding to stage 3B CKD,11 or recorded diagnosis of proteinuria. ESKF was defined as the first occurrence of any of the following during follow-up: recorded kidney transplant, recorded kidney dialysis, or glomerular filtration rate <15 ml/min/1.73 m2, corresponding to stage 5 CKD.11 Glomerular filtration rates were calculated using the MDRD (modification of diet in renal disease) equation, using laboratory-reported creatinine values.12
Statistical analysis
For each patient, an entry date was determined, which was the latest of the date of their 30th birthday, date of registration with the practice, date on which the practice computer system was installed plus 1 year, and the beginning of the study period (1 January 2002). Only patients with a minimum of 1 year's complete data in their medical record were included in the analysis. The observation time was calculated from the entry date to an exit date, which was defined as the earliest date of recorded CKD, date of death, date of deregistration with the practice, date of last upload of computerised data, or the study end date (1 July 2008).
Smoking status was derived from combining two risk factors; (1) whether the patient was a non-smoker, ex-smoker, or current smoker, and (2) the amount of cigarettes smoked, defined as light (<10 cigarettes/day), moderate (10–19 cigarettes/day), or heavy (≥20 cigarettes/day).
Multiple imputation using all predictors plus the outcome variable was used to replace missing values for smoking status, amount of cigarettes smoked, systolic blood pressure, body mass index, and serum creatinine. This involves creating multiple copies of the data and imputing the missing values with sensible values randomly selected from their predicted distribution. Twenty imputed datasets were generated and results from analyses on each of the imputed datasets were combined using Rubin's rules to produce estimates and confidence intervals that incorporate the uncertainty of imputed values.13 The 5-year estimated risk of CKD (moderate-severe CKD and ESKF) for every patient in the THIN cohort was calculated using QKidney scores (with and without serum creatinine).
The predictive performance of QKidney scores for the THIN cohort was assessed by examining measures of calibration and discrimination. Calibration refers to how closely the predicted 5-year CKD risk agrees with the observed 5-year CKD risk. This was assessed for each tenth of predicted risk, and each 5-year age band, by calculating the ratio of predicted to observed CKD risk, separately for males and for females. Calibration was assessed by plotting the observed proportions versus the predicted risk. Observed 5-year CKD risks were obtained using the method of Kaplan–Meier.
Discrimination is the ability of the risk score to differentiate between patients who experience a CKD event during the study and those who do not. This measure is quantified by calculating the area under the receiver operating characteristic curve (AUROC) statistic; a value of 0.5 represents chance and 1 represents perfect discrimination. The D statistic14 and R2 statistic15 were also calculated; these are measures of discrimination and explained variation respectively, and are tailored towards censored survival data. Higher values of D indicate greater discrimination, where an increase of 0.1 over other risk scores is a good indicator of improved prognostic separation.14
All statistical analyses were carried out in R (version 2.12.2), and the ICE procedure for multiple imputation in Stata (version 11.2).
RESULTS
Between 1 January 2002 and 1 July 2008, 1 593 506 eligible patients from 364 general practices in the UK were registered in the THIN database. For the moderate-severe kidney disease end-point, 1 581 745 patients (49.6% females) were eligible for analysis, contributing 7 549 339 person-years of observation, among whom there were 43 186 incident cases of moderate-severe kidney disease. For the ESKF end-point, 1 593 506 patients were eligible for analysis, contributing 7 708 730 person-years of observation, among whom there were 2663 incident cases of ESKF. The median follow-up for the moderate-severe kidney disease end-point was 6.21 years (interquartile range (IQR) = 3.06 to 6.49 years) and for the ESKF end-point was 6.41 years (IQR = 3.24 to 6.49 years). In total, 939 718 patients (59%) were followed up for 5 years or more for the moderate-severe kidney disease end-point and 970 083 patients (61%) for the ESKF end-point.
The 5-year observed risk of moderate-severe kidney disease in females (24 371 incident cases of moderate-severe kidney disease) aged between 35 and 74 years was 3.19 (95% confidence interval [CI] = 3.15% to 3.23%) and in males (18 815 incident cases of moderate-severe kidney disease) was 2.43% (95% CI = 2.39 % to 2.46%). The 5-year observed risk of ESKF in females (1171 incident cases of ESKF) aged 35–74 years was 0.15% (95% CI = 0.14% to 0.16%) and in males (1492 incident cases of ESKF) was 0.19% (95% CI = 0.18% to 0.20%). Table 1 details the characteristics of the patients in the THIN cohort.
Complete data on smoking status, smoking category, body mass index, and systolic blood pressure were available for 70.1% of females (n = 549 661) and 62.4% of males (n = 497 451) for the models without serum creatinine. Most patients (n = 1 415 575; 89.5%) had no, or only one, missing risk factor (Table 2). For the models containing serum creatinine, complete data on smoking status, smoking category, body mass index, systolic blood pressure, and serum creatinine were available for 10.6% of females (n = 83 906) and 9.24% of males (n = 73 713). For the moderate-severe kidney disease model including serum creatinine, 1 101 565 patients (69.6%) had no, or only one, missing risk factor (Table 2). There were considerably higher levels of missing data for serum creatinine, with only 227 156 patients (14.3%) having complete data recorded. Both males and females with serum creatinine recorded, tended to be older, have higher systolic blood pressure, have a diagnosis of type 2 diabetes or cardiovascular disease, and be treated for hypertension.
Table 3 shows the incidence rates (per 1000 person-years) for moderate-severe kidney disease and ESKF by age and sex in the THIN cohort. Females experienced 24 371 incident cases of moderate-severe kidney disease during 3 771 740 person-years of follow-up: incidence rate 6.46 per 1000 person-years (95% CI = 6.38 to 6.54). Males experienced 18 815 incident cases of moderate-severe kidney disease during 3 777 599 person-years of follow-up: incidence rate 4.98 per 1000 person-years (95% CI = 4.91 to 5.05). The rates of moderate-severe kidney disease were highest in those aged 70–74 years at baseline: incidence rate in females of 27.47 (95% CI = 26.91 to 28.04) and in males 24.86 (95% CI = 24.24 to 25.49) per 1000 person-years of observation. Overall, 1171 incident cases of ESKF were recorded for females during the 3 866 065 person-years of follow-up, giving an overall incidence rate of 0.30 per 1000 person-years (95% CI = 0.29 to 0.32) in females, whereas for males, 1492 incident cases of ESKF occurred during 3 842 665 person-years of follow-up, giving an incident rate of 0.39 per 1000 person-years (95% CI = 0.37 to 0.41)
Table 4 presents the performance data on discrimination and calibration for the QKidney scores in the THIN cohort. The R2 statistic (percentage of explained variation) was about 4% higher for the models predicting moderate-severe kidney disease including serum creatinine. For the models predicting ESKF, the R2 statistic was about 25% higher for the models including serum creatinine. The D statistic was noticeably higher for the models predicting ESKF including serum creatinine in both females and males (3.80 and 3.97, respectively) than for the models without serum creatinine (2.21 and 2.18, respectively). Values for the AUROC were comparable between females and males for the model predicting moderate-severe kidney disease without serum creatinine (both 0.875) and for the models predicting ESKF (0.833 and 0.841, respectively). Corresponding values for the models including serum creatinine were noticeably higher, especially for the models predicting ESKF, where the values were 0.948 and 0.954 for females and males respectively. The performance of the models, restricting the analyses to those with available data to judge the adequacy of missing data mechanism, shows clear differences for the ESKF model including serum creatinine, revealing a possible missing not-at-random mechanism. However, regardless of this, the results are noticeably higher both for those with available data and for those with multiple imputed approaches, indicating improved performance for the models including serum creatinine.
Figures 1 and 2 present the calibrations of QKidney scores for males and females by tenth of risk for the models with and without serum creatinine. Model calibration was good, with close agreement between the predicted and observed risk of moderate-severe kidney disease and ESKF across all tenths of risk, with no discernible over-prediction or under-prediction. Similarly, Figure 3 displays the calibration of QKidney scores for males and females by age group. Agreement between predicted and observed kidney disease risks was generally good across all age groups, with a slight over-prediction in the latter age groups for the ESKF model.
DISCUSSION
Summary
This independent evaluation of the performance of QKidney scores was carried out on the THIN database, based on general practices recording clinical data using the INPS Vision system, which is used in 20% of UK general practices. The dataset comprised 1.6 million patients registered between 1 January 2002 and 1 July 2008, contributing 7.5 million person-years of observation. The performance data on the THIN cohort presented in this article provide strong evidence to support the external validity of QKidney scores in predicting the 5-year risk of developing moderate-severe kidney disease and ESKF. The differences between observed and predicted risks for the model to predict moderate-severe kidney disease (both with and without serum creatinine) were negligible across age groups for males and females.
Strengths and limitations
A major strength of this study is the size and the representativeness of the cohort, achieved by including a large number of general practices using the EMIS computer system. Furthermore, the treatment of missing data was handled using the current recommended approaches to avoid the biases that occur when omitting individuals with incomplete data.8,16,17 Both models for moderate-severe kidney disease and ESKF with or without serum creatinine had impressive discrimination properties and were well calibrated to the setting. When considering which model to use in routine clinical practice, with or without serum creatinine, the likelihood of the availability of risk factors is a major feature for consideration. There were substantial amounts of missing data for serum creatinine that should be taken into consideration when deciding which model to use. It is likely that serum creatinine was predominantly recorded in those who were suspected of being at an increased risk of kidney disease and thus is not routinely collected in those who are at low or negligible risk.
Comparison with existing literature
To date, QKidney scores are the first multivariable prediction models to predict the risk of moderate-severe kidney disease and ESKF for use in UK primary care. Existing prediction models for kidney disease have been developed in the US,18 Canada,19 the Netherlands,18,20 and Taiwan;21 however, the performance data to support these models are, at best, comparable19,20 to the QKidney scores without serum creatinine. There are also no data to support their use in the UK. Furthermore, three of these prediction models include laboratory-based risk factors, making their use in routine clinical practice limited.19–21
Implications for practice and research
The performance data from this independent validation of QKidney scores support the case of using these scores to identify individuals who are at increased risk of developing moderate-severe CKD and ESKF. The scores without serum creatinine have the potential to be used in clinical practice, as they only require information that can be obtained during a routine general practice appointment. While the addition of serum creatinine improved the predictive performance of the models, use of the models including serum creatinine as a population-based screening tool are limited, due to the laboratory-measured serum creatinine.
This study has provided an independent and external evaluation of the QKidney scores on a large cohort of patients in the UK to predict moderate-severe chronic kidney disease and ESKF. It has assessed the performance of QKidney scores with and without the inclusion of serum creatinine and has provided good evidence to confirm the predictive power of the risk scores without the need for recalibration. Given the large amounts of missing data for serum creatinine, the model without serum creatinine is the preferred model.
Notes
Funding
This study was not funded.
Ethical approval
Trent Multicentre Research Ethics Committee.
Provenance
Freely submitted; externally peer reviewed.
Competing interests
The authors have declared no competing interests.
Discuss this article
Contribute and read comments about this article on the Discussion Forum: http://www.rcgp.org.uk/bjgp-discuss
- Received September 20, 2011.
- Revision received October 26, 2011.
- Accepted January 2, 2012.
- © British Journal of General Practice 2012