Article Text
PDF
Abstract
Objectives Our primary objective was to compare the utility of the Deyo-Charlson Comorbidity Index (DCCI) and Elixhauser-van Walraven Comorbidity Index (EVCI) to predict mortality in intensive care unit (ICU) patients.
Setting Observational study of 2 tertiary academic centres located in Boston, Massachusetts.
Participants The study cohort consisted of 59 816 patients from admitted to 12 ICUs between January 2007 and December 2012.
Primary and secondary outcome For the primary analysis, receiver operator characteristic curves were constructed for mortality at 30, 90, 180, and 365 days using the DCCI as well as EVCI, and the areas under the curve (AUCs) were compared. Subgroup analyses were performed within different types of ICUs. Logistic regression was used to add age, race and sex into the model to determine if there was any improvement in discrimination.
Results At 30 days, the AUC for DCCI versus EVCI was 0.65 (95% CI 0.65 to 0.67) vs 0.66 (95% CI 0.65 to 0.66), p=0.02. Discrimination improved at 365 days for both indices (AUC for DCCI 0.72 (95% CI 0.71 to 0.72) vs AUC for EVCI 0.72 (95% CI 0.72 to 0.72), p=0.46). The DCCI and EVCI performed similarly across ICUs at all time points, with the exception of the neurosciences ICU, where the DCCI was superior to EVCI at all time points (1-year mortality: AUC 0.73 (95% CI 0.72 to 0.74) vs 0.68 (95% CI 0.67 to 0.70), p=0.005). The addition of basic demographic information did not change the results at any of the assessed time points.
Conclusions The DCCI and EVCI were comparable at predicting mortality in critically ill patients. The predictive ability of both indices increased when assessing long-term outcomes. Addition of demographic data to both indices did not affect the predictive utility of these indices. Further studies are needed to validate our findings and to determine the utility of these indices in clinical practice.
- EPIDEMIOLOGY
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
There have been few studies examining the utility of comorbidity indices as predictors of mortality in patients admitted to the intensive care unit.
This study uses a large multi-institution cohort of critically ill patients admitted to several different intensive care units.
The results may not be generalisable to other institutions where coding practices may differ.
There is a chance of misclassification of vital status; however, there is no reason to suspect that this was differential between the two indices studied.
Introduction
Utilisation of critical care resources has continued to escalate over the past several years, despite attempts to reduce healthcare expenditures.1 ,2 The availability of reliable outcome predictors of critically ill patients may help improve the value of care. Providing information on expected mortality may help patients and relatives make informed decisions about their goals of care. In addition, comorbidity indices have become an important tool for policymakers, administrators and researchers to predict population-based outcomes.3
In the general population, two of the most commonly used indices are the Deyo-Charlson Comorbidity Index4 (DCCI) and the Elixhauser-van Walraven Comorbidity Index5 (EVCI). Originally proposed by Charlson et al6 in 1987, and then modified in 1992, the DCCI assigns a score to various chronic medical conditions and uses the sum to predict long-term mortality. Similarly, the EVCI score is based on 30 acute and chronic comorbidities to predict in-hospital mortality.7 In 2009, the Elixhauser score was modified by van Walraven et al5 into a weighted point system for streamlined use.
In contrast to scoring modalities such as the Acute Physiology and Chronic Health Evaluation (APACHE) and Sequential Organ Failure Assessment (SOFA), the DCCI and EVCI can be calculated at the time of intensive care unit (ICU) admission and do not require the interpretation of laboratory and bedside clinical data. Thus, they can be easily derived from administrative databases and their use in the critical care literature has been increasing. Both the DCCI and EVCI have been widely used to predict survival in acute hospital settings.8–11 There have been several studies demonstrating the validity of these indices to predict in-hospital mortality in ICU patients.12–16 Recent evidence suggests that the EVCI may be modestly superior to the DCCI for predicting mortality in hospitalised patients.17–20 However to date, there are few to no studies comparing DCCI to EVCI for use in critically ill patients when examining long-term mortality across a variety of types of ICUs. Our primary objective was to evaluate and compare the DCCI to EVCI for predicting mortality at 30, 60, 180, 365 days. Our secondary objectives were to: (1) compare DCCI and EVCI across medical and surgical ICU settings; (2) determine if adding three readily available patient characteristics (age, race and sex) to the DCCI and EVCI would improve prediction of mortality; and (3) to calculate the number of patients who would be reclassified based on predetermined mortality risk levels depending on which index was used.
Methods
Setting
We performed an observational study of ICU patients from two teaching hospitals in Boston, Massachusetts—Massachusetts General Hospital and Brigham and Women's Hospital. Massachusetts General Hospital is a 1052-bed hospital with burn, cardiac surgery, coronary, medical, neonatal, neuroscience, paediatric and surgical ICUs. Brigham and Women's Hospital is a 793-bed hospital with burn, cardiothoracic, coronary, medical, neonatal, neuroscience and surgical ICUs.
Data source and study population
We performed an electronic chart review of all admissions to the ICUs at the two study centres. We included patients 18 years and older, admitted to any adult ICU from 1 January 2007 to 3 December 2012. In-hospital mortality data were captured via the medical record and mortality after discharge was obtained using the Social Security Administration's Death Master File. We excluded foreign nationals since they did not have social security numbers, and therefore mortality could not be verified in the Social Security Administration's Death Master File. For patients with multiple admissions to the ICU over the study time period, only the first admission to the ICU during the study time period was included in the analysis. Patients under the age of 18 were excluded (n=1206), as were patients with coding errors that prevented the calculation of either index (n=162).
Data collection and processing
We abstracted demographic and pertinent clinical information on all patients who met inclusion/exclusion criteria using a system-wide (including both hospitals) longitudinal patient data registry. Specific elements included were: (1) age; (2) sex; (3) race; (4) ICU type; (5) International Classification of Disease Clinical Modification 9 (ICD-9) discharge diagnosis codes; (6) hospital admission and discharge dates; (7) ICU admission and discharge dates; and (8) date of death.
The EVCI and DCCI were calculated using ICD-9 codes based on published algorithms.21 Race and ethnicities were coded into Caucasian, African-American, Hispanic, Asian and Unknown/Other based on self-reported data. Age was treated as a continuous variable, and a sensitivity analysis was performed to ensure that there was no difference in model discrimination when age was included as 10-year categories. Time to death was calculated from the ICU admission date.
Statistical analysis
For both the DCCI and EVCI, receiver operator characteristic (ROC) curves were constructed to generate the areas under the curve (AUCs) for predicting 30, 60, 90, 180, and 365-day mortality following ICU admission. To test differences in the discriminative ability of the indices, the AUCs were compared using a previously described non-parametric approach.22 This analysis was repeated for each type of ICU. For the primary analysis, each index was compared with the other both as the sole predictor as well as with age, sex and race included in the model. These factors were selected since they are typically easily available in administrative databases.
Given that the implications of differences in AUCs can be difficult to interpret, we set to calculate how many patients would undergo reclassification of mortality risk depending on which index score was used. The probability of mortality for an individual patient was calculated using a logistic regression model with DCCI as the sole independent covariate. Patients were then placed into predefined categories of predicted risk of mortality (0–4.9%, 5–9.9%, 10–14.9% and ≥15%). The patients were then reclassified into a higher, equal or lower risk classification based on their EVCI score as calculated from a separate logistic regression. The net reclassification improvement was calculated to determine whether this reclassification was appropriate.23
We also performed several additional analyses after the results of the above described analysis were obtained. To further understand the calibration of the models, calibration plots were created for each index with 1-year mortality as the outcome. We also constructed time-dependent ROC curves by modelling our data as a survival data set and assessed the AUC at the same time points as the primary analysis. Finally, we examined the discriminatory ability of the indices to predict prolonged length of stay in both the ICU and the hospital. These were defined as having a length of stay greater than the 75th centile, which was 5 days for the ICU and 16 days for the hospital.
The creation of the ROC curves and their analyses was performed in STATA V.12 (StataCorp, College Station, Texas, USA). The remainder of the analysis was performed in R V.3.2.0 (R Foundation for Statistical Computing, Vienna, Austria). Specifically the reclassification analysis and the calibration plots were created using the PredictABEL package, and the time-dependent ROC curves were constructed using the timeROC package. All tests were performed with a two-tailed statistical level of significance set at p<0.05.
Results
Between January 2007 and December 2012, a total of 59 816 individual patients from 12 ICUs at two study hospitals met our study criteria (table 1). ICUs with the highest percentage of admissions included the cardiothoracic surgical ICU (23%), coronary care units, neurological/neurosurgical ICU (22%), medical ICU (19%) and surgical ICU (18%). Patients had a mean age of 64 (SD 16) and 81% of the population was Caucasian. The average DCCI score was 7 (SD 4) and the average EVCI was 16 (SD 12). Overall mortality at 30, 90, 180 and 365 days was 11%, 16%, 20% and 24%, respectively. The specific frequency of each comorbidity used to calculate the respective indices is presented in table 2.
Demographic data
Frequency of individual constituents of each comorbidity index
Predictors of mortality
As a predictor of 30-day mortality, the AUC for the DCCI was 0.65 (95% CI 0.65 to 0.67) while AUC for the EVCI was 0.66 (95% CI 0.65 to 0.66), p=0.02. The AUCs increased as the time window for assessing mortality was lengthened. The AUC for 1-year mortality was 0.72 for both indices (AUC for DCCI 0.72 (95% CI 0.71 to 0.72) vs AUC for EVCI 0.72 (95% CI 0.72 to 0.72), p=0.46). Comparisons of AUCs for all study time points can be found in table 3.
Comparison of area under receiver operator characteristic curves for the Charlson and Elixhauser Comorbidity Indices as predictors of mortality at various time points
Prediction by type of ICU
We compared the DDCI and EVCI within different subtypes of ICUs. Overall, the two indices predicted mortality similarly within each ICU type across different time periods. The exception to this was the neurosciences ICU where the DCCI was superior to the EVCI (1-year mortality AUC 0.73 (95% CI 0.72 to 0.74) vs 0.68 (95% CI 0.67 to 0.70), p=0.005). Table 4 lists the AUCs for 30-day and 365-day mortality across ICU types.
Comparison of area under receiver operator characteristic curves for the Charlson and Elixhauser comorbidity indices as predictors of mortality for different intensive care units
Addition of age, race, sex to the DCCI and EVCI
There was minimal improvement in discrimination when adding the comorbidity indices to basic demographic information (ie, age, race and sex) for both the DCCI and EVCI scores. The AUC for DCCI with demographic variables added was 0.67 (95% CI 0.66 to 0.67) for the prediction of 30-day mortality versus 0.65 (95% CI 0.65 to 0.66) for the index alone (p<0.001). The AUC for the EVCI score with demographics included was 0.73 (95% CI 0.73 to 0.74) for 1-year mortality compared with 0.72 (95% CI 0.71 to 0.72) for EVCI alone (p<0.001). Table 5 lists the various combinations and compares the AUCs to a prediction model with just demographic information. All of the indices and combinations with demographic information performed better than demographic variables alone with all p<0.001.
Areas under the curve for variation combinations of comorbidity indices and demographic data
Reclassification
The reclassification data highlight the differences between the DCCI and EVCI, showing the number of patients requiring reclassification when using a different index for predicting mortality (see table 6). When patients were divided into four categories of mortality likelihood, there were a total of 26 388 (44%) patients who required reclassification for their 30-day mortality risk. The net reclassification improvement for 30-day mortality risk was 7.5% (95% CI 6.8% to 9.1%) favouring the EVCI. This means that compared with individuals without the outcome, individuals with the outcome were 7.5% more likely to move up a risk category than down. When examining 1-year mortality risk, there were a total of 18 445 (31%) patients that required reclassification, with a net reclassification improvement of 4.4% (95% CI 3.5% to 5.2%).
Reclassification data based on predicted mortality
Additional analyses
As further sensitivity analysis, model calibration of the indices was assessed through the plotting of calibration plots with the outcome of 1-year mortality. The plots are displayed in figure 1 and demonstrate adequate calibration. When analysing the data as a survival data set, the results of the calculated AUCs from the time-dependent ROC curves were similar to the primary analysis. The AUCs across time points are displayed in figure 2. When examining the ability of both indices to predict prolonged length of stay, the EVCI outperformed for the DCCI. For prolonged ICU stay, EVCI had an AUC of 0.64 (95% CI 0.63 to 0.64) versus the DCCI that had an AUC of 0.53 (95% CI 0.53 to 0.54) with a p value for difference of <0.001. When examining the ability to predict a hospital stay of greater than 16 days, EVCI had an AUC of 0.67 (95% CI 0.67 to 0.68) versus the DCCI which had an AUC of 0.57 (95% CI 0.56 to 0.57) with a p value for difference of <0.001.
Calibration plots for Elixhauser-van Walraven Comorbidity Index and Deyo-Charlson Comorbidity Index for prediction of 1-year mortality after admission to the intensive care unit.
Calculation of areas under the curve from time-dependent receiver operator characteristic curves for one-year mortality.
Discussion
In our large cohort of ICU admissions, we found that the DCCI or EVCI were similar at predicting mortality at various time intervals and that the predictive ability of both indices increased when examining long-term outcomes. These results were consistent across various types of ICUs, with the exception of the neurosciences ICU where the DCCI was found to be superior. Consistent with a previous study,17 the indices performed better than demographic data alone. However, the addition of demographic data to both indices did not meaningfully affect the predictive utility of these indices.
Previous studies have shown that the EVCI, in general, appears to predict short-term and long-term mortality better than the DCCI for a variety of patients requiring acute hospitalisation.18–20
While the EVCI was statistically superior to the DCCI at predicting mortality at 30, 90 and 180 days, the difference was small and likely reached significance due to the large sample size of the study. When dividing the cohort into predicted risk categories, a large proportion of the population was reclassified into different categories depending on which index was used. However, the net reclassification index for 30-day mortality was only 7.5%. This suggests that while the EVCI may offer some advantage over the DCCI, the majority of this reclassification was not informative.
We did detect a difference between the two indices in the subgroup of patients admitted to the neurosciences ICU. While the reason for this observation is unclear, it may be due to the DCCI's heavy point allocation to metastatic solid tumours (6 points) and the additional points given to dementia (1 point) and cerebrovascular disease (1 point). The EVCI is less clear with regard to neurological diseases, allocating points to paralysis, ‘other neurological disorders’ and metastatic cancer.
Another interesting finding was that both indices improved mortality prediction over time. The AUC for the DCCI improved from 0.65 at 30 days to 0.72 at 365 days; and similarly, the AUC for EVCI improved from 0.66 at 30 days to 0.72 at 365 days. This improvement most likely is because as critically ill patients survive past their ICU stay, the major determinants of survival are related to their baseline comorbidities, age and new organ dysfunction as a result of their critical illness. Without prospective and organ dysfunction data, this is only a theory but two previous studies are congruent with our findings, where the researchers found that patients with higher organ dysfunction scores during their ICU stay had higher utilisation of healthcare resources and mortality up to 1 year, and possibly 5 years.24 ,25
Our study examined the real-world need of risk prediction with large administrative and claims databases for ICU patients. While physiology-based severity scores may have better predictive capabilities for short-term and long-term mortality, they are difficult to calculate without extensive resources. Additionally, these measures may not be available in large, historical cohorts of critically ill patients. In this study, we were not able to compare the DCCI and EVCI to commonly used physiology-based severity of illness scores. However, a previous study found that the combined DCCI with administrative variables performed similarly to physiology-based measures.14 Our results support this finding that a risk predictor based primarily on administrative data may be sufficient to serve as risk prediction tools in future studies, especially those examining long-term outcomes. Therefore, a lack of physiology-based severity scores does not preclude adequate risk adjustment of patients in studies of ICU patients using administrative databases. Given that both indices were similar in their discriminatory abilities, the DCCI may be more advantageous to use since it requires less data points to calculate it.
Our study has several limitations inherent to its design. First, calculation of these comorbidity indices is reliant on ICD-9 coding. This shortcoming was evident by the fact that none of the patients within our cohort had codes for a neurodegenerative disorder when calculating the EVCI. However, these coding discrepancies would likely be present when applying these metrics for non-research purposes, and thus the findings are representative of the ‘real-world’ accuracy of these indices. There is a risk of misclassification when assessing vital status; however, we have no reason to suspect that this risk was differential for either measure. Our cohort consisted of patients at two large academic hospitals, and thus the results may not be generalisable to other settings where coding practices may differ.
Conclusion
Our study found that the DCCI and EVCI were similar at predicting mortality in patients admitted to the ICU. Both indices demonstrated an improvement in discrimination as the time window of interest was lengthened. Given the policy, research and administrative implications of interpreting population data, selecting an accurate, yet practical comorbidity index for risk prediction is crucial. While a physiology-based comorbidity index is preferred for mortality risk assessment in ICU patients, our study suggests that either the DCCI or EVCI may be appropriate for predicting long-term mortality when physiology-based indices are not readily available. Further studies are needed to determine if the DCCI or EVCI can be used in place of physiology-based risk indices.
References
Footnotes
Contributors KSL and JL contributed to the design and conception of the study, the analysis and interpretation of data, and drafting of the work. KZ contributed to the analysis and interpretation of data, and drafting of the work. SAQ, TK and RS contributed to the design and conception of the study, the interpretation of data, and drafting of the work. ENK contributed to the acquisition of data for the study and drafting of the work. All authors revised the work critically for important intellectual content.
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests SAQ has received within the past 2 years investigator-initiated research funding from the US National Institutes of Health (grants L30 GM102903, L30 TR001257 and T32 GM007592) and Harvard Medical School. TK has received within the past 2 years investigator-initiated research funding from the French National Research Agency and the US National Institutes of Health. Further, he has received honoraria from the BMJ and Cephalalgia for editorial services. ME has received within the past 2 years investigator-initiated research funding from Masimo, MERCK, the ResMed Foundation, and the Buzen’s Foundation. Further, he has received honoraria from the journal Anesthesiology for editorial services.
Ethics approval Partners Institutional Review Board.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.