Article Text
PDF
Abstract
Objective Create a score to identify patients at risk of death or hospice placement who may benefit from goals of care discussion earlier in the hospitalisation.
Design Retrospective cohort study to develop a risk index using multivariable logistic regression.
Setting Two tertiary care hospitals in Southeastern Minnesota.
Participants 92 879 adult general care admissions (50% male, average age 60 years).
Primary and secondary outcome measures Our outcome measure was an aggregate of inhospital death or discharge to hospice. Predictor variables for the model encompassed comorbidities, nutrition status, functional status, demographics, fall risk, mental status, Charlson Comorbidity Index and acuity of illness on admission. Resuscitation status, race, geographic area of residence and marital status were added as covariates to account for confounding.
Results Inhospital mortality and discharge to hospice were rare, with incidences of 1.2% and 0.8%, respectively. The Hospital End-of-Life Prognostic Score (HELPS) demonstrated good discrimination (C-statistic=0.866 in derivation set and 0.834 in validation set). The patients with the highest 5% of scores had an 8% risk of the outcome measure, relative risk 12.9 (10.9–15.4) when compared to the bottom 95%.
Conclusions HELPS is able to identify patients with a high risk of inhospital death or need for hospice at discharge. These patients may benefit from early goals of care discussions.
- mortality
- hospice
- clinical prediction rule
- application
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
Large retrospective data set (92 000 hospitalisations included) allows for good statistical power.
Model includes readily available variables that allow for direct calculation from the electronic medical record.
Study included only patients from two large academic medical centres in the Midwestern USA, which may limit generalisability.
Introduction
According to the Dartmouth Atlas of Health Care, hospital deaths accounted for almost a quarter of all deaths in the USA in 2012.1 The quality of these hospital deaths is intimately related to an effective discussion of goals of care.2 3 Unfortunately, even among patients with cancer, this conversation often does not occur until the last few days of life.4 One of the pre-emptive steps to address this challenge is identifying patients with an increased risk for adverse outcomes and prioritising early discussion of patient preferences as well as directing a higher amount of surveillance resources to those individuals.5
Clinical support tools are often developed to assist the care team in identifying at-risk patients. Several scales have been published for use in predicting inpatient mortality in general care wards. These include automated scales that make use of patient data available on admission6 as well as simpler scales that require straightforward assessments of patients’ functional status.7 However, none have yet used the many scales that are already regularly evaluated for other risks. These include pressure ulcers, falls, mental status, comorbidity burden and physiological failure. Nurses currently spend approximately 36% of their work time in documentation,8 so simply making use of data already routinely collected would avoid adding a significant workload to health providers at hospital admission.
The objective was to develop a tool for early identification of patients for whom goals of care discussions would be helpful. This would consist of building a statistically optimised risk index using data readily available on admission from the electronic medical record, and assessing the model’s performance in identifying patients at risk of dying or going to hospice during the hospitalisation. Rather than limit generalisability by just focusing on elderly patients or those with a specific condition, this index would be developed for hospitalised adult patients of all ages and clinical conditions.
Methods
Study design and data sources
We performed a retrospective cohort study of patients admitted to Mayo Clinic Hospital, Methodist and Saint Marys campuses in Rochester, Minnesota, between 1 January 2010 and 31 December 2011. Our institution’s Institutional Review Board considered the retrospective study ethical and approved the retrospective data collection. The data were extracted from the Mayo Clinic electronic medical record. Patients were only included if they were more than 18 years old, had documented research authorisation and were hospitalised on a general care ward (defined as any non-intensive care unit hospital bed, excluding stays solely for research or rehabilitation purposes and psychiatric hospitalisations). Patients referred from the Federal Medical Centre, psychiatric admissions and patients admitted only for research purposes were excluded. No admissions were excluded due to missing information/data. For patients transferred between hospital care units, the stays were linked together and considered a single admission. For each admission, only the first measurement of each variable was used for analysis, thereby reflecting the initial condition on arrival. If a patient was initially admitted via the emergency room, another hospital, intensive care unit, operating room or skilled nursing facility before being admitted to a Mayo Clinic hospital, then the data represented the condition of the patient on arrival to our general care ward.
Outcome variables
Analysis was performed to create a risk index predicting the aggregated outcome of inhospital mortality and transfer to hospice care on discharge. Hospice care is provided at home or in an institution, and is focused almost exclusively on alleviating symptoms and facilitating a comfortable end of life, generally for patients with a life expectancy of only a few weeks or months. These outcome variables were chosen to represent worst-case discharge status and outcomes. The inclusion of hospice accounted for patients whose death was near and no life-sustaining treatments were being used.
Prediction variables
We searched the Mayo Clinic electronic medical record to find readily available, clinically relevant scales and comorbidities. We included variables representing several measures: mobility, nutrition, mental status, comorbidity burden, physiological failure, fall risk and activities of daily living (ADLs). These included the Braden Skin Score9 and its individual components of mobility, nutrition and activity; body mass index (BMI); Richmond Agitation–Sedation Scale (RASS)10; Charlson Comorbidity Index score11; Inpatient Physiologic Failure Score (IPFS)12; Hendrich II Fall Risk score and Get-Up-and-Go test13; level of ADL assistance required; number of caregivers needed to perform ADLs. Additionally, we chose to include other potential risk factors for mortality: age, gender, surgical/medical patient and individual comorbidities. All of the above measures were assessed as described by the original authors. Comorbidities were measured using previous medical history and diagnosis at admission to a general care ward and included: myocardial infarction, congestive heart failure, peripheral vascular disease, cerebrovascular disease, dementia, chronic pulmonary disease, ulcer, moderate or severe liver disease, diabetes, hemiplegia, renal disease, rheumatologic disease and malignancy. Charlson Comorbidity Index score represented a patient’s overall comorbidity burden, while individual comorbidities accounted for those with particularly high significance.
Covariates
Mortality is related to several other patient characteristics. To control for this potential confounding, we included marital status, race, geographic area of residence and ‘do not resuscitate’ (DNR) status. A patient’s area of residence, marital status and race controlled for some of the differences between patient populations. DNR status adjusted for patients who desired less aggressive treatment towards end of life. This allowed us to select predictor variables with the most significant explanatory power beyond that explained by those risk measures alone.
Multivariable model
Since our study was focused on discharge status, rather than time to event, we used logistic regression with split-sample validation to develop the statistically optimised risk indexes. A single model was fit to predict the aggregate outcome of inhospital mortality and discharge to hospice. Prior to creation of the multivariable model, univariable analysis was performed on each of the continuous variables to evaluate possible categorisation. This involved using simple logistic regression to analyse the variables’ relationships with the outcome, measured using the independent variable versus predicted risk graph and the x² test. Age was found to perform in a non-linear pattern and was therefore changed into a categorical variable with 5-year increments. All continuous variables were found to have a skewed distribution, so median (IQR) was used to report summary statistics. Before building the prediction tool, the full data set was randomly split into a 2/3 derivation and a 1/3 validation data set.
The multivariable model was created by first fitting a ‘parent’ model of covariates to the derivation cohort, and then adding predictor variables to that parent model. Covariates were fit with a forward stepwise approach to create the parent model. Predictor variables were then added to that parent model using a forward stepwise procedure. In order to reduce multicollinearity and simplify the model, only one variable measuring mobility was kept. Of these, only Braden mobility and activity remained after stepwise selection. To determine which to retain, we fit two separate models, each retaining only one of the two. The models were compared using the C-statistic, Akaike information criterion and number of missing values. Interactions were then added to the chosen model using a two-way stepwise procedure to create a final multivariable model.
For this derivation of the multivariable model, a consistent set of criteria was followed. Because of the high number of covariates and independent variables included (33), Bonferroni correction was needed to adjust the significance level for the stepwise regressions. This resulted in a cut-off P value of 0.001 (equivalent to α=0.033) for the full multivariable model and 0.0125 (equivalent to α=0.05) for the parent model. Each output model from the stepwise regressions was measured for multicollinearity using the variance inflation factor (VIF). Any variable with high multicollinearity (VIF >5) was dropped from further analysis. Any variable with missing values >5% was also dropped from the analysis. Because missing values were rare (<1% among variables included in the final model), simple listwise deletion was used to omit any admissions containing missing values. Any categorical variable in the final multivariable model with more than one statistically insignificant category was collapsed accordingly into fewer categories. Before converting the final model into risk indices, all covariates were dropped as they would not be used as predictors in the index.
Risk index
We used the methods described in the Framingham Risk Score Study14 to convert the final multivariable model of only predictor variables to a prediction index by assigning a number of points to each variable equal to its regression coefficient divided by the smallest regression coefficient in the model. The most common level for each categorical variable was chosen to be the reference category and assigned zero points. A patient’s final risk index was calculated by summing the points of the appropriate attributes. Another logistic regression model was run for the risk index to determine the logistic regression equation for predicting the corresponding outcome. Using this regression equation, we calculated the predicted risk of adverse outcome for each point total.
A logistic regression of the risk index was then run on the validation cohort, and the C-statistic was used to measure the index’s ability to predict inhospital mortality and discharge to hospice. The C-statistic expresses the discriminatory power of a model for distinguishing between events and non-events. For binary outcomes, such as those studied here, it is identical to the area under the receiver operating characteristic curve. Its maximum value is 1 (perfect predictions), with 0.5 corresponding to a model whose predictions are no better than chance. We measured the calibration of the prediction index by comparing the predicted and observed risks of adverse outcomes for each point total. Overall calibration was summarised using the Hosmer-Lemeshow goodness-of-fit test. SAS V.9.3 (SAS Institute, Cary, North Carolina, USA) was used to conduct all statistical analysis.
Results
Study population
The final data set contained 64 944 unique patients, with an average of 1.43 admissions each, for a total of 92 879 admissions. Table 1 presents a summary of the derivation and validation cohorts for the predictors and covariates included in the final model with a univariable analysis of each. The population was divided evenly between medical and surgical patients. The patients ranged in age from 18 to 108 years with a median (IQR) of 61 (47–73). There were equal numbers of men and women. Most patients were white (92.3%). Comorbidities were common, with 59.5% of patients having at least one. Patients had a median IPFS score of 9, indicating a moderate risk of mortality.12 The outcomes inhospital mortality and discharge to hospice were rare, with incidences of 1.2% and 0.8% of hospitalisations, respectively.
General characteristics of admissions by cohort
Risk index derivation
Only 10 of the original 29 independent variables remained in the prediction index after multivariable selection was completed. BMI and the level of assistance required to perform ADLs were both dropped due to missing values and Braden skin score was dropped due to high multicollinearity. Age was collapsed into just three categories for both outcomes. Figure 1 shows the ORs with 95 % CIs for the final multivariable models. Braden activity and nutrition, Hendrich II Fall Risk Score, abnormal RASS, IPFS , age, Charlson Comorbidity Index score, congestive heart failure, diabetes, liver disease and malignancy were all included in the final model. Additionally, several statistically significant interactions were found among surgical patient, Charlson Comorbidity Index score, Braden activity, malignancy, Braden nutrition, liver disease, RASS and hemiplegia. All P values were highly significant (highest P value= 0.0006), so any inflation of the type I error due to the lack of adjustment for multiple admissions for the same patient (only 1.43 on average) would not have significantly affected the analysis.15 Figure 2 shows the scoring system for the point values assigned to each of the predictor variables.
ORs with 95% CIs for logistic regression prediction model. IPFS, Inpatient Physiologic Failure Score; RASS, Richmond Agitation Sedation Scale.
Hospital End-of-Life Prognostic Score predicting risk of inhospital mortality or discharge to hospicea.
Risk index assessment
Figure 3 displays the predicted and observed outcomes for the Hospital End-of-Life Prognostic Score (HELPS), as well as its distribution. HELPS ranged from −34 to 104, with a relatively normal distribution slightly skewed to the right (figure 3). Predicted risk of inhospital mortality or discharge to hospice ranged from 0% to 76% across all patients in the derivation and validation sets. The observed risk followed the predicted risk very closely for the majority of patients. Above a risk estimate of approximately 5%, observed outcomes displayed greater variation, mainly due to low sample sizes. The Hosmer-Lemeshow statistic had a P value of 0.0049. The significant P value may indicate poor calibration, but is possibly due to the high sensitivity of the Hosmer-Lemeshow test for large sample sizes.16
Predicted and observed risk of hospital mortality and discharge to hospice in validation set.
Table 2 presents a summary of HELPS in both the derivation and validation cohorts, unadjusted and adjusted for the significant covariates. HELPS had C-statistics of 0.866 and 0.834 in the derivation and validation sets. A one-point increase in HELPS increased the odds of inhospital mortality or discharge to hospice by 8%–9%.
Risk index prediction models
Discussion
Using only regularly evaluated measures and patient characteristics, the HELPS provided reasonable discrimination for identifying patients at risk of inhospital mortality or transfer to a hospice on discharge (C-statistic of 0.834). These patients would likely benefit from goals of care discussion early in their hospitalisation. This would allow the care team to tailor the care to a patient’s wishes from the outset. Use of HELPS would assist caregivers in determining where to focus more detailed discussions of patient preferences.
This compares favourably to the CARING score developed to identify patients on admission for palliative care consultations.17 HELPS has a similar purpose, but expands on the score with a larger and more generalisable population and incorporates several measures of risk. Other similar indices have been developed for inpatient mortality in general care wards. A model developed by Escobar et al makes use of physiology and diagnosis data to robustly predict inpatient mortality (C-statistic of 0.88), but requires calculation by a computer and does not predict transfers to hospice.6 The Patient- And Nutrition-Derived Outcome Risk Assessment (PANDORA) score combines demographics, clinical observation and nutrition to create a simple and easily applied measure of mortality risk (C-statistic of 0.79).7 HELPS has the advantage that it can be hand-calculated and makes use of the regularly measured risk scales for several aspects of functional decline, all of which are closely related to mortality.
Most risk scores predict only a single measure, such as mortality. Because our study focused on identifying patients who might benefit from goals of care discussions, we chose to include transfer to hospice on discharge as well as inhospital mortality to create an aggregate outcome. Mortality represented the most extreme outcome, presenting the greatest need for intervention or surveillance. Discharge to hospice represented the remaining patients who would continue to receive regular medical care as they approached end of life. This could also be considered as a surrogate for mortality. The prevalence of the two outcomes in our sample was similar to the reported prevalence in the US population: inhospital mortality was estimated by the Centers for Disease Control and Prevention to be 2% in 2010 and trending down {Hall18, 2013 #227} (1.5% in our sample); discharges from hospital to hospice are estimated to be 0.5% {Lin19, 2012 #228} (0.5% in our sample).
Nutritional status was one of the largest risk factors for hospice and inhospital mortality. This seems to be consistent with other studies20 21 focusing on the role of nutrition.
HELPS’s discrimination of high-risk patients positions it as an effective screening tool for patients in need of goals of care discussions. Randomised controlled trials showed that early consultation with palliative care (which focuses on instituting measures and treatments aimed at improving pain and other symptoms and problems usually associated with terminal illness) or advance care planning can improve patient and family satisfaction and reduce the cost of end-of-life care.22 23 Additionally, transition to palliative care or hospice could even increase life expectancy in some patients with advanced disease.24 25 A recent mortality review identified a lack of discussion or inadequate discussion of goals of care as the biggest opportunity for improvement.5 Proactively understanding which patients are most in need of such a discussion before end of life becomes imminent can be difficult for physicians alone.26 Prognostic models such as ours could greatly assist clinical intuition in directing limited hospital resources towards those who would benefit most.27
Admission to a hospital often reflects a change in prognosis and is an ideal time to renew discussion of care goals.27 HELPS could immediately identify patients on admission for whom a discussion regarding goals of care and/or the consideration of hospice may be appropriate. A study by Wright et al found that over 60% of dying patients did not recall having an end-of-life discussion with their physician, either because it never occurred or because the patient was not alert enough by that time to actively engage in a thorough conversation.2 Incorporating prognostic information and patient preferences for care into conversations from the beginning of the hospital stay will ensure that patients’ desires are met appropriately all throughout the care experience.
Of important note is that the outcomes studied were very rare, with prevalence of only 1.2% and 0.8%; any patients identified as having a 10%–20%+risk of either outcome would clinically be considered very high risk. Since the HELPS includes an estimate of risk, clinicians can easily set the threshold of risk that they consider appropriate, and do so taking into account the resources available. We suggest three levels: mild-risk patients with a HELPS of 20 or less, that would comprise approximately the 70% of patients with an estimated risk of less than 1%; moderate-risk patients with a HELPS between 21 and 45, the 25% of patients with an estimated risk between 1% and 10%; and severe-risk patients with a HELPS of 46 or greater, comprising the top 5% highest-risk patients, with a risk of >10% for inhospital death or discharge to hospice during that hospitalisation.
The generalisability of HELPS for use in other health systems was an important aspect of our study design. Even though our main goal was to produce an index that could be automatically calculated electronically, we realise the fact that many hospitals may not have or prefer that option. For this reason, we wanted to balance the score’s accuracy with ease of use so a paper-based version would also be viable. Many disease-specific prognostic models exist, but are often not applicable to a general care hospital population, in which multiple comorbidities are common. Our study was not limited to a single disease or age group, while also accounting for individual diseases of particularly high risk in the developed score.
There are, however, limitations to our results. The fact that our study only involves patients admitted to a single centre in the Midwest limits its generalisability. For example, 92.3% of our patients were white. However, the majority of the hospitalised lived outside Olmsted County (79%). A referral hospital serving a remote, rural population may differ from the population used to develop this risk index, but hospitals in developed countries should have a similar prevalence of chronic conditions and comorbidities.28
Some hospitals may not routinely use all of the measures included in HELPS, which limits the generalisability of the index. We specifically chose common, standardised measures to facilitate easier implementation in other hospitals. For those hospitals lacking routine measurement of variables in the proposed risk index, we recommend substitution of a similar variable. For example, RASS serves as a surrogate for acutely altered mental status, so could be replaced by other mental status measures such as the Glasgow Coma Scale. The Braden scores for mobility and nutrition could be replaced by other measures of functional decline. IPFS requires some calculation, so we included a table in the online supplementary appendix with instructions.
Supplementary file 1
Split-sample internal validation has been used to assess the performance of the index after its derivation. The fact that the model’s C-statistic (0.834) was similar in the validation cohort and in the training cohort demonstrates internal validity of the model. The predictive performance of HELPS will need to be confirmed and validated in other referral centres and community hospitals.
More studies will be needed to assess the impact of HELPS and similar indices on goals of care discussions and whether earlier discussion improves outcomes, reduces cost and increases comfort and satisfaction.
Conclusion
HELPS is able to identify patients with a high risk of hospital death or need for hospice at discharge. Early goals of care discussions in this setting may alter clinical course trajectories, and provide a greater opportunity to meet patient needs and expectations.
Acknowledgments
We thank Matthew Moore from the Mayo Clinic Center for Innovation for his insight and advice in the figure and table design, Patricia R Simonson from the Research and Academic Support Services at Mayo Clinic for her efforts in formatting and submitting this manuscript and Megan S Reinalda from the Department of Health Sciences Research at Mayo Clinic for her help in data acquisition.
References
Footnotes
Contributors Study concept and design: JMH, SR-B. Acquisition of data: DW. Analysis and interpretation of data: DW, SR-B, JMH, KJW, DMM. Drafting of the manuscript: SR-B, DW. Critical revision of the manuscript for important intellectual content: KJW, DMM, JMH. Final approval of version to be published: SR-B, DW, KJW, DMM, JMH.
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent Not required.
Ethics approval Mayo Clinic Institutional Review Board.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement The deidentified data set may be made available on request by email to the corresponding author if Mayo Clinic authorises the data sharing.