Article Text
PDF
Abstract
Objectives To identify risk factors for inpatient mortality after patients’ emergency admission and to create a novel model predicting inpatient mortality risk.
Design This was a retrospective observational study using data extracted from electronic health records (EHRs). The data were randomly split into a derivation set and a validation set. The stepwise model selection was employed. We compared our model with one of the current clinical scores, Cardiac Arrest Risk Triage (CART) score.
Setting A single tertiary hospital in Singapore.
Participants All adult hospitalised patients, admitted via emergency department (ED) from 1 January 2008 to 31 October 2017 (n=433 187 by admission episodes).
Main outcome measure The primary outcome of interest was inpatient mortality following this admission episode. The area under the curve (AUC) of the receiver operating characteristic curve of the predictive model with sensitivity and specificity for optimised cut-offs.
Results 15 758 (3.64%) of the episodes were observed inpatient mortality. 19 variables were observed as significant predictors and were included in our final regression model. Our predictive model outperformed the CART score in terms of predictive power. The AUC of CART score and our final model was 0.705 (95% CI 0.697 to 0.714) and 0.817 (95% CI 0.810 to 0.824), respectively.
Conclusion We developed and validated a model for inpatient mortality using EHR data collected in the ED. The performance of our model was more accurate than the CART score. Implementation of our model in the hospital can potentially predict imminent adverse events and institute appropriate clinical management.
- Inpatient mortality
- emergency department (ED)
- predictive model
- electronic health records (EHR)
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
The study identified several risk factors and developed a novel model for predicting future risk of inpatient mortality based on features collected at the emergency department.
Large electronic health record database and high predictive power.
Single-site study without external validation.
Introduction
Inpatient mortality, a key performance indicator of health services, provides general information concerning patient care delivery. Despite decades of research, inpatient mortality remains an issue.1–3 Lu et al showed that preventable deaths in emergency admitted patients with early mortality are not rare.4 The Harvard Medical Practice Study I estimated 27.6% of the adverse events as a result of negligence.5 Even a delay of a few hours in transferring critically ill patients to the intensive care unit results in increased mortality.6 Several studies7–9 have shown that physiological deterioration or abnormal vital signs before cardiac arrest or death were common, making it possible to predict the progression of adverse events. Previous intervention studies have demonstrated that inpatient mortality can be avoided by adequate care,10 frequent physiological measurement11 or other necessary measures. However, few studies managed to model the risk factors related to inpatients mortality after patients’ emergency admission through the emergency department (ED). Therefore, we proposed to use medical features collected at the ED to conduct predictive analysis, anticipating imminent adverse events and thus allowing physicians to respond appropriately.
There are numerous models for detecting mortality in the hospital, including the Early Warning Scores (EWS) system,12 which have been implemented in many hospitals to recognise early clinical deterioration. The concept of EWS was proposed by Morgan et al in 1997 and it included mainly the vital signs variables such as heart rate, blood pressure, respiratory rate, temperature and neurological status.13 Subsequently, multiple variants have been developed, such as NEWS,14 Modified Early Warning Scores (MEWS)15 and VitalPAC Early Warning Scores (ViEWS).16 The adoption of EWS in the hospital was found to correlate with reduced mortality rates and improved overall patient outcomes in a systematic review.17 However, several studies18–20 pointed out its limitations, such as oversensitivity, low specificity and the need for an accompanying critical care outreach team. Accordingly, there still is a need for improvement in accurate recognition. In 2012, the Cardiac Arrest Risk Triage (CART) score21 was developed with higher predictive power and usability than the MEWS. Furthermore, the increasing popularity of electronic health records (EHRs)22 creates an opportunity to acquire a more comprehensive and usable model for risk stratification in the hospital. Besides patient factors, non-patient factors, including prolonged emergency boarding,23 ED overcrowding24 and day of the week25 were used to augment the model’s sensitivity and specificity. Despite the common view of these worthwhile interventions, few clinical trials demonstrated a consistent improvement in reducing the hospital-wide mortality rate.
Currently, there are few studies on early risk stratification of ED patients for inpatient mortality in Singapore. A study in the USA has focused on patients with a specific diagnosis.26 Increased age, low systolic blood pressure or sodium levels, elevated heart rate or creatinine at admission were identified as important predictors for inpatient mortality in patients hospitalised for heart failure. However, few studies report the general risk of inpatient mortality from the information gathered when patients are presented to the ED in Singapore. In this study, we aimed to derive and validate a mortality prediction model from the available information commonly collected in the ED, assisting doctors in identifying high-risk patients.
Methods
Study design and setting
We performed a retrospective, single-centre study to derive a novel model to predict inpatient mortality in wards using routinely collected data in the ED and compared its accuracy to the CART score. Singapore is a city–state in Southeast Asia with 5.6 million people and a diverse ethnic composition. Its mixed healthcare system provides affordable care funded through both compulsory savings and partial subsidies. The site of this study is Singapore General Hospital (SGH), the largest and oldest tertiary hospital with more than 30 clinical disciplines and 1700 inpatient beds. Its ED receives over 120 000 visits and refers 36 000 inpatient admissions annually. EHR data were obtained from Singapore Health Services and were employed in this study. This study was approved by Singapore Health Services’ Centralised Institutional Review Board where patient consent was waived.
Patient and public involvement
Patients and the public were not involved in the design or planning of the study.
Study population and outcome
All patients visiting the ED from 1 January 2008 until 31 October 2017 who were subsequently admitted after their ED discharge across all clinical specialties in SGH were included in this study. We excluded patients who were below 21 years old and died in the ED. The primary outcome of interest was inpatient mortality, identified by the hospital’s admission and discharge administrative database.
Data collection and variables
We extracted data from the hospital’s EHR, named as the SingHealth Electronic Health lntelligence System (eHints). Patients’ details were deidentified to ensure that the data were sufficiently anonymised. Death records were obtained from the national death registry and were matched to specific patients in the hospital. We selected variables that were available in the ED prior to hospital admission to ensure the model was clinically useful for early identification. Selected variables included 4 demographical variables, 4 ED administrative variables and 11 clinical variables. Demographic variables include age, gender, nationality and race. ED administrative variables included consultation waiting time (unit: hour), ED boarding time (unit: hour), day of the week and shift time. Among these, ED boarding time is the amount of time that patients spent from the first consultation to ED discharge. Consultation waiting time is the amount of time that patients spent from ED registration to the first consultation with ED physicians. Clinical variables included one clinical service variable, six commonly sampled vital signs and four commonly sampled laboratory tests; specifically, they were blood gas (yes/no), pulse (beats/min), respiration rate (breaths/min), fraction of inspired oxygen (FiO2), blood oxygen saturation (SPO2), diastolic blood pressure (mm Hg), systolic blood pressure (mm Hg), bicarbonate (mmol/L), creatinine (μmol/L), potassium (mmol/L) and sodium (mmol/L).
Statistical analysis
The data were analysed using R V.3.42 (R Foundation, Vienna, Austria). After confirming the cohort, the data were randomly split into a derivation set (n=333 187, 77%) and a validation set (n=100 000, 23%). The derivation set was used to generate the model. Model accuracy was reported on the validation set, and bootstrapped samples were applied to calculate 95% CIs. During this analysis, a value of vital signs or laboratory tests would be considered as an outlier if it was beyond the normal range on the basis of domain knowledge. All detected outliers were set to missing. Then, all missing values were imputed using the median value of the derivation dataset.
Baseline characteristics of the study population were analysed on both derivation and validation sets to confirm similarity. In the descriptive summaries, frequencies and percentage were reported for categorical variables, while means and SDs were reported for continuous variables. We compared admitted patients with and without inpatient mortality using two-tailed Student's t-test for continuous variables and χ2 test for categorical variables. The p-value shows the significance of difference for admitted patients between inpatient mortality and successful discharge. Because of the large sample size associated with EHRs, the threshold for declaring the statistical significance level was set at p<0.01, much smaller than the usual 0.05 level, in order to reduce the chances of finding spurious effects.
The prediction model was built by applying two-step logistic regression to the derivation set. First, univariate analysis was performed on all variables to access their independent association with inpatient mortality. The largest cohort of each variable was selected as the baseline for comparison with other groups. ORs and the corresponding CIs were calculated. Second, variables with p<0.01 from the first step were selected to be analysed using multivariate logistic regression with backward stepwise variable selection.
In the final regression model, the modelling performance was evaluated on the validation set. Our model generated a probability of inpatient mortality from 0 to 1 for each admission episodes. The predictive power of the model was calculated using the area under the curve (AUC) in the receiver operating characteristic analysis. In order to compare our model with current clinical scores, we also applied the CART21 score into the same validation set and compared the performance between the CART score and our novel model.
Results
Basic characteristics
A total of 433 187 unique emergency admission episodes were included in this study. Of the 433 187 eligible episodes, 15 758 episodes (3.64%) met the outcome, that is, inpatient mortality. The mean age of the whole cohort was 62.1 (SD=17.7) years; 50.1% were female (n=216 914); most patients were Singaporean (90.5%, n=392 219); the ethnic compositions were similar to population norms (71.2% for Chinese, 12.1% for Malay, 10.6% for Indian and 6.1% for others); 2.1% (n=9144) of the patients received blood gas services in the ED; the mean ED boarding time was 4.78 (SD=3.83) hours; and the mean ED consultation waiting time was 0.77 (SD=0.79) hours.
The whole cohort was subsequently divided into the derivation set and validation set, as displayed in figure 1. Table 1 shows the statistics of highly similar populations in both sets. The derivation set was constitutive of patients with a mean age of 62.1 (SD=17.7), with similar male (49.9%) and female (50.1%) proportions and with the ethnic breakdown representing the general Singaporean population. Compared with the patients who survived to discharge, patients who died in the hospital were older, had shorter ED boarding time and consultation waiting time, and had a higher probability of receiving blood gas services while in the ED. They also had lower SPO2, blood pressure, bicarbonate and sodium concentration with a higher pulse, respiration rate, FiO2, and potassium and creatinine concentrations.
Flow of patients’ emergency admissions.
Description of the study cohort
Univariate analysis
Table 2 shows the OR and adjusted OR of all demographic, administrative and clinical variables. All variables were respectively significant in the univariate regression in terms of the p value. We treated vital signs and laboratory test values as continuous variables, and their ORs represented the increase or decrease in the odds of inpatient mortality for a one-unit increase in this feature. Observed from the demographical data, patients who were male, ethnic Chinese had a higher risk of inpatient mortality. Patients who were foreigners and other races beyond Chinese were unlikely to die in the hospital after emergency admission. Administratively, patients who had shorter consultation waiting time and ED boarding time were more likely to die in the hospital. Clinically, patients with a higher pulse, respiration rate, FIO2, creatinine and potassium concentrations, and lower blood pressure, SPO2, bicarbonate and sodium concentration had a higher risk of inpatient death. All 19 variables were selected for multivariate stepwise analysis as a result of all their p values being below 0.01.
Univariate and multivariate analyses
Multivariate analysis
All variables were used to create the stepwise regression model and no variable was removed through stepwise variable selection. The final model contains 19 variables, and the multivariate analysis with the corresponding adjusted ORs is shown in table 2. Older Singaporeans with Chinese ethnicity had a higher change of inpatient mortality. Although diastolic blood pressure, shift time and day of the week were not very significant in multivariate analysis, they were included in the final model after backward stepwise variable selection and due to clinical judgements.27
Predictive model performance
Our model shows good discriminatory capability on predicting inpatient mortality. On the validation set, the model achieved an AUC of 0.817 (95% CI 0.810 to 0.824) with a sensitivity of 73.1% (95% CI 70.7% to 77.6%) and a specificity of 75.4% (95% C: 70.9% to 76.9%) under the optimal threshold (probability=0.037), as shown in figure 2. In contrast, the performance of the existing CART score achieved an AUC of 0.705 (95% CI 0.697 to 0.714) with a sensitivity of 72.1% (95% CI 70.7% to 73.6%) and a specificity of 56.1% (95% CI 55.8% to 56.4%) under the optimal threshold (CART value=7). The calibration curve of our developed model is shown in figure 3.
Receiver operating characteristic curves of our model and CART score on the validation set. AUC, area under the curve; CART, Cardiac Arrest Risk Triage.
Model calibration curve on the validation set.
Discussion
In this study, the main finding is that 19 routinely collected variables from the ED EHR system can be used to predict inpatient mortality for patients after their emergency admission. Our predictive model has better discriminative power than the CART score (AUC 0.817 vs 0.705) on the same validation set. The results suggest the possibility of building a reliable inpatient mortality model from the basic demographic, administrative and limited clinical information acquired from the ED when patients are admitted to the hospital through ED. By deriving a model of inpatient mortality using routinely collected ED data, our study identifies factors associated with inpatient mortality and provides a potentially useful tool for risk stratification of ED patients.
A major strength of our model is the size of the dataset, which was used for deriving this model. This is among the largest datasets used to generate an inpatient mortality predictive model with a cohort of over 430 000 patients in a 10-year period, targeting almost the whole hospital. In addition, it included a large amount of diversity due to Singapore’s diverse population. Another advantage of our model is its comprehensiveness, making it applicable to the general patient population presenting to the ED rather than some specific patient subgroups. Furthermore, the application of EHR systems will make our model easy to implement.
There are several reasons why the CART score underperformed in our novel model in our study. At first, the CART score did not comprise laboratory test variables. The importance of including routine laboratory test values in the risk predictive model has been demonstrated in other studies. For example, in a study by Churpek and colleagues,28 including laboratory values in their model contributes important knowledge to the field. Pine et al 29 and Froom et al 30 also gave evidence of laboratory values improving predictions of hospital mortality. Second, CART was unable to make use of valuable routine administrative data. Guttmann et al 31 and Parker et al 32 have previously shown that waiting time, work shifts and other administrative variables were greatly associated with inpatient mortality and hospital admission. In comparison, our model takes both ED administrative data and laboratory test values into account, proving a higher accuracy than the CART score.
Previous researchers have created several predictive tools for inpatient mortality. For example, Prytherch et al 16 developed the ViEWS score, mainly using vital signs to predict mortality for hospitalised patients within 24 hours. The significant predictors of mortality were the pulse, breathing rate, temperature, systolic BP, SPO2, FiO2 and mental status. Although vital signs are potential predictors of adverse events, they give the rapid response team (RRT) a very short time to respond, especially in a hospital with full capacity or lack of manpower. Since changes in vital signs occur hours before the event, these changes may not be seen at the time of consultation at the ED when potentially high-risk patients have non-discriminatory vital signs similar to those of other healthy patients. Second, elderly patients may not have the expected vital sign changes associated with the clinical deterioration, and modelling using vital signs alone might miss out cases. It was demonstrated in a study by Churpek and colleagues,33 who suggest additional predictors of adverse events for elderly patients. Our model is notably different from this because it involved laboratory test values and administrative data besides vital signs and were presumably appropriate for the rapidly ageing population in Singapore.34
Another study35 in Australia employed multivariate logistic regression of variables from datasets obtained at triage in one hospital to derive and validate a mortality prediction model, the Triage Information Mortality Model (TIMM). This TIMM included age, gender, time of the year, ambulance, Australasian triage scale and nine chief complaint codes. However, it did not include any physiological variables that were considered as strong predictors and could be obtained conveniently from the EHR system. In comparison, our model combined demographic, administrative and physiological variables, which will provide a much more comprehensive profile and capture sufficient information from patients in the ED, hence improving the model’s predictive power.
Our data analysis also produces some notable findings regarding risk factors related to inpatient mortality. It identified increased age, low blood pressure, high heart rate and elevated creatinine and potassium concentrations, and decreased sodium and bicarbonate concentrations when patients present to the ED as important predictors for inpatient mortality. Besides these factors, our study identified some non-patient factors, such as emergency boarding time, day of the week and shift time, which can affect patient outcomes. Presenting to ED on Friday or the weekend and a shift time of 24:00 to 8:00 were found to increase risk, consistent with a large study by Aylin and colleagues36 in the USA, which shows 10% higher odds of death for all emergency admission during the weekend compared with admission during a weekday. An excess in mortality may reflect differences in quality of care, potentially as a result of the ED overcrowding, insufficient services, change of shift and slower access to critical investigations. However, the differences in mortality decreased after adjustment for other factors in our analysis. Shorter ED boarding time and consultation waiting time become predictors potentially due to severely critical patients with a fast track to admission and intensive resources.
The information needed for this novel model is readily available at the time of consultation at the ED when the first set of laboratory tests is done, when a physician has to make a decision on further management and disposition of the patient. Our model can be deployed for early identification of high-risk patients. Afterward, we can allocate more intensive resources to high-risk patients with a sufficient level of monitoring, increasing nursing attention,37 activation of an RRT38 or a medical emergency team.39 Thus, through our model, these patients could be seen early after emergency admission, and the previously mentioned interventions can be started to avoid severe sudden adverse events during their inpatient stay. Similarly, low-risk patients below the predictive threshold could potentially be safely identified who might not need admission or intensive monitoring and thus save precious in-patient resources. Overall, the good performance, usability and widespread adoption of an advanced EHR system make our model easy to integrate into the hospital electronic system such that the probability of inpatient mortality or real-time risk score can be calculated for every patient when they are presented to the ED and ready for admission to the hospital. The model can supplement the physician’s judgement in decision-making.
Limitation
There are several limitations in this study. First, all variables included in this study are based on EHR, and it only contains routinely collected information and does not include all information available that should, in theory, be elicited early when patients present to the ED. For example, comorbidity information or the Charlson Comorbidity Index40 was considered as significant predictors. However, they were not available in our current analysis. Other health uses, such as intubation and resuscitation, have been proven to be predictive of overall mortality and should have been included in our model. Furthermore, due to the lack of neurological features such as the Glasgow Coma Scale (GCS) score, which were not common variables collected in a Singaporean hospital, we were not able to calculate the MEWS score and compare it with our model. In future studies, the GCS and other important features should be recorded and incorporated into prospective investigations. Second, this is a single-site study at a tertiary hospital, and our findings may not be generalised to other settings; thus, our results need to be validated in different hospital settings in Singapore or other countries in future research, especially population consisting of different ethnicities to avoid centre-specific bias. Prospective data collection is supposed to explore the clinical value and effect of our model in practice and to further prove its efficacy. Third, our model is complex and the calculation should be done electronically. The ability to implement an EHR system varies in different hospitals and the lack of features monitored by the system may limit the generalisability of our model.
Conclusion
In summary, we identified several risk factors and developed a novel model for inpatient mortality using 10-year EHR data routinely collected at the ED. The discriminative capability of our model was better than that of the traditional clinical score, the CART score. Implementation of our model in the ED can allow accurate and timely identification of a high-risk cohort for interventions during their inpatient stay, resulting in a potential reduction in avoidable inpatient mortality.
References
Footnotes
Contributors NL, FX, and MEHO conceived and designed the study. NL, MEHO and BC supervised the study. FX and SXW performed data retrieval and preprocessing. FX analysed the data. FX, NL, YA, LLL, AFWH, SSWL, DBM, MEHO and BC interpreted the results. FX wrote the first draft of the paper and all authors critically revised the paper and gave final approval for publication.
Funding This research received funding from Duke-NUS Medical School and the Estate of Tan Sri Khoo Teck Puat under the Khoo Pilot Award (Collaborative).
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval This study was approved by Singapore Health Services (SingHealth) Centralised Institutional Review Board (CIRB Ref 2017/2666) with a waiver of informed consent.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Details of the variables and derived predictive model are available from the corresponding author.