Article Text
PDF
Abstract
Objective To mitigate the burden of COVID-19 on the healthcare system, information on the prognosis of the disease is needed. The recently developed Risk Stratification in the Emergency Department in Acutely ill Older Patients (RISE UP) score has very good discriminatory value for short-term mortality in older patients in the emergency department (ED). It consists of six readily available items. We hypothesised that the RISE UP score could have discriminatory value for 30-day mortality in ED patients with COVID-19.
Design Retrospective analysis.
Setting Two EDs of the Zuyderland Medical Centre, secondary care hospital in the Netherlands.
Participants The study sample consisted of 642 adult ED patients diagnosed with COVID-19 between 3 March and until 25 May 2020. Inclusion criteria were (1) admission to the hospital with symptoms suggestive of COVID-19 and (2) positive result of the PCR or (very) high suspicion of COVID-19 according to the chest CT scan.
Outcome Primary outcome was 30-day mortality, secondary outcome was a composite of 30-day mortality and admission to intensive care unit (ICU).
Results Within 30 days after presentation, 167 patients (26.0%) died and 102 patients (15.9%) were admitted to ICU. The RISE UP score showed good discriminatory value for 30-day mortality (area under the receiver operating characteristic curve (AUC) 0.77, 95% CI 0.73 to 0.81) and for the composite outcome (AUC 0.72, 95% CI 0.68 to 0.76). Patients with RISE UP scores below 10% (n=121) had favourable outcome (zero deaths and six ICU admissions), while those with scores above 30% (n=221) were at high risk of adverse outcome (46.6% mortality and 19.0% ICU admissions).
Conclusion The RISE UP score is an accurate prognostic model for adverse outcome in ED patients with COVID-19. It can be used to identify patients at risk of short-term adverse outcome and may help guide decision-making and allocating healthcare resources.
- infectious diseases
- accident & emergency medicine
- internal medicine
- COVID-19
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
We performed a retrospective analysis in a relatively large cohort of patients with COVID-19 in one of the most heavily affected areas in the Netherlands.
The Risk Stratification in the Emergency Department in Acutely ill Older Patients is easy to use and can be readily implemented in routine practice in the emergency department (ED).
This study was performed in the two EDs of one medical centre, which may limit generalisability of the results. The study was limited to patients admitted to the hospital.
Background
The COVID-19 pandemic has affected millions of people and has put enormous strains on healthcare resources.1–4 To mitigate the burden on the healthcare system, information on the prognosis of the patients is needed. Although most patients with COVID-19 develop only mild symptoms, some develop severe and potentially fatal complications.1 2 5 6
Prediction models could help to estimate the risk of a poor or good outcome and may, therefore, assist in triaging patients when allocating healthcare resources. Many risk prediction models have been developed for patients with COVID-19, most of which still need rigorous external validation efforts to assess their added value to clinical practice; however, our group developed and validated the Risk Stratification in the Emergency Department in Acutely ill Older Patients (RISE UP) score, an accurate and easy to use model, predicting mortality in medical patients older than 65 years in the emergency department (ED).7 The model had very good discriminatory value with an area under the receiver operating characteristic curve (AUC) of 0.84 in the derivation and 0.83 in the validation cohort.
We hypothesised that the RISE UP score may also be valuable in predicting prognosis in patients with COVID-19, as this disease predominantly affects the same age group. The aim of the present retrospective study was to evaluate the discriminatory value of the RISE UP score with regards to poor outcome (mortality and/or admission to the intensive care unit (ICU)) in ED patients with COVID-19.
Materials and methods
This study has been reported in accordance with the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement.8
Study design and setting
We performed a retrospective analysis at the two EDs of the Zuyderland Medical Centre (MC), location Heerlen and Sittard-Geleen. This is one of the largest teaching hospitals in the Netherlands providing secondary care to a total of 55 000 patients during ED visits on a yearly basis.
Study sample
The study sample consisted of consecutive adult (18 years or older) medical ED patients diagnosed with COVID-19 in the period from 3 March to 25 May 2020. Patients were included if they met the following criteria: (1) admission to the hospital with symptoms suggestive of COVID-19 (ie, coughing, common cold, sore throat, dyspnoea, acute diarrhoea, vomiting, fever or an unexpectedly discovered oxygen saturation below 92%) and (2) positive result of the PCR for SARS-CoV-2 in respiratory specimens or (very) high suspicion of COVID-19 according to the CT scan of the thorax (COVID-19 Reporting and Data System (CO-RADS) 4 or CO-RADS 5, see online supplemental table 1).9 We excluded patients who revisited the ED after an earlier ED presentation during the study period.
Supplemental material
Data collection
From electronic medical records, we collected data on age, sex and information regarding comorbidity according to the Charlson Comorbidity Index (CCI).10 The following vital signs were retrieved: heart rate, mean arterial blood pressure, respiratory rate (RR), oxygen saturation, temperature and Glasgow Coma Scale (GCS). Supplemental oxygen was taken into account when measuring oxygen saturation. When the RR or GCS were missing, we used paCO2 and descriptions in the medical records to deduce these values, similar to other studies.7 11 12 In addition, we collected routinely assessed laboratory tests: blood urea nitrogen (BUN), lactate dehydrogenase (LDH), bilirubin, albumin, lymphocytes and C reactive protein (CRP).
We retrieved all items of the RISE UP score. The original formula of the RISE UP score to calculate an individual’s probability of 30-day mortality is P(30-day mortality)=1/(1+exp (-(−3.908+0.050*Age +1.115*≥2 Abnormal Vital Signs (yes=1, no=0)–0.112*Albumin (in g/L)+0.284*(BUN (in mmol/L)/5)+0.120*(LDH (in U/L)/100)+0.875*Bilirubin>20 µmol/L (yes=1, no=0))).7 The RISE UP score is shown as a percentage.
Furthermore, we collected the results of the reverse-transcriptase PCR for SARS-CoV-2 in respiratory specimens and the results of the CT scan of the thorax.9 Finally, we retrieved data on length of hospital stay, admission to the ICU and mortality. Data on mortality were verified using medical records, which are connected to the municipal administration office.
Outcomes
The primary outcome was all-cause mortality within 30 days after ED presentation. The secondary outcome was a composite of mortality within 30 days after ED presentation and admission to the ICU. In our hospital, the main reason for admitting patients to the ICU was mechanical ventilator support.
Statistical analysis
Baseline characteristics were analysed using descriptive statistics on the observed data. For each included patient, we calculated their individual probability of 30-day mortality and of the composite endpoint of 30-day mortality and/or admission to ICU using the initial RISE UP score formula. In case the probability could not be computed in over 5% of patients due to missing values, data would be imputed using multiple imputation, as per our study protocol. Predictive performance of the RISE UP score in this external validation was quantified as discriminative ability (ie, the ability of the model to separate those who will experience the event from those who will not) and model calibration. We calculated the AUC of the receiver operating characteristic (ROC) curve to quantify the discriminative ability. An AUC of 0.5 would correspond to no more discriminative value than the toss of a coin, whereas an AUC of 1.0 would mean perfect separation. Calibration of the RISE UP score was assessed by visually inspecting the calibration plot that shows the agreement between predicted and observed probabilities for subgroups of patients based on similar predicted risk. In case of poor calibration (ie, the intercept of the plot is not equal to 0 and/or the slope of the plot is not equal to 1), the model was recalibrated using logistic regression analysis with the linear predictor of the RISE UP score, the linear sum of the regression coefficient multiplied by their respective predictor values, as the only independent variable.
Finally, we aimed to identify cut-off values of the RISE UP score that will support clinical decision-making, by identifying patients with low and high risk of an adverse outcome (low-risk and high-risk group). To achieve this, we calculated sensitivity, specificity, positive and negative predictive values from multiple cut-off values of the RISE UP score.
All data were analysed using IBM SPPS Statistics for Windows, IBM Corporation, Armonk, New York, USA, V.25.0.
Patient and public involvement
The patients and the public were not involved in the design, conduct, reporting or dissemination plans of our research.
Results
Study sample
During the study period, 642 consecutive ED patients diagnosed with COVID-19 met the inclusion criteria and were included (table 1). The median age of patients was 72 years (IQR 62–80), and 69.2% were 65 years or older. Most patients (63.4%) were men. More than half of the patients (59.8%) had two or more abnormal vital signs. The laboratory results showed elevated LDH (median 350 U/L). The median length of hospital stay was 6 days (IQR 3–12).
Baseline characteristics of study population
In our sample, 102 patients (15.9%) were admitted to ICU and 167 patients died, yielding a 30-day mortality of 26.0% (figure 1). The median number of days until death was 8 (IQR 4–15).
Kaplan-Meier curve.
RISE UP score
The RISE UP score could not be calculated in 21 patients (3.2%) because of missing values (laboratory tests). As this was below our 5% threshold, no imputation step was performed. For 621 patients, the RISE UP score yielded an AUC of 0.77 (95% CI 0.73 to 0.81) with respect to the 30-day mortality and an AUC of 0.72 (95% CI 0.68 to 0.76) with respect to the composite endpoint of 30-day mortality and/or admission to the ICU.
The initial calibration plot showed average underestimation of 30-day mortality by the model and a slope of >1 (figure 2). This could at least in part be due to the fact that in the derivation and validation cohort of the RISE UP score, we observed a 30-day mortality of 10.9% and 13.3%, respectively; therefore, we performed recalibration of the model.7 The formula was adjusted by changing the intercept to −2.083 and multiplying the linear sum of regression coefficients multiplied by their respective predictor values by 0.795. The adjusted formula for the RISE UP score in patients with COVID-19 was P(30-day mortality)=1/(1+exp (-(−2.083+0.795 * (0.050*Age +1.115*≥2 Abnormal Vital Signs (yes=1, no=0)–0.112*Albumin (in g/L)+0.284*(BUN (in mmol/L)/5)+0.120*(LDH (in U/L)/100)+0.875*Bilirubin>20 µmol/L (yes=1, no=0)))).
Calibration plots of the predicted 30-day mortality (x axis) versus the observed 30-day mortality (y axis) are represented by the diagonal line. The left calibration plot shows calibration of the RISE UP score before recalibration. The right calibration plot shows calibration of the RISE UP score after recalibration. The adjusted formula of the RISE UP score in COVID-19: P(30-day mortality)=1/(1+exp (-(−2.083+0.795 * (0.050*Age +1.115*≥2 abnormal vital signs (yes=1, no=0)–0.112*Albumin (in g/L)+0.284*(BUN (in mmol/L)/5)+0.120*(LDH (in U/L)/100)+0.875*Bilirubin>20 µmol/L (yes=1, no=0)))). RISE UP, Risk Stratification in the Emergency Department in Acutely ill Older Patients.
Determination of clinical cut-off values
The individual RISE UP scores were stratified according to their probability to predict 30-day mortality. Patients with a RISE UP score <10% were considered to be at (very) low risk of mortality (tables 2 and 3). In this subgroup of 121 patients (19.5%), there were no deaths, yielding a sensitivity of 100% (95% CI 97.7 to 100) and a negative predictive value of 100% (95% CI 97.7 to 100). Only six patients (5.0%) were admitted to ICU, yielding a sensitivity of 97.3% (95% CI 94.1 to 99.0) and a negative predictive value of 95.0% (95% CI 89.6 to 97.7). A RISE UP score <5% also yielded high sensitivity and negative predictive value; however, this subgroup of patients consisted of 41 patients only (6.6%).
Cut-off values of the RISE UP score and prognostic accuracy for mortality
Cut-off values of the RISE UP score and prognostic accuracy for mortality or ICU admission
Patients with a RISE UP score >30% were at high risk of a poor outcome (table 2). In this subgroup of 221 patients (35.6%), we found a mortality of approximately 50% and 15% ICU admission.
Discussion
In this retrospective study in two EDs in the Netherlands, we found that the RISE UP score showed good discriminatory value with respect to 30-day all-cause mortality and/or admission to ICU in ED patients with COVID-19. The model yielded an AUC of 0.77 (95% CI 0.73 to 0.81) for 30-day mortality and an AUC of 0.72 (95% CI 0.68 to 0.76) for a composite of 30-day mortality and/or admission to ICU. After performing recalibration, the RISE UP score was well calibrated. We found that patients with a RISE UP score <10% had favourable outcomes and that patients with a RISE UP score >30% were at high risk of adverse outcomes.
Using the RISE UP score, the probability of both favourable and poor outcomes can be predicted in the first 2 hours of the ED visit. This is important because this may guide clinical decision-making, especially during the current COVID-19 pandemic with healthcare system facing problems with clinical healthcare facilities (ie, shortage of ICU beds) and the development of out-of-hospital low care facilities (ie, corona hotels). The finding that the RISE UP score may be useful to guide clinical decision-making is important because a clinical algorithm like this score is essential for allocation of healthcare resources. The RISE UP score is applicable for both sides of the clinical spectrum. In our cohort, 121 patients (19.5%) had a RISE UP score <10% and had very low risk of short-term mortality or ICU admission. In these patients, the clinician can choose safely to refer the patient to outpatient treatment or to discharge at an early stage. On the other hand, the 74 patients (11.9%) in our cohort with a RISE UP score >30% had a high risk of adverse outcome. This may support the decision to transfer these patients to ICU at an early stage. Furthermore, in patients with high RISE UP scores, pre-existing multiple comorbidities and poor clinical performance, the model might support the decision to choose supportive care only.
The RISE UP score was recently developed to predict 30-day all-cause mortality in older medical ED patients. The score consists of six easily and readily available items during the ED visit.7 The RISE UP score can be easily implemented in routine practice with an online calculator (https://jscalc.io/calc/o1vzp36bIDGQUCYl). The model has face validity as it includes items that are markers of severe diseases, such as sepsis, renal failure, liver failure or shock. Because of this, the RISE UP score reflects the severity of illness in ED patients; therefore, it is not surprising that the model has good discriminatory value for poor outcome in ED patients with COVID-19, since our sample largely consisted of older medical patients (69.2% were 65 years or older). Other studies have also shown higher mortality in older patients with COVID-19.13–16 Although we considered subgroup analysis of patients aged 65 years and over, we refrained from doing so because the power was not sufficient for subgroup analysis. We state that application of the RISE UP score easily identifies patients with COVID-19 who are at high and at low risk of dying within the first 30 days after presentation to the ED.
The original research article concerning the RISE UP score showed a better predictive performance compared with our current study (AUC 0.83 vs 0.77).7 The difference can be explained in several ways. First, 30-day mortality was around 11% in the original cohort, while in our cohort it was 26.0%.7 This can be explained because, in this study, we only included ED patients who were admitted to the hospital, whereas patients that were discharged from the ED were excluded. Furthermore, COVID-19 can have a serious course in admitted patients. Therefore, the probability of a poor outcome was higher in our cohort. Second, the levels of LDH were often elevated in patients with COVID-19 (81.1% were higher than the reference value). Since LDH is one of the items in the RISE UP score, this may affect the discriminatory value. Third, in our clinical experience, some patients with COVID-19 show a rapid deterioration of their clinical condition, while there were few objective symptoms in the initial assessment. This phenomenon may play a role as well, since the RISE UP score includes the initial assessment only.
The RISE UP score still has good discriminatory value in a population that is different from the population in which it was designed. This can be explained because the model accurately reflects the severity of illness in ED patients. The severity is determined by both the severity of the disease, and pre-existing comorbidities. In COVID-19, the prognosis is also determined by the disease itself and patient-related factors (ie, age and comorbidities).1 5
In a recent systematic review, a total of 16 prognostic models for predicting mortality, progression to severe disease or length of hospital stay in patients with COVID-19 were analysed.17 The AUC estimates ranged from 0.85 to 0.99; however, most proposed models were not clearly described and, therefore, it is unclear whether they are feasible in other settings. The authors of the systematic review do not recommend the use of any of these models in current practice.17 New prediction models will be developed for patients with COVID-19, such as two recently published studies, that showed promising results and should definitely be included in future studies.18 19 The RISE UP score is an accurate and externally validated prediction model with good discriminatory value in unselected older medical patients.7 We state that the RISE UP score can be used in current clinical practice.
Our study has several limitations. First, our study was performed in the two EDs of one MC, which may limit generalisability of the results. However, our study has a relatively large cohort of patients with COVID-19 in one of the most heavily affected areas of the Netherlands. Second, we included patients with symptoms only, but 89 patients (13.9%) did not have a positive PCR result. Because there is no gold standard for the diagnosis of COVID-19 and the diagnostic accuracy of the PCR is limited, we believe that it is diligent to include patients with a positive result of the chest CT scan as well.20 Other studies have used these inclusion criteria as well.21 If we apply the RISE UP score only to the patients who were confirmed with PCR, we found similar discriminatory value (AUC 0.76 (95% CI 0.71 to 0.80) for 30-day mortality). Next, our study was limited to patients admitted to the hospital. It would be interesting to apply the RISE UP score to patients who were discharged as well; however, these data were not available. Finally, the number of ICU admissions in our study was relatively low (15.9%), which may result from decisions to initiate conservative care in patients with pre-existing frailty or severe comorbidity. These decisions may be different in other countries, which made us decide to study ICU admissions as a composite outcome only. Since the existing literature is dominated by predicting mortality in patients with COVID-19, we consider the inclusion of ICU admission as a composite endpoint a strength of our current study.
The RISE UP score can be readily implemented in routine practice at the ED. It is easy to use and not time-consuming to fill in. Our current study shows that the discriminatory value in ED patients with COVID-19 is good and that the RISE UP score is, therefore, also applicable in these patients.
Conclusion
In conclusion, the RISE UP score is an accurate and easily available prediction model for 30-day mortality and ICU admission in ED patients with COVID-19. The score is useful to identify patients at low and high risk for clinical complications. So this model may contribute to fast recognition of patients who are at low or high risk for short-term adverse outcome and guide clinical decision-making and allocating healthcare resources in this severe pandemic, dealing with scarcity of clinical facilities and materials.
References
Footnotes
Contributors PMELvD, NZ, PS, DJLvT and JB collected the clinical data. PMELvD, NZ and SMJK performed the statistical analysis. Data were interpreted by all authors. PD drafted the first version of the manuscript. NZ, PMS, PWDL, SvK, DJLvT and JB critically reviewed the manuscript. All authors have read and approved the final version of the manuscript. The corresponding author has the right to grant on behalf of all authors and does grant on behalf of all authors, an exclusive licence (on non-exclusive for government employees) on a worldwide basis to the BMJ Publishing Group Ltd to permit this article (if accepted) to be published in BMJ editions and any other BMJPGL product and sublicences such use and exploit all subsidiary right, as set in our licence. The corresponding author affirms that the manuscript is an honest, accurate and transparent account of the study being reported. No important aspects of the study have been omitted.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests All authors have completed the Unified Competing Interest form (available on request from the corresponding author) and declare: no support from any organisation for the submitted work, no financial relationships with any organisations that might have an interest in the submitted work in the previous three years, no other relationships or activities that could appear to have influenced the submitted work.
Patient consent for publication Not required.
Ethics approval This study was approved by the medical ethics committee of the Zuyderland MC (METCZ20200136). Informed consent was obtained retrospectively by an opt-out method by writing a letter to the included patients.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available upon reasonable request. Additional data are available upon reasonable request.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.