Article Text

Original research
Development and internal validation of a simple prognostic score for early sepsis risk stratification in the emergency department
  1. Bofu Liu1,
  2. Dongze Li2,
  3. Yisong Cheng2,
  4. Jing Yu3,
  5. Yu Jia2,
  6. Qin Zhang2,
  7. Yanmei Liu4,
  8. Yu Cao1
  1. 1Emergency Department, Sichuan University West China Hospital, Chengdu, China
  2. 2Department of Emergency Medicine, West China Hospital, Sichuan University, Chengdu, China
  3. 3West China School of Nursing, Sichuan University West China Hospital, Chengdu, China
  4. 4Chinese Evidence-based Medicine Center and CREAT Group, West China Hospital, Sichuan University, Chengdu, China
  1. Correspondence to Dr Yu Cao; caoyu{at}wchscu.cn

Abstract

Objectives No validated, simple, powerful and continuously monitorable risk prediction tools are available for patients with sepsis during the early phases in the emergency department (ED). We sought to derive a novel Simple Sepsis Early Prognostic Score (SSEPS) composed of physiological indicators that do not depend on laboratory tests and that can be used by emergency clinicians in predicting outcomes in patients with sepsis.

Design Retrospective cohort analysis of a collected data source.

Participants Patients with sepsis admitted to the ED of the West China Hospital of Sichuan University between July 2015 and June 2016 were included. We excluded patients who were pregnant, those with cardiac or respiratory arrest, and those using vasoactive drugs before admission to the ED.

Primary outcome measures 28-day all-cause mortality.

Results The SSEPS consisted of age, heart rate, respiratory rate and altered consciousness. Patients in the development cohort with higher SSEPS had a significantly higher mortality (first tertile vs second tertile vs third tertile: 12.5% vs 28.6% vs 53.5%, p<0.001). The area under the receiver operating characteristic curve for SSEPS was 0.762 (95% CI 0.686 to 0.838), which was similar to Sequential Organ Failure Assessment (SOFA) (area under the curve: 0.745, 95% CI 0.692 to 0.798) and Acute Physiology and Chronic Health Evaluation (APACHE II) (area under the curve: 0.750, 95% CI 0.681 to 0.819). Moreover, the decision curve analysis showed that the net benefit of SSEPS was higher than SOFA and APACHE II at any probability threshold.

Conclusion The SSEPS is simple and useful for clinicians in stratifying high-risk patients with sepsis at the early phase of ED admission.

  • infectious diseases
  • risk management
  • intensive & critical care

Data availability statement

Data are available upon reasonable request. The data sets generated and/or analysed during the current study are not publicly available, in accordance with China Ethics Committee of Registering Clinical Trials, but are available from the coauthor on reasonable request.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Strengths and limitations of this study

  • One of the strengths of the study was finding a new model developed on the basis of vital signs so as to accurately predict 28-day all-cause mortality of patients with sepsis.

  • This study focuses on recognition/identification of patients at risk immediately after arrival to the emergency department.

  • The new model can generally be applied to identify high-risk patients with sepsis during the early stages in the emergency department.

  • Limitations of this study are its small sample size (only 236 patients in the validation group) and the lack of an external validation data set, which may limit the rigour of the study and confidence in findings.

  • The variables included in the Simple Sepsis Early Prognostic Score were those recorded on admission to our emergency department.

Introduction

Sepsis is a fatal organ dysfunction disorder with high mortality that is caused by a dysregulated host response to infections.1 2 Imbalanced immune response in sepsis underlies serious organ failures and clinical outcomes.3 Early diagnoses and appropriate management of sepsis decrease the risk of adverse outcomes; however, the mortality of patients with sepsis remains high. The incidence of sepsis in hospitals and intensive care units (ICUs) is 2% and 6%–30%, respectively, and these percentages are higher in developing countries.4 Mortality from sepsis varies substantially across hospitals (30%–50%),5 6 suggesting the need for more accurate assessment to guide individualised treatment decision algorithms. In addition, the ‘2016 surviving sepsis campaign’ indicated that early risk stratification and appropriate management during the initial hours after onset of symptoms improve outcomes.7 Therefore, risk stratification tools to identify high-risk patients with sepsis during the early phase of admission are critical to treatment decisions.

Although current predictive scores such as the quick Sequential Organ Failure Assessment (qSOFA), the Modified Early Warning Scoring (MEWS), the Mortality in the Emergency Department Sepsis (MEDS), the Acute Physiology and Chronic Health Evaluation (APACHE II), and the Sequential Organ Failure Assessment (SOFA) have been proven to have excellent risk stratification efficiency,2 8–11 most of them rely on data from laboratory tests or proved to be of poor validity and discriminating ability. Thus, these complex scoring systems may miss out many early critical emergency department (ED) or prehospital patients. Moreover, the conditions of patients with sepsis can change rapidly. With this in mind, traditional scoring systems cannot immediately and repeatedly assess patients during ED admission or prehospital and realise dynamic evaluation of their risk level. Therefore, a simple prognostic scoring system that does not rely on detailed medical history or invasive medical testing is needed to carry out early assessments in the ED and even before admission.

The Sepsis 3.0 consensus emphasises the organ dysfunction caused by imbalanced host responses. Studies have shown that state of consciousness and vital signs are associated with imbalance in host responses in patients with sepsis, and these variables have commonly guided the treatment of patients with sepsis.12 Thus, we hypothesised that simple physiological indicators which reflect host response during the early stages of sepsis can provide important information for evaluating the prognosis of patients. We aimed at constructing and validating a novel and instrumental independent Simple Sepsis Early Prognostic Score (SSEPS) to predict clinical adverse outcomes in patients with sepsis during ED admission or even prehospital.

Methods

Study design

This was a single-centre, retrospective cohort study designed to derive a novel SSEPS and validate it for predicting clinical adverse outcomes in patients with sepsis during the early stages in the ED. We obtained informed consent from all participants involved. Patients and/or the public were not involved in the design, conduct, reporting or dissemination plans of this research.

Study population

We obtained data from the West China Hospital sepsis database. Briefly, the database contains data from all adult patients (≥18 years) diagnosed with sepsis in the ED of the West China Hospital of Sichuan University between July 2015 and June 2016. All sepsis diagnoses were based on the Sepsis-3 criteria. We excluded patients who were pregnant, those with prehospital cardiac or respiratory arrest, and those using vasoactive drugs (eg, norepinephrine, dopamine or dobutamine) (online supplemental figure 1).

Development and validation sets

To both analyse multivariate risk predictors of short-term outcomes in sepsis and test the validity of the risk predictors in a separate population, we constructed development and validation sets. Based on the development cohort data, the mortality rate is approximately 20%. To satisfy this difference with 80% power at 5% significance (two-tailed), assuming that the predicted value of the scoring system for death is higher than the area under the curve (AUC) value of 0.75, at least 24 deaths are required, and the number of patients in the validation group was 120. Therefore, we randomised all patient data into one of the two sets, with approximately 70% in the development set and 30% in the validation set. We did not use the validation set until after obtaining the results of the multivariate regression model and the predicting score.

Data collection

We collected demographic characteristics, admission vital signs, medical history, laboratory data, imaging data and final in-hospital diagnoses from the database. Laboratory data included routine haemograms with white cell count, platelet count, haemoglobin and haematocrit. We extracted serum biochemical indicators such as creatinine, urea nitrogen, cystatin-C, total bilirubin, alanine aminotransferase, aspartate aminotransferase and albumin levels. D-dimers were measured using a Sysmex CA-7000 analyser (Siemens Healthcare Diagnostics) and procalcitonin level was tested using Cobas S6000 Hitachi (Roche Diagnostics). We calculated the estimated glomerular filtration rate (eGFR) in millilitres per minute per 1.73 m2, according to the Modification of Diet in Renal Disease equation.13 The treating physicians estimated the scoring systems for sepsis, including Glasgow Coma Scale (GCS), qSOFA, MEWS, MEDS, APACHE II and SOFA, according to patients’ baseline data after admission to the ED. Patients with GCS score less than 13 were identified to have altered consciousness.14 15

Endpoints and follow-up

All patients in this study were followed up for 28 days by an emergency doctor in our department using a structured telephone interview. We contacted the relatives if the patients could not be contacted directly. We obtained all other data from the hospital medical records. Our primary endpoint was 28-day all-cause mortality, and the secondary endpoint was incidence of ICU admission, mechanical ventilation and shock. ICU admission was defined as patients admitted to the ICU due to severe dystrophy, severe hydroelectrolyte balance and unstable vital signs and need for organ support. Mechanical ventilation was defined as patients who require non-invasive ventilation or invasive ventilation due to hypoxic respiratory failure. Shock was defined as patients with hypotension that persists despite adequate fluid resuscitation and requires vasopressors to support perfusion.

Statistical analysis

We used the median to replace the laboratory variables missing less than 5% and if missing variables greater than 5% were not analysed at baseline. We presented continuous variables as mean (SD) for normally distributed data or as median and IQR for non-normally distributed data. We carried out continuous variable analyses using an independent sample t-test or Mann-Whitney U test. We presented categorical variables as frequencies (proportions) using χ2 or Fisher’s exact test as appropriate.

In the development cohort, we selected physiological indicators in the univariate logistic regression analyses to obtain candidate predictors of 28-day mortality and selected those for our multivariate logistic regression model. We chose age, heart rate, respiratory rate, altered consciousness, systolic blood pressure (SBP), diastolic blood pressure (DBP), oxygen saturation (SPO2) and temperature with p<0.05 to set up the SSEPS scoring system. There were no missing data on vital signs and consciousness, which are basic information for each admitted patients, and thus no imputation would be required for development/validation of SSEPS. We categorised SSEPS predictors based on median (IQR) cut-off values. We divided all unstandardised coefficients in the logistic regression model by the smallest coefficient to normalise them, and we set the lower risk of each variable category as the reference value group. We finally calculated the SSEPS score by a simple arithmetic sum of the integers assigned to the satisfied criteria. To examine overfitting of the model, we used the Hosmer-Lemeshow goodness of fit test and the observed and expected frequencies. We also constructed receiver operating characteristic curves and calculated the AUC for SSEPS to assess its discriminating ability as well as those of other scoring systems. We used the DeLong test to compare the AUCs of these scores, comparing the discriminating ability of SSEPS and other scores on 28-day mortality.16 We used data from the validation cohort to obtain a temporal score validation.

We analysed discriminating ability using the net reclassification improvement (NRI) and integrated discrimination improvement (IDI) to estimate the degree by which the SSEPS improved predictive ability when compared with those of other prognostic scoring systems.17 We take 5% as a cut-off value for low risk of mortality. With mortality around 20% in our study, in the NRI analysis, patients with risk of mortality less than 5%, 5%–20% and more than 20% were defined as low-risk, moderate-risk and high-risk groups, respectively. A decision curve analysis (DCA) suggested by Vickers and Elkin18 was applied to quantify the clinical usefulness of the SSEPS and to compare it with other scoring systems. We applied a Kaplan-Meier test to describe the survival estimates for our three defined risk groups (low, medium and high risk) by splitting the scoring system into tertiles. We considered two-sided p values <0.05 as statistically significant. We performed data analyses using SPSS V.21.0, Stata V.13.0 and R V.3.5.1 (R Foundation for Statistical Computing, Vienna, Austria) software.

Results

Patient population

In total, we collected and analysed data from 821 patients with a mean age of 57.5±17.8 years; 64.3% were men and 173 patients (21.1%) died within the 28-day follow-up (online supplemental figure 1). We randomly assigned 70% of the patients to the development group (n=585) and 30% to the validation group (n=236). We found no significant differences in baseline characteristics, vital signs, laboratory examination values, incidence of infection site or endpoint event occurrences among the patients in both groups (online supplemental table 1).

Baseline characteristics in the development group

The development group included 585 patients (62.4% men) with a mean age of 59.3±17.3 years and a large proportion (57.6%) had suspected pneumonia. Of these patients, 125 (21.4%) died during the 28-day follow-up period.

Patients who died were more likely to be older, have higher respiratory rates, have higher serum lactate, urea nitrogen, creatinine, cystatin-C and eGFR levels, and have higher APACHE Ⅱ, MEWS, MEDS, SOFA and qSOFA scores, in addition to lower levels of pH, platelet count, haemoglobin, haematocrit, albumin, GCS and fibrinogen, compared with patients who survived (table 1).

Table 1

Comparison between survivors and non-survivors in the development cohort

Logistic regression analysis for 28-day mortality in the development group

Age, respiratory rate, heart rate, altered consciousness, SBP, DBP and SPO2 were selected in the univariate analysis. The multivariate logistic regression analysis showed that age (OR, 1.032; 95% CI 1.008 to 1.056; p=0.010), heart rate (OR, 1.055; 95% CI 1.010 to 1.100; p=0.040), respiratory rate (OR, 1.042; 95% CI 1.003 to 1.083; p=0.034) and altered consciousness (OR, 3.606; 95% CI 2.296 to 5.664; p<0.001) were independent predictors of 28-day sepsis mortality (table 2).

Table 2

Univariate and multivariate logistic regression analyses in the development group

Construction and scoring of SSEPS in the development group

In order to construct the SSEPS, we selected independent risk factors such as age, heart rate, respiratory rate and altered consciousness in the logistic regression. These predictors were classified into quartiles. Online supplemental figure 2 shows that patients aged ≤46 years and with heart rate ≤95 beats per minute, respiratory frequency ≤21 beats per minute or altered consciousness had a significantly lower mortality than other groups. Thus, we took the lowest risk group as reference. Table 3 shows the risk factors in the SSEPS model and the points derived for each category according to β coefficients. The SSEPS score ranged from 0 to 21 points for 28-day sepsis mortality. We classified risks based on total scores (0–6 points, low; 7–9 points, medium; and 10–21 points, high); the 28-day sepsis mortality for these groups was 3.8%, 13.9% and 43.6%, respectively, with a statistically significant difference (p<0.001).

Table 3

Logistic regression analysis of the correlations between 28-day mortality and clinical elements of patients with sepsis in the development cohort

Goodness of fit of the SSEPS model

Mortality (53.5% vs 28.6% vs 12.5%) and incidence of ICU admission (76.7% vs 52.6% vs 29.5%, p<0.001), mechanical ventilation (86.0% vs 53.1% vs 30.4%, p<0.001) and shock (37.2% vs 16.4% vs 5.5%, p<0.001) in the high-risk group were higher than those in the moderate-risk and low-risk groups (online supplemental table 2). The Hosmer-Lemeshow goodness of fit statistic for SSEPS in the development group was 0.669 (figure 1A). The AUC of the SSEPS in predicting 28-day sepsis mortality was 0.762 (95% CI 0.702 to 0815; p<0.001), similar to those of SOFA (AUC: 0.762; 95% CI 0.701 to 0.814; p>0.05) and APACHE II (AUC: 0.761; 95% CI 0.602 to 0.814; p>0.05), higher than those of MEDS (AUC: 0.724; 95% CI 0.662 to 0.780; p>0.05) and qSOFA (AUC: 0.695; 95% CI 0.632 to 0.753; p>0.05), and higher than that of MEWS (AUC: 0.656; 95% CI 0.592 to 0.717; p=0.03; online supplemental figure 3).

Figure 1

Calibration histogram of SSEPS for predicting 28-day sepsis mortality based on logistic regression in the development (A) and validation (B) groups. SSEPS, Simple Sepsis Early Prognostic Score.

Validation of the SSEPS model in the validation group

In the validation group, mortality (43.6% vs 13.9% vs 3.8%) and incidence of ICU admission (60.3% vs 32.9% vs 22.8%, p<0.001), mechanical ventilation (67.9% vs 29.1% vs 25.3%, p<0.001) and shock (32.1% vs 11.4% vs 6.3%, p<0.001) in the high-risk group were higher than those in the moderate-risk and low-risk groups (online supplemental table 3). The Hosmer-Lemeshow goodness of fit statistic for SSEPS in the validation group was 0.522 (figure 1B). The AUC of the SSEPS in predicting 28-day sepsis mortality was 0.762 (95% CI 0.686 to 0.838; p<0.001), similar to those of SOFA (AUC: 0.745; 95% CI 0.692 to 0.798; p>0.05) and APACHE II (AUC: 0.750; 95% CI 0.681 to 0.819; p>0.05) and higher than those of qSOFA (AUC: 0.695; 95% CI 0.607 to 0.782; p<0.01), MEWS (AUC: 0.656; 95% CI 0.567 to 0.746; p<0.01) and MEDS (AUC: 0.724; 95% CI 0.646 to 0.802; p<0.01) (figure 2).

Figure 2

AUC of SSEPS and other predicting scoring systems for 28-day sepsis mortality in the validation group. APACHE II, Acute Physiology and Chronic Health Evaluation; AUC, area under the curve; MEDS, Mortality in the Emergency Department Sepsis; MEWS, Modified Early Warning Scoring; qSOFA, quick Sequential Organ Failure Assessment; Ref, reference; SOFA, Sequential Organ Failure Assessment; SSEPS, Simple Sepsis Early Prognostic Score;ROC, Receiver operating characteristic.

Since the AUCs of the SSEPS and SOFA systems for predicting 28-day mortality were identical and the highest of all the scoring systems, we further analysed NRI, IDI and DCA to compare their discriminating abilities. SSEPS did not show improvement in prognostic performance compared with SOFA (table 4). Six patients who died were reclassified to moderate-risk or high-risk group and seven who survived were reclassified to low-risk or moderate-risk group; thus, the NRI between SSEPS and SOFA was −0.088 (p=0.42) and the IDI was −0.039 (p=0.27). However, the DCA showed that the net benefit of SSEPS was higher than those of SOFA or APACHE II at any threshold (figure 3).

Figure 3

Decision curve analysis for SSEPS, SOFA and APACHE II in the validation group. APACHE II, Acute Physiology and Chronic Health Evaluation; SOFA, Sequential Organ Failure Assessment; SSEPS, Simple Sepsis Early Prognostic Score.

Table 4

NRI of SOFA and SSEPS in predicting 28-day sepsis mortality in the validation group

Discussion

In this study, we tested a simple early prognostic score for sepsis consisting of age, heart rate, respiratory rate and altered consciousness measured in the ED. Our data show that patients with higher SSEPS scores had higher risk of 28-day all-cause mortality and incidence of ICU admission, mechanical ventilation and shock. Irrespective of the severity of sepsis, the prognostic value of the SSEPS was superior to that of qSOFA, MEWS, MEDS, SOFA and APACHE II. Therefore, our SSEPS seems to be a powerful tool to stratify high-risk patients with sepsis during the early phases in the ED, and we could assess the severity of sepsis by monitoring age, heart rate, respiratory rate and altered consciousness during the early phases.

Sepsis is considered a severe and life-threatening condition due to organ dysfunctions caused by inappropriate host responses to infections.1 Each hour of delay in treatment initiation increases the risk of sepsis-related mortality by approximately 8%.19 Early and quick risk stratification plays a vital role in the early management of sepsis, and many prognostic scoring systems have been put in place to stratify patients with sepsis in the ED according to their risk. The qSOFA has been recommended in the guidelines for screening patients with infections for organ dysfunction, but has shown poor sensitivity and specificity in predicting 28-day mortality in ED patients with sepsis.20 Studies have also shown that MEDS and MEWS scores predict adverse outcomes in patients with sepsis, but with limited risk stratification power.11 21 22 SOFA and APACHE II scores consist of multiple physiological and haematological variables and are powerful in predicting risk among patients with sepsis; however, they are so complex that they cannot be used to continuously monitor severity of patients outside the ICU or during the early phases in the ED. Therefore, a novel and powerful prognostic score that can be applied in the ED is needed. In our study, the novel SSEPS, which considers age, heart rate, respiratory rate and altered consciousness, could independently and continuously predict mortality and incidence of ICU admission, mechanical ventilation and shock in patients with sepsis during the early phases in the ED.

Age has been traditionally considered a risk factor for sepsis mortality probably due to the associated high rate of comorbidities, immune weakening, poorer preadmission status and high colonisation with Gram-negative bacteria.23 24 Patients aged 80 and over have higher in-hospital mortality than those between 65 and 79 years, and age is strongly associated with sepsis in-hospital mortality.25 In our study, age was also independently associated with 28-day mortality.

Cardiac dysfunction due to irreversible cardiac damage in sepsis is one of the main causes of poor prognosis in patients with sepsis. Heart rate and left ventricular ejection fraction increase during the early stages of sepsis, but cardiac output decreases due to increased vascular permeability, decreased preload and decreased peripheral circulatory capacity, leading to insufficient perfusion of important organs.26 Although heart rate was not independently associated with sepsis mortality in our study, we included it in our SSEPS model (accounting for a maximum of 3 points of the total score) due to the simplicity of its measurement and to the published results that associate it with outcomes.

Respiratory rate and consciousness state are two indicators of the qSOFA score proven to be associated with prognosis of patients with sepsis. In our study, respiratory rate and altered consciousness were significantly higher in patients who died than in those who survived, and both variables were independently associated with risk of mortality in patients with sepsis. During sepsis, acute lung injury or acute respiratory distress syndrome occurs often due to diffuse alveolar damage caused by epithelial barrier and endothelial dysfunctions and by the resultant pulmonary oedema.27 In our study, the proportion of pulmonary infections was higher than those of other systems. Central nervous system dysfunction typically manifests as obtundation or delirium.28 We chose what we considered the most important and easy physiological indicators for our SSEPS and achieved good predictive ability with the system.

Our study also has limitations. First, it was a single-centre, retrospective cohort study with a small population and we have not investigated the SSEPS for long-term outcomes. Second, we only recorded the variables for SSEPS on admission to our ED and we did not obtain recordings at different time points. A series evaluation of the SSEPS at different time points in our ED may provide more prognostic information. Third, the study lacked an external validation data set (only using internal validation), which may limit its rigour and confidence in findings.

Conclusions

SSEPS, a novel and easy-to-assess score, was an independent prognostic marker for patients with sepsis during the early phases in the ED, irrespective of severity of sepsis. The prognostic ability of the SSEPS in patients with sepsis was superior or not inferior to those of traditional prognostic sepsis scores. Therefore, SSEPS may be useful for emergency clinicians in risk stratification of patients with sepsis during the early phases of ED admission.

Data availability statement

Data are available upon reasonable request. The data sets generated and/or analysed during the current study are not publicly available, in accordance with China Ethics Committee of Registering Clinical Trials, but are available from the coauthor on reasonable request.

Ethics statements

Ethics approval

This study complied with the Declaration of Helsinki and was approved by the Human Ethical Committee of the West China Hospital of Sichuan University (number: 2019 (373)).

Acknowledgments

The authors would like to acknowledge the participants and their families for taking part in the study.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Contributors All authors have contributed significantly to the manuscript. BL, DL and YCa designed the study. YCs, YJ, JY and QZ provided critical feedback and helped shape the research. JY, DL and YL analysed and interpreted the data. BL, DL and JY wrote the original draft of the manuscript. YCa wrote, reviewed and edited the manuscript. All authors read and approved the final version of the manuscript.

  • Funding This work was supported financially by grants from the Science Foundation of Science and Technology Department of Sichuan (no. 2018RZ0139 and 2019JDRC0105) and Science and Technology Benefit People Project of Chengdu Municipality(2016-HM02-00099-SF).

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.