You are here

Article Text

Article menu

Download PDFPDF

XML
Emergency medicine
Original research
Retrospective study on the possible existence of a treatment paradox in sepsis scores in the emergency department
  1. Jan Willem Uffen1,
  2. Harriet van Goor1,
  3. Johannes Reitsma2,
  4. Jan Jelrik Oosterheert3,
  5. Marieke de Regt4,
  6. Karin Kaasjager1
  1. 1Department of Internal Medicine and Acute Medicine, University Medical Centre Utrecht, Utrecht, The Netherlands
  2. 2Department of Epidemiology, Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands
  3. 3Department of Internal Medicine and Infectious Diseases, University Medical Center Utrecht, Utrecht, The Netherlands
  4. 4Department of Internal Medicine, Onze Lieve Vrouwe Gasthuis, Amsterdam, Noord-Holland, The Netherlands
  1. Correspondence to Dr Jan Willem Uffen; j.w.uffen{at}umcutrecht.nl

Abstract

Objective The quick Sequential Organ Failure Assessment (qSOFA) is developed as a tool to identify patients with infection with increased risk of dying from sepsis in non-intensive care unit settings, like the emergency department (ED). An abnormal score may trigger the initiation of appropriate therapy to reduce that risk. This study assesses the risk of a treatment paradox: the effect of a strong predictor for mortality will be reduced if that predictor also acts as a trigger for initiating treatment to prevent mortality.

Design Retrospective analysis on data from a large observational cohort.

Setting ED of a tertiary medical centre in the Netherlands.

Participants 3178 consecutive patients with suspected infection.

Primary outcome To evaluate the existence of a treatment paradox by determining the influence of baseline qSOFA on treatment decisions within the first 24 hours after admission.

Results 226 (7.1%) had a qSOFA ≥2, of which 51 (22.6%) died within 30 days. Area under receiver operating characteristics of qSOFA for 30-day mortality was 0.68 (95% CI 0.61 to 0.75). Patients with a qSOFA ≥2 had higher odds of receiving any form of intensive therapy (OR 11.4 (95% CI 7.5 to 17.1)), such as aggressive fluid resuscitation (OR 8.8 95% CI 6.6 to 11.8), fast antibiotic administration (OR 8.5, 95% CI 5.7 to 12.3) or vasopressic therapy (OR 17.3, 95% CI 11.2 to 26.8), compared with patients with qSOFA <2.

Conclusion In ED patients with suspected infection, a qSOFA ≥2 was associated with more intensive treatment. This could lead to inadequate prediction of 30-day mortality due to the presence of a treatment paradox.

Trial registration number 6916.

  • respiratory infections
  • intensive & critical care
  • internal medicine
  • accident & emergency medicine
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjopen-2020-046518

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • This study suggests the existence of a treatment paradox in the interpretation of the quick Sequential Organ Failure Assessment in sepsis care.

    • This study addresses the consequences of a treatment paradox when physicians or researchers are not aware of this phenomenon.

    • The concept of a treatment paradox in the field of sepsis remains a suggestion and cannot be proven.

    • Results of this study could have been influenced by delayed administration in the emergency department, leading to a possible underestimation of the effect of abnormal parameters on antbiotic administration.

    Introduction

    In absence of a gold diagnostic standard for sepsis, the Third International Consensus Definition Task Force (Sepsis-3) introduced the quick Sequential Organ Failure Assessment (qSOFA) as a prognostic tool for sepsis outside the intensive care unit (ICU).1 This score was developed and validated in large retrospective cohorts of patients with suspected infection to identify patients with an increased risk of dying.2–8 Based on the fundamental idea that sepsis is more severe and has a higher mortality rate than ordinary infections, this prognostic score was introduced as a proxy for a diagnostic tool to enhance sepsis recognition and guide treatment decisions.1

    In clinical medicine, prognostic scores are often used to identify patients at a certain risk of disease or unwanted outcome who might benefit from particular interventions. Ideally, these prognostic scores are developed using patient populations that have not been treated for the outcome of interest.9 However, this is often impossible due to ethical considerations. Furthermore, if the risk of disease or unwanted outcome is high, additional treatments are often initiated. This introduces the risk of a treatment paradox: when a strong prognostic factor of an adverse outcome triggers an effective treatment, the incidence of this outcome will be reduced.10 11 In this situation, the prognostic factor that initiated the treatment will appear to have a poorer prognostic performance than it actually has (figure 1). Two factors are essential for a treatment paradox to occur: (1) the prognostic factor has a strong relationship with the outcome and (2) when the prognostic factor is present, it triggers an effective treatment. The presence of this phenomenon may lead to a biased underestimation of adverse outcomes when prognostic scores, developed in a treated population, are applied to treatment-naive patients. In other words, patients with a high benefit of treatment may not be recognised properly, posing the risk of under treatment. The treatment paradox has been recognised in several medical fields, such as obstetrics and cardiovascular management.11–14

    Figure 1
    Figure 1

    Graphical representation of a possible treatment paradox when using qSOFA as a clinical score in sepsis care. The observed predictor outcome relation becomes a combination of the direct effect of a qSOFA ≥2 and of the indirect effect of oxygen administration, intravenous fluid therapy, antibiotic therapy, vasopressic therapy and lactate determination. qSOFA, quick Sequential Organ Failure Assessment.

    With the introduction of prognostic sepsis scores, the same problem may have been introduced in the field of sepsis. The qSOFA contains baseline characteristics that are likely to alarm the treating physician to act when abnormal, such as a high respiratory rate and a low systolic blood pressure.1 Moreover, qSOFA has been developed and validated in large retrospective cohorts in which patients with infection and sepsis were treated, thereby introducing the risk of a treatment paradox.

    In this study, we explore the potential existence of a treatment paradox in sepsis care by analysing the two essential factors required: (1) the relationship between the baseline qSOFA in the ED and 30-day mortality and (2) the relation between the qSOFA and subsequent treatment decisions within the first 24 hours after admission.

    Methods

    Study design and setting

    Analysis was performed on data from the prospective SePsis in the ACutely ill patients in the Emergency department (SPACE) cohort.15 This cohort consists of all consecutive patients that presented to the ED of the University Medical Center Utrecht for internal medicine with suspected infection between 13 September 2016 and 13 September 2018 and started shortly after the publication of Sepsis-3. This ED has an annual presentation number of 23 000 cases. Patient data collected in this cohort consist of clinical data on presentation in the ED and, if applicable, hospital admission. Furthermore, data is collected on diagnostic tests, treatment and follow-up up to 30 days. The SPACE cohort was registered in the Netherlands Trial Register.

    Population and data collection

    The SPACE cohort consists of all consecutive patients who meet the following criteria: (1) ≥18 years or older; (2) presentation at the ED with suspected infection defined by the treating physician in the ED; and (3) registration in the ED for the internal medicine department or its subspecialties: oncology, rheumatology, immunology, haematology, nephrology, endocrinology, gastroenterology, infectious disease and vascular medicine. All patients received standard care.

    All patients in the ED were treated according to a local sepsis clinical pathway. This pathway includes qSOFA and the Systemic Inflammatory Response Syndrome (SIRS) criteria. Patients suspected of sepsis by either positive SIRS, qSOFA or clinical suspicion receive care following different diagnostic and treatment steps based on international guidelines. For example, this includes resuscitation according to ABCDE method, diagnostic tests including blood culture withdrawal and administration of empiric antibiotics <1 hour. It also advices the use of vasopressor when a patient is not responding to fluid resuscitation of 1.5 L of intravenous fluid.

    The qSOFA was automatically calculated and reported in the electronic health record (EHR) system after the treating physician answered a non-obligatory question if he or she clinically suspected infection or sepsis. The calculation of the qSOFA was based on the first available recorded data and was considered positive in case of a qSOFA ≥2. Independent-trained physicians analysed all EHRs on documented suspected infection or sepsis. If the infection and sepsis questions were not answered by the treating physician, the independent physician marked the questions positive when respectively infection or sepsis was recorded by the ED physician as (differential) diagnosis in the ED patient record.

    General patient information, data on hospital or ICU or medium care (MC) admission, vital signs, laboratory testing and mortality were automatically extracted from the EHR.

    Data on comorbidities (categorised using the Charlson Comorbidity Index (CCI)),16 immunocompromised status and information on treatments administrated in the ED (administration of intravenous fluids and antibiotics, time to first antibiotics, oxygen therapy and use of vasopressor agents) were manually extracted from the EHR by researchers, using a predefined set of well-described definitions. If Glasgow coma scale was not registered, free text notes by the treating ED physician on the mental status were used. The (differential) diagnosis at admission and diagnosis at discharge were retrieved from the ED record and hospital discharge letter, respectively. These diagnoses were reviewed on correctness and accuracy by a standardised independent review of the medical record by the principal investigators, using predefined definitions. This review was based on symptoms, vital signs, laboratory results, radiology results and microbiology results and was standardised for most common infections.

    Outcome measurements

    The potential presence of a treatment paradox was investigated by: (1) evaluating the prognostic accuracy of the qSOFA for 30-day mortality within this cohort and by (2) analysing the relationship between a positive qSOFA and abnormal vital parameters imbedded in the qSOFA and the intensity of initiated therapy within the first 24 hours after ED presentation. Five therapy elements were investigated and were considered intensive (vs less intensive) in the following situations: (1) volume of intravenous fluid resuscitation within the first 3 hours on ED admission, (2) oxygen administration in the ED, (3) the use of vasopressors within the first 24 hours on admission, (4) a lactate measurement in the ED (which is suggested by the clinical pathway when at least one qSOFA parameter is abnormal) and (5) antibiotic treatment <1 hour. The latter was determined by calculating the time between arrival in the ED and the first antibiotic administration, registered by the nurse in the EHR. These interventions were used as a proxy for intensive therapy. Choice of these interventions do not suggest these are by means beneficial to this group of patients.

    Patient and public involvement

    Patients and/or public were not involved in this research.

    Statistical analysis

    For prognostic validation of the qSOFA for 30-day mortality a sensitivity/specificity for a qSOFA ≥2 and an area under receiver operating characteristics (AUROC) curve for qSOFA were calculated.

    Binary logistic regression analyses were performed to study the relationship between qSOFA as binary covariate and the choice of a positive qSOFA on initiated intensive therapy as outcome. A separate regression analysis was done per specific therapy element and for a combined outcome defined as receiving at least one form of intensive therapy, resulting in six analyses. Age and CCI were added as potential confounders to all models. Relationships were expressed as OR with 95% CI, and p values were derived. A p value <0.05 was considered as statistically significant. Multicollinearity of determinants was explored by deriving Spearman’s correlation coefficients. Hosmer-Lemeshow goodness of fit test was used to determine the fit of the extracted model.

    Results

    In total, 3178 consecutive patients were included. Patient characteristics are described in table 1. In this cohort, the most common infection diagnosed in the ED was a lower respiratory tract infection (20.7%, n=658). In 338 (10.6%) patients, an alternative non-infectious diagnosis was made during hospital admission or outpatient follow-up, most commonly a side effect of medication (24.0%, n=81). These patients were included in the analysis, because they were treated as suspected infectious in the ED.

    View this table:
    Table 1

    Patients characteristics of all patients and patients with a qSOFA ≥2

    Of all patients, 1089 (34.3%) were immunocompromised. Two-thirds (n=2134, 67.1%) of the patients were admitted to the hospital, and in 2174 (68.4%) patients, antibiotics were started in the ED. In total, 315 (9.9%) patients were admitted to the ICU or MC at any point during admission and 195 (6.1%) patients died within 30 days.

    Prognostic accuracy of qSOFA for 30-day mortality

    The risk of dying within 30 days after ED presentation increased from 6.1% (195/3178) in all patients to 22.6% (51/226) in the subgroup of patients with a baseline qSOFA ≥2. For 30-day mortality, a qSOFA ≥2 had a sensitivity of 0.26 (95% CI 0.20 to 0.33) and specificity of 0.94 (95% CI 0.93 to 0.95). The qSOFA had an AUROC curve of 0.68 (95% CI 0.61 to 0.75).

    Treatment paradox

    Patients with a baseline qSOFA ≥2 or with an abnormal individual element of qSOFA more frequently received any form of intensive therapy (table 2). Table 3 shows that a qSOFA ≥2 was independently associated with more frequently: (1) receiving antibiotics within 1 hour (OR 8.5 (95% CI 5.7 to 12.3)), (2) receiving more than 1 L intravenous fluids (OR 8.8 (95% CI 6.6 to 11.8)), (3) receiving vasopressor therapy (OR 17.3 (95% CI 11.2 to 26.8)), (4) receiving oxygen therapy (OR 6.4 (95% CI 4.7 to 8.7)) and (5) lactate measurement in the ED (OR 6.9 (95% CI 5.0 to 9.4)) compared with patients who had a qSOFA <2. Furthermore, there was an increased odds of 11.4 (95% CI 7.5 to 17.1) for patients with a qSOFA ≥2 of receiving at least one form of intensive therapy compared with patients with a qSOFA <2. Table 3 also shows that abnormal individual elements of the qSOFA were associated with more forms of intensive therapy.

    View this table:
    Table 2

    Therapy aggressiveness in patients with a qSOFA ≥2 and abnormal elements of the qSOFA and mortality rates per different therapy element

    View this table:
    Table 3

    Showing the association between a qSOFA ≥2 and the choice for intensive therapy

    Discussion

    With this article, we under scribe the theoretical existence of a treatment paradox in sepsis care in the ED by demonstrating that an abnormal qSOFA and abnormal individual elements of the qSOFA are associated with intensive treatment in the ED in patients with suspected infection. The problem lies in the fact that the prognostic qSOFA score was developed and validated in a cohort of patients that received treatment on clinical indication. Assuming these therapeutic interventions were, at least partly, effective in treating the suspected infection and sepsis, the incidence of adverse outcomes will have been reduced. As a consequence, the effect of strong predictors of mortality that trigger effective treatment will be underestimated. Briefly, the qSOFA is especially suited to identify patients that die despite treatment. This could potentially lead to an underestimation of adverse outcomes in treatment-naïve patients that actually benefit from treatment the most.

    An illustrative example of the risks of ignoring the treatment paradox in prognostic models can be found in the field of obstetrics.17 A retrospective study aimed to develop a prediction model for adverse maternal outcomes in suspected pre-eclampsia failed to identify maternal hypertension as a risk factor for adverse outcome due to a treatment paradox.13 In the study cohort, maternal hypertension was such a strong trigger for physicians to start an effective treatment that significantly less adverse events occurred. As a consequence, the statistical inference between maternal hypertension and adverse outcomes completely disappeared, and this well-known risk factor was not included in the prognostic model. However, ignoring a strong risk factor such as maternal hypertension in pre-eclampsia in new treatment-naïve patients would certainly lead to undertreatment and adverse outcomes. Although the treatment paradox effect in sepsis will probably be less strong than in pre-eclampsia, because sepsis is a far more heterogeneous syndrome with more heterogeneity in treatment effects, the results of our study support the presence of the effect.

    In the constant search for new screening tools in the field of sepsis, studies developing new or validating existing prognostic models rarely address the possible existence of a treatment paradox in sepsis recognition and treatment. Treatment paradoxes in the field of sepsis are likely to occur in different types of prognostic scores. Two meta-analyses on the qSOFA, Early Warning Scores and the SIRS criteria, only briefly recognise and address the risk of bias introduced by the treatment paradox in all studies included in their analysis, without discussing the further consequences. None of the individual studies in the meta-analyses discuss the possible existence of a treatment paradox.18 19 One retrospective analysis aimed to improve the National Early Warning Score by adding inflammatory blood marker addresses that their results could have been influenced by a treatment paradox, but this study was not specifically performed in patients at risk for sepsis.20 Future research on validating existing scores and developing new prediction models in sepsis should address these methodological issues by using other data sources like additional testing, follow-up, response to treatment or expert panels estimating sepsis risk and combining the imperfect information through latent class models and/or measurement error models. A recent framework by van Geloven et al21 is a useful starting point to evaluate the various types of predictions that can be made that explicitly incorporate the use of treatments.

    This study has several strengths. First, we address an important epidemiological phenomenon in using the qSOFA in sepsis care and that probably applies for all scoring systems used to predict patient outcomes that can be prevented by initiating effective treatment and support this by illustrating this with real patient data. Second, we discuss the consequences of a treatment paradox when clinicians or researchers are not aware of this phenomenon. Furthermore, the study population exists of a heterogeneous group of patients most at risk for developing sepsis. Therefore, results obtained from this study are applicable in daily practice.

    This study has several limitations. The true existence of a treatment paradox cannot be proven. We only provided evidence for the requirements for a potential treatment paradox in ED sepsis care by showing the prognostic accuracy of qSOFA and 30-day mortality and the relation between qSOFA and the initiation of intensive treatment decisions. Furthermore, only 35.9% of the patients who had a qSOFA ≥2 received antibiotic treatment within 1 hour, despite antibiotic treatment within 1 hour is mandatory according to local protocols. This is probably due to a delay in administrating antibiotics to the patients and registration of the antibiotic in the EHR. The actual antibiotic administration could have happened earlier than registration times. This probably led to an underestimation of the association between a qSOFA ≥2 and the administration of antibiotics within 1 hour after presentation in the ED.

    Lastly, the SPACE cohort started shortly after the introduction of Sepsis-3. This could have resulted in unfamiliarity with the qSOFA during early stages of the study.

    The qSOFA should be used as intended. It is a prognostic score for predicting short term (in-hospital) mortality in patients with suspected of confirmed infection in non-ICU settings. It should not be used as a diagnostic tool for the presence or absence of sepsis. Many studies have validated the qSOFA on their own patient cohorts, resulting in prognostic accuracy measures comparable with our findings. The question arises if sepsis recognition should rely on the use of a prognostic score with many intrinsic problems. Other clinical scores, like early warning scores, have been suggested of use in sepsis care.22 23 Future studies on these scores, aimed at external validation and/or updating, should explicitly indicate how treatments will be incorporated in these models and what kind of predictions will be made by the model. Other modelling approaches and other data sources are indispensable to build better models for predicting sepsis or mortality. Until then, clinicians should be aware of how a treatment paradox affects the interpretation of the qSOFA in sepsis care. Keeping this in mind, combined with careful consideration of its results within the complex of clinical data and clinical bedside judgement, the qSOFA (or any other clinical score) might still be helpful in recognising and treating sepsis.

    References

    1. ↵
      1. Singer M,
      2. Deutschman CS,
      3. Seymour CW, et al
      . The third International consensus definitions for sepsis and septic shock (sepsis-3). JAMA 2016;315:801–10.doi:10.1001/jama.2016.0287
      OpenUrlCrossRefPubMed
    2. ↵
      1. Seymour CW,
      2. Liu VX,
      3. Iwashyna TJ, et al
      . Assessment of clinical criteria for sepsis: for the third International consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 2016;315:762–74.doi:10.1001/jama.2016.0288pmid:http://www.ncbi.nlm.nih.gov/pubmed/26903335
      OpenUrlCrossRefPubMed
    3. ↵
      1. Askim Åsa,
      2. Moser F,
      3. Gustad LT, et al
      . Poor performance of quick-SOFA (qSOFA) score in predicting severe sepsis and mortality - a prospective study of patients admitted with infection to the emergency department. Scand J Trauma Resusc Emerg Med 2017;25:56. doi:10.1186/s13049-017-0399-4pmid:http://www.ncbi.nlm.nih.gov/pubmed/28599661
      OpenUrlPubMed
    4. ↵
      1. Williams JM,
      2. Greenslade JH,
      3. McKenzie JV, et al
      . Systemic inflammatory response syndrome, quick sequential organ function assessment, and organ dysfunction: insights from a prospective database of ED patients with infection. Chest 2017;151:586–96.doi:10.1016/j.chest.2016.10.057pmid:http://www.ncbi.nlm.nih.gov/pubmed/27876592
      OpenUrlCrossRefPubMed
    5. ↵
      1. González Del Castillo J,
      2. Julian-Jiménez A,
      3. González-Martínez F, et al
      . Prognostic accuracy of SIRS criteria, qSOFA score and GYM score for 30-day-mortality in older non-severely dependent infected patients attended in the emergency department. Eur J Clin Microbiol Infect Dis 2017;36:2361–9.doi:10.1007/s10096-017-3068-7pmid:http://www.ncbi.nlm.nih.gov/pubmed/28755060
      OpenUrlPubMed
    6. ↵
      1. Goulden R,
      2. Hoyle M-C,
      3. Monis J, et al
      . qSOFA, SIRS and news for predicting inhospital mortality and ICU admission in emergency admissions treated as sepsis. Emerg Med J 2018;35:345–9.doi:10.1136/emermed-2017-207120
      OpenUrlAbstract/FREE Full Text
    7. ↵
      1. Moskowitz A,
      2. Patel PV,
      3. Grossestreuer AV, et al
      . Quick sequential organ failure assessment and systemic inflammatory response syndrome criteria as predictors of critical care intervention among patients with suspected infection. Crit Care Med 2017;45:1813–9.doi:10.1097/CCM.0000000000002622pmid:http://www.ncbi.nlm.nih.gov/pubmed/28759474
      OpenUrlPubMed
    8. ↵
      1. Freund Y,
      2. Lemachatti N,
      3. Krastinova E, et al
      . Prognostic accuracy of sepsis-3 criteria for in-hospital mortality among patients with suspected infection presenting to the emergency department. JAMA 2017;317:301–8.doi:10.1001/jama.2016.20329
      OpenUrlCrossRefPubMed
    9. ↵
      1. Moons KGM,
      2. Royston P,
      3. Vergouwe Y, et al
      . Prognosis and prognostic research: what, why, and how? BMJ 2009;338:b375–20.doi:10.1136/bmj.b375
      OpenUrlFREE Full Text
    10. ↵
      1. Schuit E,
      2. Groenwold RHH,
      3. Harrell FE, et al
      . Unexpected predictor-outcome associations in clinical prediction research: causes and solutions. Can Med Assoc J 2013;185:E499–505.doi:10.1503/cmaj.120812pmid:http://www.ncbi.nlm.nih.gov/pubmed/23339155
      OpenUrlFREE Full Text
    11. ↵
      1. Pajouheshnia R,
      2. Peelen LM,
      3. Moons KGM, et al
      . Accounting for treatment use when validating a prognostic model: a simulation study. BMC Med Res Methodol 2017;17:103. doi:10.1186/s12874-017-0375-8pmid:http://www.ncbi.nlm.nih.gov/pubmed/28709404
      OpenUrlCrossRefPubMed
    12. ↵
      1. Westerhuis MEMH,
      2. Schuit E,
      3. Kwee A, et al
      . Prediction of neonatal metabolic acidosis in women with a singleton term pregnancy in cephalic presentation. Am J Perinatol 2012;29:167–74.doi:10.1055/s-0031-1284226pmid:http://www.ncbi.nlm.nih.gov/pubmed/21815125
      OpenUrlCrossRefPubMed
    13. ↵
      1. Cheong-See F,
      2. Allotey J,
      3. Marlin N, et al
      . Prediction models in obstetrics: understanding the treatment paradox and potential solutions to the threat it poses. BJOG 2016;123:1060–4.doi:10.1111/1471-0528.13859pmid:http://www.ncbi.nlm.nih.gov/pubmed/26806638
      OpenUrlCrossRefPubMed
    14. ↵
      1. Roffi M,
      2. Mukherjee D
      . Treatment-risk paradox in acute coronary syndromes. Eur Heart J 2018;39:3807–9.doi:10.1093/eurheartj/ehy577pmid:http://www.ncbi.nlm.nih.gov/pubmed/30247536
      OpenUrlPubMed
    15. ↵
      1. Uffen JW,
      2. Oomen P,
      3. de Regt M, et al
      . The prognostic value of red blood cell distribution width in patients with suspected infection in the emergency department. BMC Emerg Med 2019;19:76. doi:10.1186/s12873-019-0293-7
    16. ↵
      1. Charlson ME,
      2. Pompei P,
      3. Ales KL, et al
      . A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 1987;40:373–83.doi:10.1016/0021-9681(87)90171-8pmid:http://www.ncbi.nlm.nih.gov/pubmed/3558716
      OpenUrlCrossRefPubMedWeb of Science
    17. ↵
      1. von Dadelszen P,
      2. Payne B,
      3. Li J, et al
      . Prediction of adverse maternal outcomes in pre-eclampsia: development and validation of the fullPIERS model. Lancet 2011;377:219–27.doi:10.1016/S0140-6736(10)61351-7
      OpenUrlCrossRefPubMedWeb of Science
    18. ↵
      1. Maitra S,
      2. Som A,
      3. Bhattacharjee S
      . Accuracy of quick sequential organ failure assessment (qSOFA) score and systemic inflammatory response syndrome (SIRS) criteria for predicting mortality in hospitalized patients with suspected infection: a meta-analysis of observational studies. Clin Microbiol Infect 2018;24:1123–9.doi:10.1016/j.cmi.2018.03.032pmid:http://www.ncbi.nlm.nih.gov/pubmed/29605565
      OpenUrlPubMed
    19. ↵
      1. Hamilton F,
      2. Arnold D,
      3. Baird A, et al
      . Early warning scores do not accurately predict mortality in sepsis: a meta-analysis and systematic review of the literature. J Infect 2018;76:241–8.doi:10.1016/j.jinf.2018.01.002
      OpenUrlPubMed
    20. ↵
      1. Eckart A,
      2. Hauser SI,
      3. Kutz A, et al
      . Combination of the National early warning score (news) and inflammatory biomarkers for early risk stratification in emergency department patients: results of a multinational, observational study. BMJ Open 2019;9:e024636. doi:10.1136/bmjopen-2018-024636
    21. ↵
      1. van Geloven N,
      2. Swanson SA,
      3. Ramspek CL, et al
      . Prediction meets causal inference: the role of treatment in clinical prediction models. Eur J Epidemiol 2020;35:619–30.doi:10.1007/s10654-020-00636-1pmid:http://www.ncbi.nlm.nih.gov/pubmed/32445007
      OpenUrlCrossRefPubMed
    22. ↵
      1. Churpek MM,
      2. Snyder A,
      3. Han X, et al
      . Quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores for detecting clinical deterioration in infected patients outside the intensive care unit. Am J Respir Crit Care Med 2017;195:906–11.doi:10.1164/rccm.201604-0854OC
      OpenUrlCrossRefPubMed
    23. ↵
      1. de Groot B,
      2. Stolwijk F,
      3. Warmerdam M, et al
      . The most commonly used disease severity scores are inappropriate for risk stratification of older emergency department sepsis patients: an observational multi-centre study. Scand J Trauma Resusc Emerg Med 2017;25:91. doi:10.1186/s13049-017-0436-3

    Footnotes

    • Contributors JWU, HvG and MdR jointly conceived the hypothesis for the study. JWU wrote the study protocol. JWU and MdR analysed all electronic health records (EHR) on documented suspected infection or sepsis. JWU and HvG extracted data on comorbidities, immunocompromised status, any treatment in the emergency department and diagnosis at admission and discharge from the EHR. JWU and HvG undertook all data analyses. JWU and HvG drafted the manuscript. MdR, JJO, JBR and KK provided a critical review of the manuscript and provided advice. All authors read and approved the final manuscript. The corresponding author attest that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted, had full access to the data in the study and had final responsibility for the decision to submit for publication.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests JWU has received a consultancy grant from Becton Dickinson (BD) for educational presentations; no other relationships or activities that could appear to have influenced the submitted work.

    • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

    • Patient consent for publication Not required.

    • Ethics approval Approval of the study and use of the SePsis in the ACutely ill patients in the Emergency department cohort was granted by the University Medical Centre Utrecht institutional review board number 16/594.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data availability statement Data are available on reasonable request. The datasets generated and/or analysed during the present study are not publicly available, but they are available from the corresponding author on reasonable request.