Article Text

Download PDFPDF

Original research
Comparison of SIRS criteria and qSOFA score for identifying culture-positive sepsis in the emergency department: a prospective cross-sectional multicentre study
  1. Lisette Mignot-Evers1,2,
  2. Vivian Raaijmakers1,
  3. Gerba Buunk3,
  4. Steffie Brouns4,
  5. Lorenzo Romano3,
  6. Thijs van Herpt3,
  7. Arvind Gharbharan3,
  8. Jeanne Dieleman5,
  9. Harm Haak2,4
  1. 1 Emergency Department, Máxima Medical Centre, Veldhoven, The Netherlands
  2. 2 Department of Health Services Research, Maastricht University, Care and Public Health Research Institute, Maastricht, The Netherlands
  3. 3 Internal Medicine, Amphia Ziekenhuis, Breda, The Netherlands
  4. 4 Internal Medicine, Máxima Medical Centre, Veldhoven, The Netherlands
  5. 5 Máxima MC Academy, Máxima Medisch Centrum Veldhoven, Veldhoven, The Netherlands
  1. Correspondence to Dr Lisette Mignot-Evers; l.evers{at}mmc.nl

Abstract

Objective To compare the daily practice of two emergency departments (ED) in the Netherlands, where systemic inflammatory response syndrome (SIRS) criteria and quick Sequential Organ Failure Assessment (qSOFA) score are used differently as screening tools for culture-positive sepsis.

Design A prospective cross-sectional multicentre study.

Setting Two EDs at two European clinical teaching hospitals in the Netherlands.

Participants 760 patients with suspected infection who met SIRS criteria or had a qualifying qSOFA score who were treated at two EDs in the Netherlands from 1 January to 1 March 2018 were included.

Methods SIRS criteria and qSOFA score were calculated for each patient. The first hospital treated the patients who met SIRS criteria following the worldwide Surviving Sepsis Campaign protocol. At the second hospital, only patients who met the qualifying qSOFA score received this treatment. Therefore, patients could be divided into five groups: (1) SIRS+, qSOFA−, not treated according to protocol (reference group); (2) SIRS+, qSOFA−, treated according to protocol; (3) SIRS+, qSOFA+, treated according to protocol; (4) SIRS−, qSOFA+, not treated according to protocol; (5) SIRS−, qSOFA+, treated according to protocol.

Primary and secondary outcome measures To prove culture-positive sepsis was present, cultures were used as the primary outcome. Secondary outcomes were in-hospital mortality and intensive care unit (ICU) admission.

Results 98.9% met SIRS criteria and 11.7% met qSOFA score. Positive predictive values of SIRS criteria and qSOFA score were 41.2% (95% CI 37.4% to 45.2%) and 48.1% (95% CI 37.4% to 58.9%), respectively. HRs were 0.79 (95% CI 0.40 to 1.56, p=0.500), 3.42 (95% CI 1.82 to 6.44, p<0.001), 18.94 (95% CI 2.48 to 144.89, p=0.005) and 4.97 (95% CI 1.44 to 17.16, p=0.011) for groups 2–5, respectively.

Conclusion qSOFA score performed as well as SIRS criteria for identifying culture-positive sepsis and performed significantly better for predicting in-hospital mortality and ICU admission. This study shows that SIRS criteria are no longer necessary and recommends qSOFA score as the standard for identifying culture-positive sepsis in the ED.

Trial registration number NL8315.

  • accident & emergency medicine
  • general medicine (see internal medicine)
  • adult intensive & critical care
  • internal medicine

Data availability statement

Data are available in a public, open-access repository. Extra data can be accessed via the Dryad data repository at http://datadryad.org/withthedoi.org/10.5061/dryad.d2547d82n.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Strengths and limitations of this study

  • Large sample size of 750 patients.

  • A prospective multicentre study from two clinical teaching hospitals.

  • Analysis of five different patient groups to compare systemic inflammatory response syndrome (SIRS) criteria to quick Sequential Organ Failure Assessment (qSOFA) score.

  • This study did not include patients with negative SIRS criteria and qSOFA score.

  • Data collection was performed during the winter season.

Introduction

Sepsis is one of the main causes of death in critically ill patients worldwide and causes an estimated 8 million deaths annually.1–3 Sepsis is a complex syndrome that is difficult to identify since it may present in many ways and can change quickly over time.2–4 For most patients with sepsis and are critically ill, the emergency department (ED) is the main setting for the first diagnosis and treatment.5 6 Recent studies show the importance of immediate and adequate treatment with fluids and antibiotics in order to reduce the amount of severe morbidity and mortality.2 3 7 8 A quick and accurate ED screening system to identify these patients therefore is of vital importance.

Since 1992, the systemic inflammatory response syndrome (SIRS) criteria have been used worldwide to identify sepsis and estimate the risk of mortality.6 SIRS criteria are positive for sepsis if two or more of the following vital signs are present: heart rate >90/min; respiratory rate >20/min; temperature <36°C or >38°C; or leucocytosis <4x10ˆ9/L or >12x10ˆ9/L, combined with a suspicion of infection.2 9–11 The SIRS criteria, however, remain negative in one out of eight patients with sepsis.1 3 Additionally, the SIRS criteria lead to many false positives which makes their use less than ideal for identifying sepsis.6 In 2016, the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) Task Force came with a new definition for sepsis: ‘life-threatening organ dysfunction caused by a deregulated host response to infection’.2 3 6 9 10 12 13 Based on six organ systems, new criteria, the Sequential Organ Failure Assessment (SOFA), were developed.14 For bedside use, the criteria were translated into the quick SOFA (qSOFA) score, which is positive in the case of a score ≥2 (systolic tension ≤100 mm Hg; respiratory rate ≥22/min; or Glasgow Coma Scale (GCS) score <15).14–16 Although qSOFA was initially not developed as a screening instrument but as a risk stratification for sepsis prognosis, it is used as such in many Dutch EDs.17

Several studies comparing the diagnostic properties of the SIRS criteria and the qSOFA score have led to conflicting conclusions.3 6 9 10 15 18–20 Some studies showed that the qSOFA score is not sensitive enough and favour the use of SIRS criteria in the ED.2 15 21 However, other studies have concluded that the qSOFA score is superior to the SIRS criteria because of better prognostic accuracy.9 10 12 22 Until now, most of the studies have investigated the difference between the SIRS criteria and the qSOFA score in intensive care units (ICU) or in specific patients, such as elderly patients or those suffering from pneumonia.2 3 6 12 The aim of our study is to compare the daily practice of two EDs in the Netherlands, where SIRS criteria and qSOFA score are used differently as screening tools for culture-positive sepsis.

Methods

Study design and setting

A prospective observational and cross-sectional multicentre study was performed at two different EDs in the Netherlands with a qualitatively comparable staffing. The ED of the Máxima MC in Veldhoven is staffed 24 hours/day and 7 days/week with emergency physicians (EPs) and residents in training and not in training to become EPs. Internal medicine is also staffed with residents available round the clock for the ED with the acute internist on call. Máxima MC is a teaching hospital with a capacity of 550 beds; there were 22 485 ED visits in total in 2018. The ED of the Amphia Hospital in Breda is staffed with residents and specialists with specific specialties such as surgery, internal medicine, pulmonary medicine and neurology. During office hours, there is an (acute) internist available on the ED who is on call during the evening and at night. Amphia Hospital is an 837-bed teaching hospital that fielded 33 460 ED visits in 2018. Both hospitals use the Netherlands Triage System (NTS) for triaging undifferentiated patients at presentation. The NTS is a five-level system used to categorise ED patients into levels of urgency: U1 (life threatening, requires immediate assessment), U2 (very urgent, requires evaluation within 10 min), U3 (urgent, requires evaluation within 1 hour), U4 (non-urgent, requires evaluation within 2 hours) and U5 (advice, requires evaluation within 4 hours).22 In both EDs, the last NTS category (U5) is not used.

Data collection

The study population comprised entirely patients aged 18 years and older at ≥2 of the SIRS criteria or ≥2 of the qSOFA score who visited the ED with a suspected infection or sepsis.11 16 These patients were triaged at level U1, U2 or U3 by the NTS23 and visited the ED for internal, pulmonary, gastrointestinal or urology medicine. The inclusion of patients was conducted by one medical student at Máxima MC and three residents and one medical student at Amphia Hospital. Data collection started at Máxima MC and was supplemented at Amphia Hospital until 750 patients were achieved and the patient input from both hospitals came balanced. Data were obtained manually from the electronic hospital records using structured electronic data collection forms. All data were deidentified and stored in a secure data management system.

As part of routine assessment for triage, vital signs including heart rate, blood pressure, respiratory rate and temperature were measured for each patient on arrival at the ED using the bedside monitor and a tympanic ear thermometer. Blood samples were taken for C-reactive protein (CRP) and leucocyte counts. Degree of consciousness was measured by using the GCS. The SIRS criteria and qSOFA score were calculated for each patient. As a measure of comorbidity, the Charlson Comorbidity Index was used.24 To prove culture-positive sepsis, cultures were performed at the local microbiology and pathology departments as part of the routine procedure. Cultures could be of any type (ie, blood cultures, urine cultures, sputum cultures, throat swaps, wound cultures and/or cultures of cerebrospinal fluid) and were taken to detect the presence of pathogens and identify the type of infection.

Patients received treatment that followed the worldwide Surviving Sepsis Campaign protocol, which consists of antibiotics, fluid resuscitation and oxygen supplementation. Both hospitals used the same treatment protocol. However, different criteria were used for the start of this protocol. According to Máxima MC, patients are treated for sepsis if they meet the SIRS criteria. In the Amphia Hospital, on the other hand, only patients who meet the requisite qSOFA score receive this treatment. Because of this, patients included in this study could be divided into five a priori groups: (1) SIRS positive, qSOFA negative, no sepsis protocol treatment (Amphia patients); (2) SIRS positive, qSOFA negative, sepsis protocol treatment (Máxima MC patients); (3) SIRS positive, qSOFA positive, sepsis protocol treatment (both hospitals); (4) SIRS negative, qSOFA positive, no sepsis protocol treatment (Máxima MC patients); (5) SIRS negative, qSOFA positive, sepsis protocol treatment (Amphia patients) (figure 1). Patients who were not treated according to the sepsis protocol (groups 1 and 4) received required treatment consisting of antibiotics and fluid resuscitation when the treating physician considered it necessary or when bacterial infection was suspected. For the other groups (groups 2, 3 and 5), the treating physician could deviate from antibiotic therapy when a viral infection was suspected. Group 1 was considered the reference group, or the healthiest patients potentially with sepsis, who were not treated according to sepsis protocol. To minimise bias, all healthcare providers were blinded regarding study aim and outcome measures.

Figure 1

Flow chart describing division of the study population into five different patient groups. qSOFA, quick Sequential Organ Failure Assessment; SIRS, systemic inflammatory response syndrome.

The primary outcome of interest was culture-positive sepsis. The secondary outcomes were in-hospital mortality and ICU admission.

We used the Standards for Reporting Diagnostic Accuracy (STARD) checklist when writing our report.25

Statistical analysis

All results were analysed using SPSS V.23 (SPSS). According to the sample size calculation, a sample of 750 patients was required to achieve >80% power, assuming a two-sided significance level of 5% and an anticipated effect size of 5.5%. We compared the daily practice of two EDs, where SIRS criteria and qSOFA score are used differently as screening tools, by calculating the positive predictive value (PPV). Differences in baseline characteristics between hospitals and study groups were tested using χ2 statistics and Mann-Whitney U tests for categorical and continuous variables, respectively. To test the differences in PPV between the two hospitals and between the SIRS criteria and the qSOFA score, the χ2 test was used. A Kaplan-Meier survival curve was generated and the log-rank test was used to test differences in in-hospital mortality between the five different patient groups. HRs for in-hospital mortality were calculated by using multiple Cox regression analyses including the study groups as predictor variables; in-hospital mortality and ICU admission were analysed using Cox regression analysis and logistic regression analysis, respectively. ORs were calculated with 95% CIs using logistic regression analysis. Missing data were categorised as ‘unknown’ and included in the analysis when suitable. In all analyses, a two-sided p value <0.05 was considered to be statistically significant.

Patient and public involvement

No patients were directly involved in developing the research question or design of the study. Patients were also not involved in conducting the study. There are no plans to disseminate the study results to individual participating patients.

Results

Patient characteristics

In total, 9321 patients visited the ED in one of the study hospitals between 1 January and 1 March 2018. Seven hundred and fifty patients who met the inclusion criteria were included in the study (figure 2). A total of 8571 patients were excluded. The mean age of the patients was 68.2 years (SD 15.8) and 48.6% were male. Intravenous antibiotic treatment was given to 69.5% of the patients with a median CRP level of 63 mg/L and leucocyte level of 12.1×109/L (table 1). Máxima MC contained patients with significantly less comorbidities, less heart failure, a higher number of dementias and a lower median CRP level compared with Amphia Hospital (p=0.001, p=0.002, p=0.014 and p=0.014, respectively). Comparison of other demographics, comorbidities, biomarkers and intravenous antibiotic treatment showed no significant difference between both hospitals.

Table 1

Patient characteristics

Figure 2

Flow chart describing inclusion of patients. ED, emergency department; GI, gastrointestinal; INT, internal; PUL, pulmonary; qSOFA, quick Sequential Organ Failure Assessment; SIRS, systemic inflammatory response syndrome; URO, urology.

Culture outcomes

98.9% of the patients met the SIRS criteria while 11.7% achieved a positive qSOFA score. There was no significant difference in the accuracy of the SIRS criteria and the qSOFA score for identifying culture-positive sepsis at the ED. The PPV of the SIRS criteria was slightly lower than the PPV of the qSOFA score—41.2% (95% CI 37.4% to 45.2%) vs 48.1% (95% CI 37.4% to 58.9%), respectively—but the difference was not statistically significant (table 2). The ORs for predicting culture-positive sepsis were not statistically significant either (1.5 (95% CI 0.9 to 2.4) vs 1.6 (95% CI 0.4 to 7.3), p>0.05, respectively).

Table 2

Logistic regression and descriptive statistics for analysing the accuracy of SIRS criteria and qSOFA score by identifying culture-positive sepsis

In-hospital mortality

For analysing the secondary outcome measure in-hospital mortality, the five different patient groups designed to compare both hospital screening tools were used to accommodate the effect of sepsis treatment. No significant difference was seen between the patient groups with positive SIRS criteria and a negative qSOFA score (table 3; groups 1 and 2) (HR 0.8, p>0.05). As previously mentioned, patient groups that were not treated according to the sepsis protocol (groups 1 and 4) were treated with intravenous antibiotics when the treating physician considered it necessary or when bacterial infection was suspected. For the other groups (2, 3 and 5), the EP could deviate from antibiotic therapy when a viral infection was suspected. It was therefore possible for 56.1% of group 1 (SIRS+, qSOFA−, not treated according to sepsis protocol) to receive intravenous antibiotics in comparison with 55.3% of group 2 (SIRS+, qSOFA−, treated according to sepsis protocol) (table 3). Both results show that when the SIRS criteria are positive and the qSOFA score is negative, there is no significant difference in in-hospital mortality regardless of the treatment according to sepsis protocol.

Table 3

Cox regression for analysing in-hospital mortality by five patient groups according to the different start of sepsis treatment protocol in both hospitals

The HRs for predicting in-hospital mortality were statistically significant in the patient groups with a positive qSOFA score (table 3; groups 3, 4 and 5) (3.3 (95% CI 1.7 to 6.2), 18.8 (95% CI 2.5 to 143.8) and 5.0 (95% CI 1.5 to 17.4), p<0.001, p=0.005 and p=0.011). This meant that when the qSOFA score was positive, the risk of in-hospital mortality significantly increased, as was also seen in the Kaplan-Meier survival curve (figure 3). In particular, the groups with a positive qSOFA score and negative SIRS criteria (groups 4 and 5, currently missed by the screening tool of hospital 1) showed poor results for in-hospital mortality. However, these groups are too small to make a significant statement about this.

Figure 3

Kaplan-Meier survival curve+log-rank test showing the affinity between in-hospital mortality and the five different patient groups. Study time ends by death (censored) or discharge from hospital. qSOFA, quick Sequential Organ Failure Assessment; SIRS, systemic inflammatory response syndrome; T+, treated according to sepsis protocol; T−, not treated according to sepsis protocol.

ICU admission

Table 4 shows the ORs for predicting ICU admission when patients with suspected infection or sepsis had a positive qSOFA score or SIRS criteria. The OR of the SIRS criteria was not statistically significant for predicting ICU admission (1.4 (95% CI 0.2 to 12.0), p>0.05), while the OR of the qSOFA score was (2.5 (95% CI 1.2 to 5.0), p=0.011). Therefore, patients with a positive qSOFA score had a significantly increased chance to be admitted to the ICU.

Table 4

Logistic regression for predicting ICU admission by positive SIRS criteria/qSOFA score

Discussion

Sepsis is a main cause of death in critically ill patients and is difficult to identify because of its complexity and changeability.1–3 A patient’s prognosis depends on early diagnosis and treatment, which mostly begins at the ED.6 8 This study compared the daily practice of two EDs in the Netherlands, where SIRS criteria and qSOFA score are used differently as screening tools for culture-positive sepsis, and investigated the predictive value on in-hospital mortality and ICU admission. Analysis of 750 patients showed a slightly lower PPV of the SIRS criteria compared with the qSOFA score, with no statistically significant difference. Additionally, the qSOFA score performed significantly better for predicting in-hospital mortality and ICU admission.

A meta-analysis recently published by Liu et al, including 24 trials with 121 237 patients, compared the diagnostic accuracy of the SIRS criteria and the qSOFA score by calculating the sensitivity and specificity.6 The meta-analysis showed no significant difference between both screening systems either (sensitivity of SIRS criteria and qSOFA score was 0.72 (95% CI 0.67 to 0.77) and 0.54 (95% CI 0.50 to 0.58), respectively; specificity was 0.71 (95% CI 0.69 to 0.73) and 0.67 (95% CI 0.65 to 0.68), respectively; p>0.05).6 Two other studies that compared the diagnostic accuracy of the SIRS criteria and the qSOFA score by calculating the PPV found no significant difference either.19 21

A large meta-analysis published by Song et al that compared the secondary outcome measures and included 23 trials with 146 551 patients found that the qSOFA score had significantly high specificity for early detection of in-hospital mortality and ICU admission (0.83 (95% CI 0.74 to 0.89) and 0.75 (95% CI 0.75 to 0.76), respectively).15 The secondary outcome measures from this study are comparable with the outcomes from this large meta-analysis.

A third meta-analysis conducted by Franchini et al included 12 studies with 80 941 patients and determined that the qSOFA score is a better predictor for poor outcomes and ICU admission as well.10 However, Franchini et al conclude that the SIRS criteria are a better clinical rule-out tool for overall mortality (sensitivity of SIRS criteria and qSOFA score was 0.86 (96% CI 0.78 to 0.92) and 0.51 (95% CI 0.38 to 0.65), respectively) and recommend the use of both screening systems together at the ED.10 Our study did not use the outcome measure overall mortality, but proved that it is not necessary to add the SIRS criteria for predicting in-hospital mortality. Our study found no significant difference between patient groups with positive SIRS criteria, regardless of the treatment according to sepsis protocol. Only when the qSOFA score became positive did the outcome measures of in-hospital mortality and ICU admission change significantly.

The strengths of this study include (1) the large sample size of 750 patients. (2) It was a prospective and multicentre study at two clinical teaching hospitals in the Netherlands using different sepsis screening tools. (3) The outcome measures were analysed using SIRS and qSOFA subgroups and five different patient groups following both hospital screening tools. (4) All healthcare providers were blinded for study aim and outcome measures to minimise the effect of performance bias; double blinding was not possible for this study. This study had also several limitations. (1) The data collection was performed by one author in hospital 1 and four authors in hospital 2, which could have affected the interobserver reproducibility. The researchers attempted to mitigate this issue by giving one uniform explanation to all authors who would collect data to make sure the data collection was performed in the same way at both hospitals. (2) Comparing both hospital screening tools, patients could be divided into five groups. Two of them contained such a small number of patients (group 4: n=2, group 5: n=6) that no significant statements can be made about these groups. These groups have a small number because it is unusual for patients to be SIRS negative and qSOFA positive given the type of vital parameters contained in these screening instruments. (3) A serious limitation of our study is that we could only calculate the PPVs of both screening systems, as this study did not include patients with negative SIRS criteria or qSOFA score. (4) Data collection started at Máxima MC and was supplemented at Amphia Hospital until 750 patients were achieved and the patient input from both hospitals came balanced. Because of this, 389 patients from Máxima MC and 361 patients from Amphia Hospital participated. (5) Patients treated in January and February 2018, winter season in the Netherlands, were included, which could have led to relatively more patients with cold and influenza symptoms and, therefore, proportionally more registrations for pulmonary medicine at the ED. This may be at the expense of generalisability.

The purpose of this study was to deliver results that would contribute to a uniform practicable standard to use as diagnostic and predictive tool for identification of culture-positive sepsis and risk for in-hospital mortality and ICU admission by critically ill patients at the ED. Initially, SIRS criteria have been developed in particular for identifying sepsis, while qSOFA score was meant as a risk stratification for sepsis prognosis.6 In many Dutch EDs, qSOFA is used in combination as both a screening instrument and as risk stratification.17 This study found the qSOFA score to be the best available screening tool at this moment for culture-positive sepsis, only when a clinical view of an experienced EP/acute internist is available at the ED; especially to avoid missing critically ill patients who do not fit qSOFA score but who certainly need treatment with antibiotics.

Future research is necessary to compare the difference between the SIRS criteria and the qSOFA score on other outcome measures, such as 30-day mortality, overall mortality, acute medical unit admission, general ward admission and health costs. More sizeable trials comparing the SIRS criteria and qSOFA score with other screening systems at the ED, such as the National Early Warning Score, would be necessary to provide a worldwide uniform standard for identifying sepsis at the ED.26

Conclusion

The qSOFA score performed as well as the SIRS criteria for identifying culture-positive sepsis at the ED and performed significantly better for predicting in-hospital mortality and ICU admission. The SIRS criteria appear to miss critically ill patients with poor outcomes who required treatment consisting of intravenous antibiotics and fluid resuscitation according to sepsis protocol. This study showed that the SIRS criteria are no longer necessary and recommends the qSOFA score as the standard for identifying culture-positive sepsis in undifferentiated patients at the ED, in hospitals where a clinical view of an experienced EP/acute internist is available.

Data availability statement

Data are available in a public, open-access repository. Extra data can be accessed via the Dryad data repository at http://datadryad.org/withthedoi.org/10.5061/dryad.d2547d82n.

Ethics statements

Ethics approval

The Medical Research Ethics Committee of Máxima Medical Centre approved the study (reference NL 17.180), which will be conducted according to the 1964 Helsinki Declaration and its later amendments. They declared that this study is not subject to the Dutch Medical Research Involving Human Subjects Act (WMO) and approved the procedure.

References

Footnotes

  • Contributors LME and HH had primary responsibility for protocol development, patient screening, enrolment, outcome assessment and preliminary data analysis. LME, HH, GB and SB drafted the study concept and the design of the protocol. VR, LR, TvH and AG contributed through acquisition of data. VR drafted the manuscript. LME, VR and JD interpreted the data. All previously cited authors contributed to the writing of the manuscript. LME takes responsibility for the paper as a whole.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.