Article Text

Download PDFPDF

External validation of the quick Sequential Organ Failure Assessment score for mortality and bacteraemia risk evaluation in Japanese patients undergoing haemodialysis: a retrospective multicentre cohort study
  1. Hiroki Nishiwaki1,2,3,
  2. Sho Sasaki2,4,
  3. Takeshi Hasegawa1,2,3,
  4. Fumihiko Sasai1,
  5. Hiroo Kawarazaki5,
  6. Shun Minatoguchi6,7,
  7. Daisuke Uchida8,
  8. Kenichiro Koitabashi9,
  9. Takaya Ozeki7,10,
  10. Fumihiko Koiwa1
  11. the JOINT-KD collaborators
    1. 1 Division of Nephrology, Department of Medicine, Showa University Fujigaoka Hospital, Yokohama, Japan
    2. 2 Center for Innovative Research for Communities and Clinical Excellence, Fukushima Medical University, Fukushima, Japan
    3. 3 Research Administration Center (SURAC), Showa University, Tokyo, Japan
    4. 4 Department of Nephrology/Clinical Research Support Office, Iizuka Hospital, Iizuka, Japan
    5. 5 Department of Nephrology, Inagi Municipal Hospital, Inagi, Japan
    6. 6 Department of Nephrology, Chubu Rosai Hospital, Nagoya, Japan
    7. 7 Department of Nephrology, Nagoya University Graduate School of Medicine, Nagoya, Japan
    8. 8 Department of Nephrology and Hypertension, Kawasaki Municipal Tama Hospital, Kawasaki, Japan
    9. 9 Division of Nephrology and Hypertension, Saint Marianna University School of Medicine, Kawasaki, Japan
    10. 10 Department of Nephrology, Toyohashi Municipal Hospital, Toyohashi, Japan
    1. Correspondence to Dr Takeshi Hasegawa; tahasegawa-npr{at}umin.net

    Abstract

    Objectives We aimed to examine the validity of the quick Sequential Organ Failure Assessment (qSOFA) score for mortality and bacteraemia risk assessment in Japanese haemodialysis patients.

    Design This is a retrospective multicentre cohort study.

    Setting The six participating hospitals are tertiary-care institutions that receive patients on an emergency basis and provide primary, secondary and tertiary care. The other participating hospital is a secondary-care institution that receives patients on an emergency basis and provides both primary and secondary care.

    Participants This study included haemodialysis outpatients admitted for bacteraemia suspicion, who had blood drawn for cultures within 48 hours of their initial admission.

    Primary and secondary outcome measures The primary outcome measure was overall in-hospital mortality. Secondary outcomes included 28-day in-hospital mortality and the incidence of bacteraemia diagnosed based on blood culture findings. The discrimination, calibration and test performance of the qSOFA score were assessed. Missing data were handled using multiple imputation.

    Results Among the 507 haemodialysis patients admitted with bacteraemia suspicion between August 2011 and July 2013, the overall in-hospital mortality was 14.6% (74/507), the 28-day in-hospital mortality was 11.1% (56/507) and the incidence of bacteraemia, defined as a positive blood culture, was 13.4% (68/507). For predicting in-hospital mortality among haemodialysis patients, the area under the receiver operating characteristic curve was 0.61 (95% CI 0.56–0.67) for a qSOFA score ≥2. The Hosmer-Lemeshow χ2 statistics for the qSOFA score as a predictor of overall and 28-day in-hospital mortality were 5.72 (p=0.02) and 7.40 (p<0.01), respectively.

    Conclusion On external validation, the qSOFA score exhibited low diagnostic accuracy and miscalibration for in-hospital mortality and bacteraemia among haemodialysis patients.

    • dialysis
    • infectious diseases
    • nephrology

    This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

    View Full Text

    Statistics from Altmetric.com

    Strengths and limitations of this study

    • This is the first study to assess the diagnostic performance of the quick Sequential Organ Failure Assessment (qSOFA) score for in-hospital mortality and bacteraemia among haemodialysis patients, according to the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis statement.

    • We could not precisely determine the performance of the qSOFA score in haemodialysis patients with symptoms that did not warrant blood culture evaluation because we did not evaluate the reasons blood was drawn for culture.

    • We used consecutive data of haemodialysis patients suspected of having bacteraemia, which is expected to increase the generalisability of our findings.

    • Our cohort contains patients who used antibiotics during the week leading up to the hospital visit, which could have decreased infection-related mortality and decreased the rate of positive blood cultures.

    • Our cohort was geographically and temporally different from the cohort used to derive the qSOFA criteria, which enabled us to perform a true external validation study.

    Introduction

    Patients undergoing haemodialysis are at high risk for bloodstream infections due to immunocompromised status and daily punctures required for vascular access.1 Moreover, the morbidity and mortality of bacteraemia are higher among haemodialysis patients than in the general population,2–10 as is the incidence of Staphylococcus aureus bloodstream infections.11 Therefore, appropriate diagnosis and timely treatment of bacteraemia are of critical importance in haemodialysis patients.

    While many risk stratification tools are available for the general population, their diagnostic accuracy is likely to differ when applied in specific populations. Adequate validation of population-specific diagnostic performance is particularly important in high-risk populations such as haemodialysis patients. For example, we previously reported that the Systemic Inflammatory Response Syndrome (SIRS) score has low sensitivity for predicting bloodstream infections in haemodialysis patients (SIRS score ≥2: sensitivity, 71.9%; specificity, 45.2%; positive likelihood ratio, 1.31; negative likelihood ratio, 0.62).12 These previous findings suggested that the prediction criteria for bacteraemia or sepsis, which are well-established for the general population, might have different diagnostic accuracy among haemodialysis patients.

    We also proposed a clinical prediction rule for bacteraemia among haemodialysis outpatients with suspicion of bacteraemia (BAC-HD).13 The BAC-HD score takes into account body temperature, heart rate, C-reactive protein levels, alkaline phosphatase levels and use of antibiotics within the week leading up to the assessment. A BAC-HD score ≥2 was useful for predicting bacteraemia in haemodialysis patients (sensitivity, 89.6%; specificity, 51.4%; positive likelihood ratio, 1.8; negative likelihood ratio, 0.2; area under the curve (AUC), 0.76).13

    The quick Sequential Organ Failure Assessment (qSOFA) score was introduced as a novel risk-stratification tool intended for use outside the intensive care unit (ICU). The qSOFA score is based on three clinical criteria: systolic hypotension, defined as a systolic blood pressure ≤100 mm Hg; tachypnea, defined as a respiratory rate ≥22 breaths/min; and altered mentation.14 In a previous study, the qSOFA score showed predictive validity (area under the receiver operating characteristic curve, 0.81; 95% CI 0.80 to 0.82) for sepsis in non-ICU patients with suspected infection identified as the combination of antibiotics use and body fluid cultures.14 Several studies have been conducted to validate the diagnostic performance of the qSOFA score among patients in various settings or with specific comorbidities.15–21 However, the validity of qSOFA for risk evaluation in haemodialysis patients has not been confirmed to date.

    In the present study, we aimed to examine the external validity of qSOFA as an easy-to-use tool for rapid evaluation of the risk of in-hospital death and bacteraemia in patients undergoing haemodialysis.

    Materials and methods

    Study design and participants

    Seven hospitals participated in this multicentre, retrospective cohort study of maintenance haemodialysis patients. The six participating hospitals are tertiary-care institutions that receive patients on an emergency basis and provide primary, secondary and tertiary care. The other participating hospital is a secondary-care institution that receives patients on an emergency basis and provides both primary and secondary care. The study results are reported in accordance with the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement.22

    The present study included consecutive haemodialysis patients with suspected bacteraemia who visited the outpatient department or emergency room between August 2011 and July 2013 and had blood drawn for cultures within 48 hours of their initial arrival at the hospital. The exclusion criteria of this study were as follows: age below 18 years; low frequency of haemodialysis (less than once per week); combination dialysis regimen (peritoneal dialysis and haemodialysis); admission within ≤2 weeks of haemodialysis initiation; and referral from another hospital.

    Outcome measures

    The primary outcome measure was the overall in-hospital mortality. Considering the findings of previous validation studies, 28-day in-hospital mortality was defined as the secondary outcome. Bacteraemia incidence was another secondary outcome measure in this study. Bacteraemia was diagnosed based on the results of blood cultures at the time of the patient’s visit. Specifically, the diagnosis of bacteraemia was made if the blood cultures were positive for any bacteria and there was no suspicion of contamination. Contamination was considered the most probable cause of positive blood culture results if only one of two sets of culture bottles was positive, or if all detected bacterial species were known to be common contaminants (ie, diphtheroids, Bacillus sp, Propionibacterium sp, micrococci, Corynebacterium sp, and coagulase-negative staphylococci). Finally, an external consensus panel of two physicians well trained in infectious diseases determined whether a culture was contaminated or not, based on the above definitions and their clinical expertise.

    Method of measurement

    The following data were extracted from the medical records: age; sex; dialysis vintage; cause of end-stage renal disease; vital signs at the time of the first visit, including body temperature, systolic blood pressure, pulse rate, respiratory rate, percutaneous oxygen saturation, Glasgow Coma Scale (GCS) score, and Japan Coma Scale (JCS) score23 24; comorbidities; type of vascular access; history of bacteraemia; medication use including antibiotics use within the week leading up to the hospital visit; and laboratory data at the time of the hospital visit, including white blood cell count, platelet count, serum albumin levels and C-reactive protein levels.

    A positive qSOFA result (qSOFA score ≥2) was defined in patients who fulfilled two or more of the following criteria at the same time: systolic blood pressure ≤100 mm Hg, respiratory rate ≥22 breaths/min and altered mentation. The qSOFA score ranges from 0 to 3, with each criterion being worth one point. The initial qSOFA scores were established according to the patients’ vital signs and mental status within 24 hours of arrival. Altered mentation was defined as a GCS score <13. If the JCS score was reported instead of the GCS score, the following equivalence was applied: a JCS score of 0 (alert) was considered to correspond to a GCS score of 15, while a JCS score of 300 (no motor response) was considered to correspond to a GCS score of 3.23 24 Converting JCS scores to GCS scores has not been validated. Thus, the other value of the JCS score was considered as missing data and handled using multiple imputation.

    Statistical analysis

    Data are presented as median values and IQRs for continuous variables, and as frequencies and percentages for categorical variables. The number of patients who had complete data for each qSOFA category is listed.

    In the analysis of the discrimination, calibration and performance of the qSOFA, primary imputation was employed to handle missing values for covariates, assuming that data were missing at random. To impute the missing values, we constructed multiple regression models including variables that could potentially explain the missing data, as well as variables correlated with the outcome. The results obtained across 100 imputed data sets were combined by averaging, and SE were adjusted to reflect both within-imputation and between-imputation variability. These estimates and their SE were combined using Rubin’s rules.

    For each qSOFA score cut-off (≥1, ≥2, and 3), the discrimination for predicting overall in-hospital mortality, 28-day in-hospital mortality and bacteraemia was assessed as the AUC considering data for all patients. The calibration of the risk score predictions was assessed by plotting observed proportions versus predicted probabilities and by calculating the Hosmer-Lemeshow χ2 statistic. Performance was evaluated as sensitivity, specificity, positive and negative likelihood ratio, and positive and negative predictive value.

    The minimum required sample size was estimated at 500 patients, based on the TRIPOD statement.22 All analyses were performed using the statistical software programmes Stata V.14.2 (StataCorp) and R V.3.4.1 (The R Foundation for Statistical Computing, https://www.r-project.org). Two-sided significance was set at 0.05.

    Patient and public involvement

    The public and patients were not involved in the development of the research question and outcome measures, study design or study recruitment. We will disseminate the final results to the study participants after they are published in a peer-reviewed journal.

    Results

    A total of 507 haemodialysis patients treated during the study period fulfilled the criteria for inclusion in this study. The overall in-hospital mortality in this population was 14.6% (74/507), whereas 28-day in-hospital mortality was 11.1% (56/507) and incidence of positive blood culture was 13.4% (68/507). In-hospital mortality rates were 5.2% among patients with a qSOFA score <2 and 29.6% among those with a qSOFA score ≥2. The corresponding mortality rates among patients with a positive blood culture were 3.9% and 35.3%, respectively. Table 1 provides a summary of the final diagnoses and corresponding mortality rates.

    Table 1

    Final diagnoses and corresponding mortality rates

    Of the 507 participants (median age, 73 years), 36.5% were women. The most common cause of chronic kidney disease was diabetic nephropathy (40.0%), while the most frequent route of vascular access was arteriovenous fistula (74.0%). The mean haemodialysis vintage was 61 months, and 16.4% of patients had taken antibiotics within the week leading up to the hospital visit (table 2).

    Table 2

    Characteristics of haemodialysis outpatients admitted for suspected bacteraemia (n=507)

    The most frequent pathogen in blood cultures was S. aureus, accounting for 28 cases of all bacteraemia cases (15 cases involving methicillin-sensitive S. aureus infection and 13 cases involving methicillin-resistant S. aureus infection). Klebsiella pneumoniae and Escherichia coli were the causal agent in 11 and 9 cases, respectively. Among the 68 patients with bacteraemia, 5 had polymicrobial infection (table 3).

    Table 3

    Pathogens causing bacteraemia in haemodialysis patients

    Of the 255 patients with complete data, 140 (54.9%), 91 (35.7%), 21 (8.2%) and 3 (1.2%) had qSOFA scores of 0, 1, 2 and 3 on hospital arrival. Among the patients with a qSOFA score of 1, tachypnea (respiratory rate ≥22 breaths/min) was the clinical criterion most commonly fulfilled (61.5%; 56/91). Among the patients with a qSOFA score of 2, the combination of altered mentation and tachypnea was the most common (47.6%; 10/21).

    For predicting in-hospital mortality in haemodialysis patients, the areas under the receiver operating characteristic curves were 0.59 (95% CI 0.53 to 0.66) for a qSOFA score ≥1, 0.61 (95% CI 0.56 to 0.67) for a score ≥2 and 0.51 (95% CI 0.49 to 0.53) for a score ≥3 (table 4). A summary of sensitivity, specificity, positive and negative likelihood ratios, and positive and negative predictive values for each qSOFA score cut-off is provided in table 4.

    Table 4

    Performance of the qSOFA score for predicting in-hospital mortality and bacteraemia in haemodialysis patients

    The Hosmer-Lemeshow χ2 statistics for the qSOFA score as a predictor of overall in-hospital mortality and 28-day in-hospital mortality were 5.72 (p=0.02) and 7.40 (p<0.01), respectively. The observed and predicted overall in-hospital mortality and 28-day in-hospital mortality were compared on calibration plots (figure 1). As the number of patients with a qSOFA score of 3 was too small, calibration analysis considered patients with a qSOFA score of 2 or 3 together (figure 1).

    Figure 1

    Observed and predicted in-hospital mortality among haemodialysis outpatients admitted for suspected bacteraemia. (A) Overall in-hospital mortality. (B) 28-day in-hospital mortality. qSOFA, quick Sequential Organ Failure Assessment. 

    Discussion

    In this study, we investigated the diagnostic accuracy of qSOFA for predicting in-hospital mortality and bacteraemia incidence in haemodialysis patients who presented to the hospital with suspicion of bacteraemia. Overall, the qSOFA criteria had low accuracy for predicting mortality and bacteraemia incidence among such haemodialysis patients.

    qSOFA has several advantages including easy bedside application, reliance on very few variables and no requirement for laboratory tests. However, of the recent studies on the validity of qSOFA in the emergency department setting,15–20 one reported poor sensitivity for qSOFA-based out-of-hospital identification of severe sepsis and septic shock.21 To the best of our knowledge, the present study represents the first investigation of the external validity of qSOFA for risk stratification of haemodialysis patients with suspicion of infection. Our results revealed that qSOFA exhibits low sensitivity and miscalibration for in-hospital mortality and bacteraemia in haemodialysis patients. In particular, the calibration plot revealed that a qSOFA score of 1 overestimated, while qSOFA score of 2 or 3 underestimated both overall and 28-day in-hospital mortality. There may be several reasons for such findings. First, infection with different causal pathogens may have different manifestations. We confirmed previous observations that S. aureus is the most common bacterial pathogen causing bloodstream infection among haemodialysis patients.11 Nevertheless, sepsis may have a different causal agent in haemodialysis patients than in the general population; the qSOFA score may not be able to fully account for different clinical presentations. Second, dialysis patients often present with immune system dysfunction and uraemia, as well as with comorbidities such as diabetes mellitus and connective tissue disorder,25 which may also affect clinical manifestation, further distinguishing haemodialysis patients from the general population and detrimentally affecting the performance of the qSOFA score. In addition, most dialysis patients have hypertension,26 and thus the incidence of hypotension, which is a key qSOFA criterion, may be low in haemodialysis patients with bacteraemia. Third, our baseline data were collected at the time of the hospital visit. One study revealed that a positive qSOFA result (qSOFA score ≥2) at hospital presentation and at 3, 6 and 24 hours after admission had poor sensitivity and specificity for predicting 28-day mortality.20 In other words, the timing of data collection may also affect the performance of the qSOFA score, especially in haemodialysis patients.

    Our study has several strong points. First, we included a multicentre cohort of haemodialysis patients, which reduced selection bias. Second, we used multiple imputation, which allowed us to investigate the entire cohort without having to exclude subjects with a relatively mild clinical presentation and thus without a detailed history or laboratory test findings, which would have induced information bias. Third, our cohort was geographically and temporally different from the cohort used to derive the qSOFA criteria, which enabled us to perform a true external validation study.

    Several limitations of the present study warrant mention. First, given that we did not evaluate the reasons blood was drawn for culture, we cannot precisely determine the performance of qSOFA in haemodialysis patients with symptoms (eg, fever) that did not warrant blood culture evaluation. However, because it is not possible to predict clinical judgement in such situations, we believe this lack of consideration actually increases the generalisability of our findings, as is the case with the study that developed the clinical prediction rule for bacteraemia in the general population.27 Second, our cohort contains patients who used antibiotics during the week leading up to the hospital visit, which could have affected their vital signs at presentation and decreased infection-related mortality and the rate of positive blood cultures. Third, we could not exclude the possibility that some patients had bacteraemia that was not detected on blood culture examination (ie, blood culture-negative bacteraemia), which is considered a limitation of blood culture. Finally, the exact time from hospital arrival to vital sign collection varied, which may have affected the qSOFA score and its relationship with patient prognosis. Employing routinely collected vital signs (eg, vital signs collected at the dialysis centre) for qSOFA score calculation might have provided a better reflection of bacteraemia status; however, vital sign data from the dialysis centres were not available to us at the time of the study.

    To summarise, our validation study revealed that, in haemodialysis patients, the qSOFA score exhibits low diagnostic accuracy and miscalibration for in-hospital mortality and bacteraemia. A new prediction score is needed for mortality risk stratification of haemodialysis patients. For bacteraemia risk stratification, the BAC-HD score may outperform the qSOFA score in terms of predicting value.

    Acknowledgments

    We thank the JOINT-KD collaborators and Naomi Ako (Division of Nephrology and Hypertension, Department of Internal Medicine, St. Marianna University School of Medicine, Kawasaki) for their intellectual support in the management of this study.

    References

    1. 1.
    2. 2.
    3. 3.
    4. 4.
    5. 5.
    6. 6.
    7. 7.
    8. 8.
    9. 9.
    10. 10.
    11. 11.
    12. 12.
    13. 13.
    14. 14.
    15. 15.
    16. 16.
    17. 17.
    18. 18.
    19. 19.
    20. 20.
    21. 21.
    22. 22.
    23. 23.
    24. 24.
    25. 25.
    26. 26.
    27. 27.
    View Abstract

    Footnotes

    • Contributors All authors have seen and approved the final version of the manuscript for publication. HN was responsible for the research idea and study design. HN, SS, FS, HK, SM, DU, KK, TO and JOINT-KD collaborators were responsible for data acquisition. HN, SS and TH were responsible for data analysis/interpretation. HN and SS were responsible for statistical analysis. TH and FK provided supervision or mentorship. Each author contributed important intellectual content during manuscript drafting or revision and accepts accountability for the overall work by ensuring that questions pertaining to the accuracy or integrity of any portion of the work are appropriately investigated and resolved.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests None declared.

    • Ethics approval The present study was approved by the ethics committees of St. Marianna Medical University (No. 2713) and Showa University Fujigaoka Hospital (No. 2014072) and was conducted in accordance with the ethical standards set out in the Declaration of Helsinki and its later amendments.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement You can use the anonymous data set used in this study, only if the reason for reusing the data is clear and it is determined that the reason is appropriate. We will keep the data for the data retention period applied for our Institutional Review Boards.

    • Collaborators The JOINT-KD collaborators included Masahide Furusho (Department of Nephrology, Kagoshima Medical Center, Kagoshima, Japan), Atsushi Nomura (Department of Immunology, Juntendo University School of Medicine, Tokyo, Japan), Hideaki Shimizu (Department of Nephrology, Daido Hospital, Nagoya, Japan), Akihiro Ryuge (Department of Nephrology, Nagoya University Graduate School of Medicine, Nagoya, Japan), Takahiro Imaizumi (Department of Nephrology, Nagoya University Graduate School of Medicine, Nagoya, Japan), Masaru Murasawa (Department of Nephrology, St. Marianna University School of Medicine Yokohama City Seibu Hospital, Yokohama, Japan), Yoshihiko Raita (Department of Nephrology, Okinawa Chubu Hospital, Okinawa, Japan), Yosuke Saka (Department of Nephrology, Kasugai Municipal Hospital, Kasugai, Japan), Masahiko Yazawa (Division of Nephrology and Hypertension, Department of Internal Medicine, St. Marianna University School of Medicine, Kawasaki, Japan), Maho Terashita (Division of Nephrology and Hypertension, Department of Internal Medicine, St. Marianna University School of Medicine, Kawasaki, Japan), Yasunori Suzuki (Division of Nephrology and Rheumatology, Ishikawa Prefectural Central Hospital, Kanazawa, Japan), Yoshiro Fujita (Department of Nephrology, Chubu Rosai Hospital, Nagoya, Japan), Minoru Murakami (Department of Nephrology, Saku Central Hospital , Saku, Japan), Yuichi Maruta (Division of Nephrology, Department of Medicine, Showa University Fujigaoka Hospital, Yokohama, Japan), Megumi Oikawa (Division of Nephrology, Department of Medicine, Showa University Fujigaoka Hospital, Yokohama, Japan), Hisako Saito (Division of Nephrology and Endocrinology, The University of Tokyo, Tokyo, Japan), Aya Mizukami(Division of Nephrology, Department of Medicine, Showa University Fujigaoka Hospital, Yokohama, Japan), Shinya Omiya (Division of Nephrology, Department of Medicine, Showa University Fujigaoka Hospital, Yokohama, Japan) and Yugo Shibagaki (Division of Nephrology and Hypertension, Department of Internal Medicine, St. Marianna University School of Medicine, Kawasaki).

    • Patient consent for publication Not required.

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.