Article Text
PDF
Abstract
Objective This study describes the availability of core parameters for Early Warning Scores (EWS), evaluates the ability of selected EWS to identify patients at risk of death or other adverse outcome and describes the burden of triggering that front-line staff would experience if implemented.
Design Longitudinal observational cohort study.
Setting District General Hospital Monaragala.
Participants All adult (age >17 years) admitted patients.
Main outcome measures Existing physiological parameters, adverse outcomes and survival status at hospital discharge were extracted daily from existing paper records for all patients over an 8-month period.
Statistical analysis Discrimination for selected aggregate weighted track and trigger systems (AWTTS) was assessed by the area under the receiver operating characteristic (AUROC) curve.
Performance of EWS are further evaluated at time points during admission and across diagnostic groups. The burden of trigger to correctly identify patients who died was evaluated using positive predictive value (PPV).
Results Of the 16 386 patients included, 502 (3.06%) had one or more adverse outcomes (cardiac arrests, unplanned intensive care unit admissions and transfers). Availability of physiological parameters on admission ranged from 90.97% (95% CI 90.52% to 91.40%) for heart rate to 23.94% (95% CI 23.29% to 24.60%) for oxygen saturation. Ability to discriminate death on admission was less than 0.81 (AUROC) for all selected EWS. Performance of the best performing of the EWS varied depending on admission diagnosis, and was diminished at 24 hours prior to event. PPV was low (10.44%).
Conclusion There is limited observation reporting in this setting. Indiscriminate application of EWS to all patients admitted to wards in this setting may result in an unnecessary burden of monitoring and may detract from clinician care of sicker patients. Physiological parameters in combination with diagnosis may have a place when applied on admission to help identify patients for whom increased vital sign monitoring may not be beneficial. Further research is required to understand the priorities and cues that influence monitoring of ward patients.
Trial registration number NCT02523456; Results.
- early warning scores
- aggregate weighted track and trigger systems
- single parameter track and trigger systems
- low-income and middle-income country
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
- early warning scores
- aggregate weighted track and trigger systems
- single parameter track and trigger systems
- low-income and middle-income country
Strengths and limitations of this study
Considers score feasibility in the context of real-world application.
Large, diverse dataset for a low-income and middle-income country setting.
Single centre.
No validation of the accuracy of the vital signs measured.
Introduction
Patients who suffer adverse events in hospital wards, such as cardiac arrest and death, often show changes in basic physiological parameters during the hours before the event. Based on this, Early Warning Scores (EWS) have been developed and widely implemented in high-income countries (HICs) with the aim of early identification of clinical deterioration.1
Both aggregate weighted track and trigger systems (AWTTS) and single-parameter track and trigger systems (SPTTS) use physiological measures and other clinically significant variables (eg, age) categorised and scored based on their degree of abnormality.2 AWTTS use a range of parameters which are weighted and calculated to form a composite and often complex score. SPTTS, while often including more than one parameter, allow for a single parameter to act as an independent trigger. Although less well evaluated, SPTTS tend to have acceptable specificities and negative predictive values (NPVs), but low sensitivities and positive predictive values (PPVs) for death or adverse events.2 Collectively, these EWS allow for stratification of patients at high risk of deterioration and for the objective evaluation of clinical status over time.3–5 In HICs, EWS are often implemented as part of a system connecting ward-based and critical care teams. Such systems often include a minimum of 12 hourly observation reporting, with the frequency of monitoring titrated according to score and/or clinician suspicion, and dedicated nurse-led rapid response teams, trained in critical care and resuscitation skills to respond in the event of clinical deterioration.3 6 7
Despite a multitude of EWS being developed and validated,—each with varying ability to predict patient deterioration,—eight basic parameters feature consistently within the scores8 9: age, respiratory rate, urine output, saturation of oxygen, temperature, systolic blood pressure, heart rate and a measure of mentation such as alert, response to voice, pain or unresponsive (AVPU) or glasgow coma scale (GCS).3 4
In low-income and middle-income countries (LMICs), availability of critical care remains limited and variable.10–13 Healthcare services, and in particular inpatient wards, are usually overcrowded, poorly equipped and understaffed, hindering the systematic and accurate monitoring of physiological parameters required for multiparameter EWS implementation and validation.14 15 Disease patterns and time to presentation differ from HICs. While data is limited, studies evaluating EWS in these settings show wide variation in performance.16 17
Thus, evaluation of EWS feasibility, including availability of physiological parameters, the burden of monitoring when triggered and an estimation of EWS performance, prior to advocating for their implementation in an LMIC setting is crucial.
This study describes the availability of core parameters for EWS, evaluates the ability of selected AWTTS and SPTTS (EWS) to identify patients at risk of death or other adverse outcome and describes the potential burden of monitoring that front-line staff would experience if implemented. It further explores the impact that diagnosis, and the relationships which hospital presentation and adverse outcome have on EWS performance. This study further seeks to evaluate EWS performance at selected time points during the patient’s journey and across the most common admission diagnoses.
This study was conducted at an LMIC district-level general hospital. At the time of data collection, there were no EWS in use at the hospital, and there was no escalation policy in response to adverse observation. Vital sign measurement was reported to be on admission and then 12 hourly. Decision to admit a patient to this hospital was made by attending physicians. The 370-bed hospital is situated in a rural province in Sri Lanka, and serves a community of approximately 500 000 people. Hospital facilities include renal dialysis, an intensive care unit (ICU) and maternity services.
Methods
All consecutive adult (age >17 years) patients admitted from May to December 2015 to District General Hospital Monaragala (DGHM) were prospectively included. Measures of pulse rate, respiratory rate, blood pressure, measure of consciousness (AVPU) and temperature, were all collected on admission and then two times per day (which is the usual frequency for recording these measures as described by the clinical team in this setting). The data was extracted daily from paper-based patient records by trained data collectors and entered into an electronic data capture system. All patients were followed up daily until hospital discharge. Diagnoses were coded as per the International Statistical Classification of Diseases and Related Health Problems, Tenth revision (ICD-10).18 The following were considered to be adverse outcomes; inhospital death, ICU admissions, clinical transfers to tertiary hospitals or to other ICUs in other hospitals, and cardiac arrest or cardiopulmonary resuscitation (CPR). Transfer to higher-level facilities and CPR both carry high mortality in this setting, hence their inclusion as adverse outcomes.19
The selection of AWTTS for evaluation in this study was based on studies reporting on the use of these systems in LMIC settings.16 17 20 Age, heart rate, respiratory rate, AVPU as a measure of mentation, systolic blood pressure and oxygen saturations were included in the evaluation. The GCS and urine output are not part of routine observation in this setting outside of the ICU, and therefore AWTTS including these parameters were not considered. VitalPAC Early Warning Score (ViEWS), Standardised Early Warning Score (SEWS), Modified Early Warning Score (MEWS) and Cardiac Arrest Risk Triage Score (CART) (online supplementary table 1) were included based on their superior performance for detecting cardiac arrest, mortality, ICU transfer and composite adverse outcomes in HIC studies.2 National Early Warning Score (NEWS) was included as it is the national tool recommended in the UK by The National Institute for Health and Care Excellence and is now widely adopted within the National Health Service, UK.1
Supplementary file 1
Selection of SPTTS parameters was based on the finding of a systematic review, which measured their sensitivity and specificity for predicting inhospital mortality. The reviewers reported a wide variation in performance, and concluded that SPTTS should be validated prior to implementation in a clinical setting.21 The selected parameters were high and low pulse rate, high and low respiratory rate, high and low systolic blood pressure and high and low temperature (online supplementary table 2). Oxygen saturations and a measure of mentation were not considered in single-parameter scores due to their limited availability in the study setting.19
Supplementary file 2
The performance of both AWTTS and SPTTS was evaluated with the missing values imputed as normal. Discrimination for the AWTTS was assessed by the area under the receiver operating characteristic (AUROC) curve for adverse outcomes and for death. Time from admission to adverse event was calculated. For patients with multiple events, only the first event was used. Availability of physiological parameters and the performance of EWS were evaluated at admission and at 24 hours prior to adverse event. The highest score in the 24 hours prior to discharge was calculated for patients who did not experience an adverse outcome.5 8
Clinically recommended cut-off values and corresponding sensitivity and specificity for MEWS, SEWS and ViEWS to predict death were applied.22 A NEWS score of 5 or more is used as trigger for escalation to senior review and the commencement of ward-level continuous bedside monitoring and was taken as the clinical cut-off.23 24 No clinical cut-off for CART was proposed in the original publication, with the premise that users should decide based on clinical application and resource availability. However, existing literature validating and comparing CART with MEWS used a cut-off of 20. This was, therefore, used in this analysis.2 All tests of significance for availability of observations considered a two-sided P value of less than 0.05 to be significant. AUROC values were considered poor when less than or equal to 0.70, adequate between 0.71 and 0.80, good between 0.81 and 0.90 and excellent at 0.91 or higher.25 Discriminatory power of the AWTTS was then reassessed using complete case analysis only.
The ability to predict mortality and adverse outcomes was assessed for each AWTTS and SPTTS class using sensitivity, specificity, PPV (the proportion of patients predicted to die who die) and NPV (the proportion of patients predicted to survive who survive). In addition, to further understand the feasibility of implementation of EWS in this setting, the burden of patients triggered for every correctly identified death and number needed to escalate (NNE) for the best performing of the AWTTS and SPTTS were also described.
Given a priori knowledge of observation reporting behaviours in the study setting, performance of the best performing of the SPTTS to predict death when applied at admission was then evaluated when using either single, two or three parameters. All possible combinations of single, two and three of the four parameters were described. Discrimination, sensitivity and specificity of the best performing of the AWTTS and SPTTS, respectively, were then described for the most common diagnostic groups (ICD-10 chapters).
All analyses were performed using Stata software V.13.1.26
Results
There were 16 386 adult in patient episodes to DGHM over the 8-month period. The characteristics, adverse events and outcomes for the study population are described in table 1. Of the 16 386 patients included, 502 (3.06%) had one or more adverse outcomes. A total of 102 (0.62%) cardiac arrests and 83 (0.51%) unplanned ICU admissions were reported, and 253 (1.54) patients were transferred to tertiary facilities. Total inhospital mortality was 149 (0.91%). The availability of observations over the 8-month period is described in figure 1.
Availability of observations during the 8-month study period. AVPU, alert, response to voice, pain or unresponsive; SBP, systolic blood pressure.
Summary of study population
Availability of physiological parameters on admission varied widely; heart rate 90.97% (95% CI 90.52% to 91.40%), systolic blood pressure 86.80% (95% CI 86.27% to 87.31%), respiratory rate 65.24% (95% CI 64.51% to 65.97%), saturation 23.94% (95% CI 23.29% to 24.60%) and assessment of mentation 32.89% (95% CI 32.17% to 33.61%). With the exception of temperature, availability of physiological parameters is significantly diminished after admission (P<0.05) (table 2). Availability of physiological parameters on admission was significantly greater in patients who had an adverse event when compared with those who did not (P<0.05).
Availability of observation reporting at admission and at 24 hours before event
Of the AWTTS assessed for their ability to discriminate death on admission and at 24 hours prior to death, only CART 0.781 (95% CI 0.744 to 0.818) and SEWS 0.702 (95% CI 0.656 to 0.748) had an AUROC of >0.70 (table 3). CART performed better (P<0.05) at predicting death both at admission and at 24 hours prior to death/discharge with missing values imputed as normal, when compared with the other four selected AWTTS. Two hundred and forty-nine patients (2%) would trigger (PPV of 10.44%) if CART was applied at the recommended clinical cut-off (figure 2). Discriminatory power of all AWTTS diminished when evaluated for their ability to predict death at 24 hours compared with admission (table 3). Fifty-two per cent of adverse events occurred within the first 48 hours of the patient’s admission (online supplementary figure).
Supplementary file 3
Discrimination of the selected AWTTS for deaths and events
Performance of EWS at clinical cut-offs. CART, Cardiac Arrest Risk Triage Score; MEWS, Modified Early Warning Score; NEWS, National Early Warning Score; SEWS, Standardised Early Warning Score; ViEWS, VitalPAC Early Warning Score;.
SEWS and MEWS have an increased discriminatory ability (but AUROC <0.8.1) to predict death when applied on admission but not when applied at 24 hours prior to death when calculated without missing values imputed (complete case analysis). The discriminatory power of AWTTS when calculated with and without missing values, for all adverse outcomes, both on admission and at 24 hours prior to event was <0.70 (table 3). Specificity to predict death when applied on admission was ≥97% for all AWTTS when evaluated at the recommended clinical cut-offs (figure 2).
The performance of the SPTTS which used the four selected observations is shown in online supplementary table 3. The highest sensitivity for deaths and adverse outcomes for SPTTS applied on admission was for the system proposed by Kenward et al 27 (online supplementary table 2, row vii), which is 59.1% and 48.4%, respectively (PPV 1.72%). Five thousand and sixty-two (32.46%) patients would be triggered if this system, which includes heart rate, respiratory rate, systolic blood pressure and temperature was applied on admission.26 All other selected SPTTS had sensitivity less than 47% to predict death when applied on admission and PPVs are low (<8.24%). Sensitivities and specificities of the best performing of the SPTTS26 to predict death when applied on admission when computing only one, two or three of the four parameters are reported in the online supplementary table 4. If the best performing of the AWTTS (CART) and of the SPTTS (Kenward et al)27 were implemented, the number of patients triggered to correctly detect one death is 9.58 and 58.07, respectively. The best performing of the AWTTS and of the SPTTS ability to predict death when applied on admission for the most common diagnosis groups (ICD-10 chapters) is described in table 4. Performance was not assessed in diagnosis groups i and ii (table 4), as no deaths were reported in patients assigned to these groups.
Supplementary file 4
Performance of the best performing of the AWTTS (CART) and of the SPTTS (Kenward et al)27 to discriminate survivors from non-survivors when applied on admission for each diagnosis.
Supplementary file 7
Discussion
This study reports the availability of physiological parameters, existing practices in vital sign monitoring and the performance of existing AWTTS and SPTTS in a large and diverse LMIC population. Insights gained from this dataset may have relevance beyond this setting and reinforce concerns regarding the place of EWS described in smaller LMIC cohorts.
Availability of observations is poor in this setting. Heart rate, respiratory rate and systolic blood pressure have the highest availability at admission; however, availability of these measurements also decreases throughout the hospital stay. Low nurse-to-patient ratios, limited equipment for monitoring and limited understanding of the importance of observations in detecting unwell patients and preventing avoidable death may contribute to their poor availability in this and other resource-limited settings.14 28 While still incomplete, availability of all physiological parameters (tables 2 and 3) was significantly greater on admission, and for inpatients who went onto have events (P<0.05). Reasons for this may include established roles such as ‘admission nurses’, and expectations from consultants or nurses in charge that this information needs to be available on admission.29 Clinicians may use this information as a tool to guide diagnosis, and/or request further investigations.30–32
In this study, the behaviour of recording of physiological parameters was sustained over the study period (figure 1). Parameters which require no equipment for measurement, such as AVPU, were also often incomplete (figure 1). The paucity of some vital signs (saturation, measure of mentation) throughout the patient stay may be a reflection of the limited value placed on these signs by doctors and nurses during acute care decision-making in this setting.
Performance of the AWTTS was variable (table 3. Sensitivity was low, echoing similar studies from LMICs.16 CART had the greatest ability to discriminate death and adverse event on admission (table 3). Performance at 24 hours prior to event improved with complete case analysis when compared with normal imputation (table 3). Efforts to improve availability of vital sign reporting in this and other LMIC acute care settings remain an important priority. EWS using parameters with the least proportion of missingness need to be prioritised for evaluation. Clinicians and researchers assessing the performance of EWS with higher percentages of missingness should consider alternative methods such as multiple imputation when handling missing data. CART had the lowest burden of patients triggered when applied at the clinical cut-off of 20, when compared with the other AWTTS evaluated (figure 1).
Of the SPTTS tested, Kenward et al’s (2004) had the highest sensitivity to predict death or any adverse outcomes when applied on admission; however, this sensitivity would not be high enough for implementation in clinical practice. The burden of patients triggered would be 5062, meaning nearly one in three patients would trigger.
CART’s NNE was 9.58, compared with 58.07 for the best performing of the SPTTS; important when considering the feasibility of implementation of EWS in this and other resource-limited settings with low nurse-to-patient ratios.7 Effects of alarm or trigger fatigue may occur very rapidly, hampering efforts to improve understanding the value of vital sign monitoring in critical illness and in implementing rapid response systems.7 30
The relative proximity of time of event to admission (online supplementary figure) and the greater availability of observations may offer some explanation for the superior performance of AWTTS on admission compared with 24 hours prior to event: 59% (n=296) of events, of which 40 were deaths, occurred within 48 hours of admission. In this setting, on-admission physiology may have even greater importance in identifying at-risk patients and as a tool to guide subsequent decision-making, including the frequency of vital sign monitoring. Similar approaches such as WHO Quick Check tool for aiding triage based on on-admission physiological parameters have been shown to be effective in low-income countries.32
Performance of the best performing of the AWTTS and SPTTS when applied to the most common admission diagnoses was also varied (table 4). Limited access to non-fee-paying general practice or community facilities may contribute to patients being admitted to acute care facilities for relatively simple investigations. No deaths were reported in the ‘obstetric’ and ‘routine investigation’ groups. Frequent multiparameter vital sign reporting for these patients (30.89% of the total population admitted) may be at best viewed as impractical by front-line clinical staff or at worst be detrimental to patient outcomes by diverting precious nursing time away from those at higher risk of adverse outcomes.
The first step towards a pragmatic solution for improving identification of patients at risk of deterioration in this setting may be the implementation of AWTTS at admission that, in combination with other relevant parameters (eg, reason for admission), could help identify patients at low risk of adverse outcomes. For patients who do not have acutely deranged physiology on admission, or for whom admission is not based on clinical presentation of acute illness (eg, those admitted for routine laboratory investigation), SPTTS or a two-parameter track and trigger system (eg, based on heart rate and respiratory rate) may offer simpler tools for monitoring (online supplementary table 4). If triggered, then more complete multiparameter monitoring could be initiated along with simple remedial interventions such as oxygen therapy.20
Greater understanding of the admission criteria, frequency of measuring physiological parameters, time of presentation to hospital and patterns of disease is warranted. Identification of additional cues that clinicians may be using to prioritise patients they perceive as acutely unwell or at risk of deterioration is required; in keeping with similar approaches suggested for other resource-limited settings, these can be then further evaluated for safety and effectiveness.15
A simple electronic tool to record and visualise observations, which is increasingly feasible in LMIC settings,33–35 may assist clinicians in identifying at-risk patients, improve visibility of observations and trends, assist to overcome the limitations of disparate paper systems and facilitate education in recognition and response to deterioration.36 Such tools have been successfully implemented to assist in surveillance, clinician decision support and quality improvement efforts within and outside of critical care in this setting by the study group.21 34 35
Limitations
The accuracy of the recording of these measures was not evaluated during this study. This is a widely acknowledged limitation of similar pragmatic studies both in HIC and LMIC settings.31 36 Although a single-centre study, the large sample size, diverse case mix and focus on the practical challenges of implementation of scores in resource-limited settings mean the findings and discussion arising from this study are relevant to other LMIC research.
Conclusion
There is limited availability of observation reporting in this setting. Indiscriminate application of EWS to all patients admitted to wards in this setting may result in an unnecessary burden of monitoring and may detract clinicians from caring for sicker patients. AWTTS in combination with diagnosis may have a place when applied on admission to help identify patients for whom increased vital sign monitoring may not be beneficial. Further research is required to understand the priorities and cues that influence nurses’ and doctors’ perceptions of critical illness and decision-making.
Supplementary file 5
Supplementary file 6
Acknowledgments
The authors wish to acknowledge the support received from consultants in charge of wards at DGH Monaragala: Dr A C Weerarathne, Dr K Jayasuriya, Dr K C C Gamage, Dr H K S I Premaratne, Dr S S M Kariyawasam and the team that facilitated the data collection: N Dullewe, L Pieris, C S Suraweera, ET Jagoda and Dr M F Miskin.
References
Footnotes
Contributors JAS, AB and RH gave the original idea. All authors contributed to the development of the study protocol. Data collection was assisted and supervised by AB, NDS, JAS, RMDR, PCS, PLA and PGM. Statistical analysis was done by APDS, AR and SBM. Interpretation was by AB, APDS, PCS, AR, SBM, KSAJ, AMD and RH. Supervision and mentorship were given by AB, KSAJ, AMD and RH. All authors contributed important intellectual content during manuscript drafting, revision and accept accountability for the overall work by ensuring that questions pertaining to the accuracy or integrity of any portion of the work are appropriately investigated and resolved. All authors take responsibility that this study has been reported honestly, accurately and transparently; that no important aspects of the study have been omitted and that any discrepancies from the study as planned have been explained.
Funding The study was supported by the National Science Foundation, Sri Lanka (RG/2016/HS/02), Network for Improving Critical Care Systems and Training (NICST) and the Mahidol Oxford Tropical Medicine Research Unit (MORU), Thailand.
Competing interests None declared.
Patient consent Detail has been removed from this case description/these case descriptions to ensure anonymity. The editors and reviewers have seen the detailed information available and are satisfied that the information backs up the case the authors are making.
Ethics approval Ethical review and waiver of individual patient consent was obtained from the Ethics Review Committee of the Faculty of Medicine, University of Colombo (EC 15-034).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Data lies with the Network for Improving Critical Care Systems and Training and available on request.