Article Text

Original research
Feasibility of implementing Extubation Advisor, a clinical decision support tool to improve extubation decision-making in the ICU: a mixed-methods observational study
  1. Aimee J Sarti1,2,
  2. Katina Zheng3,
  3. Christophe L Herry2,
  4. Stephanie Sutherland1,
  5. Nathan B Scales2,
  6. Irene Watpool2,
  7. Rebecca Porteous2,
  8. Michael Hickey4,
  9. Caitlin Anstee2,
  10. Anna Fazekas2,
  11. Tim Ramsay2,
  12. Karen EA Burns5,
  13. Andrew JE Seely1,2,6,7
  14. On behalf of the Canadian Critical Care Trials Group
  1. 1Department of Critical Care, Ottawa Hospital, Ottawa, Ontario, Canada
  2. 2Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
  3. 3Medicine, University of Ottawa Faculty of Medicine, Ottawa, Ontario, Canada
  4. 4Department of Medicine, Division of Critical Care, University of Toronto, Toronto, Ontario, Canada
  5. 5St. Michael’s Hospital, University of Toronto, Toronto, Ontario, Canada
  6. 6Division of Thoracic Surgery, The Ottawa Hospital, Ottawa, Ontario, Canada
  7. 7University of Ottawa, Ottawa, Ontario, Canada
  1. Correspondence to Dr Aimee J Sarti; asarti{at}toh.ca

Abstract

Objectives Although spontaneous breathing trials (SBTs) are standard of care to extubation readiness, no tool exists that optimises prediction and standardises assessment. In this study, we evaluated the feasibility and clinical impressions of Extubation Advisor (EA), a comprehensive clinical extubation decision support (CDS) tool.

Design Phase I mixed-methods observational study.

Setting Two Canadian intensive care units (ICUs).

Participants We included patients on mechanical ventilation for ≥24 hours and clinicians (respiratory therapists and intensivists) responsible for extubation decisions.

Interventions Components included a predictive model assessment, feasibility evaluation, questionnaires and interviews with clinicians.

Results We enrolled 117 patients, totalling 151 SBTs and 80 extubations. The incidence of extubation failure was 11% in low-risk patients and 21% in high-risk patients stratified by the predictive model; 38% failed extubation when both the model and clinical impression were at high risk. The tool was well rated: 94% and 75% rated the data entry and EA report as average or better, respectively. Interviews (n=15) revealed favourable impressions regarding its user interface and functionality, but unexpectedly, also concerns regarding EA’s potential impact on respiratory therapists’ job security.

Conclusions EA implementation was feasible, and users perceived it to have potential to support extubation decision-making. This study helps to understand bedside implementation of CDS tools in a multidisciplinary ICU.

Trial registration number NCT02988167.

  • adult intensive & critical care
  • respiratory physiology
  • qualitative research

Data availability statement

No data are available.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • The Extubation Advisor tool is the first clinical decision support tool developed to standardise and optimise the assessment of extubation.

  • The mixed-methods study design allowed for a comprehensive evaluation of the feasibility, facilitators and barriers to the implementation of a novel predictive tool.

  • Technical feasibility, accuracy and key stakeholder acceptance (both respiratory therapists and intensivists) were assessed to understand barriers to future implementation.

  • The single centre design may limit generalisation to other centres and settings.

Introduction

About 35% of adults admitted to the intensive care unit (ICU) are mechanically ventilated,1 postoperatively and for respiratory failure.2 Prolonged ventilation is associated with increased length of stay, hospital costs3 and adverse events including decreased respiratory muscle strength,4 ventilator-associated pneumonia and mortality.5 Failed extubation, which occurs in approximately 15%,3 6 7 has its own detrimental effects on patient outcomes and cost.3 7 Both timely and successful extubation are critical to preventing adverse outcomes in the ICU.

One of many methods of assessing extubation readiness is the spontaneous breathing trial (SBT) which assesses extubation readiness through a trial of reduced or no ventilator support.5 Measures obtained during the SBT have been evaluated to predict extubation outcomes, including respiratory rate (RR), tidal volume (TV), their ratio the Rapid Shallow Breathing Index (RSBI=RR/TV),8 RR variability (RRV)9–11 and cough strength, among others.8 12 13 However, techniques used to conduct and assess SBTs vary.14 15

The Extubation Advisor (EA) tool is the first clinical decision support (CDS) tool developed to standardise and optimise this assessment. The web-based application combines an RRV-derived predictive model of the risk of extubation failure, RSBI, clinical impression of extubation failure risk and standardised extubation readiness checklist to generate a report to assist extubation decision-making. In a mixed-methods study, we sought to observationally implement a novel waveform-derived predictive analytical model in a bedside tool, to evaluate its feasibility, facilitators and barriers to implementation. This observational study is one of multiple components in the development of a CDS tool. A single centre implementation is first necessary, with a plan for future multicentre implementation, followed by a pilot then multicentre definitive randomised control trial.

Methods

This observational study (Phase I) included qualitative and quantitative components (figure 1). EA reports were only shown to clinicians 72 hours post-extubation. The four data sources included feasibility, predictive model assessments, quantitative questionnaire responses of respiratory therapists (RTs) and intensivists (MDs), and interviews with RTs. Patient-level data were collected with a waiver of consent as approved by the Research Ethics Board due to the observational nature of this study. This study was implemented in two multidisciplinary 28-bed ICU sites of The Ottawa Hospital, an adult, acute tertiary care academic hospital in Ottawa, Canada.

Figure 1

Breakdown of the observational mixed-methods study design.

We included patients admitted between June 2017 and October 2018 who were on invasive mechanical ventilation ≥24 hours; ready for SBTs; tolerant of pressure support ventilation ≤14 cm H2O (oxygen saturation ≥90% with fraction of inspired oxygen ≤40% and positive end-expiratory pressure ≤10 cm H2O), haemodynamically stable (off or low vasopressors: phenylephrine <50μg/min; norepinephrine <5μg/min; dobutamine <5 μg/kg/min; milrinone <0.4 μg/kg/min), stable neurological status (no deterioration in Glasgow Coma Score in the prior 24 hours, intact respiratory drive and intracranial pressure <20 mm Hg), intact airway reflexes and normal sinus rhythm at time of SBT. We excluded patients with documented do-not-reintubate orders, anticipated withdrawal of life support, tracheostomy, known or suspected severe myopathy, neuropathy or quadriplegia, one-way extubation and prior extubation during ICU admission. We consented both RTs and MDs who were responsible for extubation decisions of enrolled patients prior to completion of the questionnaires and/or qualitative interviews.

Data collection and analysis

Feasibility and data entry

Technical feasibility included time required for data entry, predictive model calculation and time to generate an EA report. We also collected RT-entered patient-level data, specifically, demographics, comorbidities and summary of SBT conducted, including a best evidence readiness for extubation checklist (see online supplemental additional file 1).

RRV-derived predictive model

All patients were mechanically ventilated and capnography data were assessed on the bedside monitors using Philips CO2 modules (Philips Healthcare, Andover, Massachusetts, USA) and a sensor affixed to the expiratory limb of the ventilator circuit. Capnography waveforms were collected by the BedMasterEx (Excel Medical Electronics, Jupiter, Florida, USA) data acquisition platform. The EA polled the BedMasterEx server for enrolled patients’ waveforms during SBT to calculate RRV. We then estimated extubation failure risk using a RRV-based predictive model derived through machine learning from the Weaning and Variability Evaluation (WAVE) study.11 Extubation failure was defined as the need for reintubation or death within 48 hours of extubation.

Finally, an EA report was generated summarising extubation failure risk, RSBI, clinical impression and checklist. The clinical impression of extubation failure was the RT and MD collective clinical impression of extubation failure risk (high or low) without specific defining criteria. SBTs were only incorporated into the EA report if they had no protocol or technical violations. EA reports were generated for all included SBTs, but only the report from the last SBT pre-extubation was sent for clinician review.

Quantitative questionnaires

RTs completed questionnaires regarding the data entry process immediately after completing the electronic case report form post-SBT (see online supplemental additional file 2). The EA report was sent with the second questionnaire to treating MDs and RTs 72 hours post-extubation to avoid impacting clinical care (see online supplemental additional file 3). The questionnaire asked RTs and MDs to evaluate the EA report. Quantitative data were analysed using descriptive statistics.

Qualitative interviews

We conducted semi-structured, face-to-face interviews with RTs experienced with EA, until data saturation. Interviewees were asked to share their thoughts and experiences with EA (see online supplemental additional file 4). The interview guide developed by two investigators (AJS and SS) allowed for a broad, predetermined line of inquiry which is flexible and permits for the exploration of emerging themes. All interviews were conducted by a qualitative expert (SS), audio-recorded and transcribed verbatim. We used purposive sampling to ensure representation from both ICUs. Four interviews were coded inductively by the full qualitative team (AJS, SS and KZ) to develop the initial codebook. Two investigators coded the remaining interviews with disagreements resolved by the third investigator. The codebook was refined throughout the coding process with team consensus (see online supplemental additional file 5). Interviews were uploaded to qualitative analysis software NVivo Pro V.12 (QSR International, Melbourne, Australia) for management and analysis. Two investigators independently assessed and came to consensus on interviewees’ general attitudes towards EA.

Patient and public involvement

No patients were involved in this study.

Results

Feasibility

We enrolled 117 patients and recorded data on 206 SBTs; however, protocol (N=12) and technical violations (N=43) resulted in subsequent exclusion of 26.7% of SBTs (figure 2). We generated SBT reports for 93.4% (median SBT: 35 min) of included SBTs. Overall, 11/151 were included after we manually corrected missing information. We generated 71 EA reports from 80 extubations (88.8%, median time to generate: 145 s) (table 1). Overall, 12 (15%) extubations failed. The median time for RTs to complete data entry was approximately 210 s, according to beta testers entering standard cases. We could not obtain precise estimates due to variable RT workflow (non-continuous data entry, transfers of care between RTs, etc).

Figure 2

Flow of patient enrolment and spontaneous breathing trials by outcomes. a N=114 unique patients. The last SBT prior to extubation is the only one validated by the Weaning and Variability Evaluation study to predict extubation risk. b Failed extubation was defined as the need for reintubation or death within 48 hours of extubation. EA, Extubation Advisor; ICU, intensive care unit; RT, respiratory therapist; SBT, spontaneous breathing trial.

Table 1

Patient demographics of the Extubation Advisor and WAVE studies

Predictive model

There were a total of 12 extubation failures in this feasibility study. Using binary WAVE extubation risk score cut-offs of <0.50 (below average) and ≥0.50 (above average),11 5/46 (10.9%) and 7/34 (20.6%) of extubations failed, respectively (table 2). Results show a 0.72-fold compared with a 1.37-fold change, respectively, in failure risk compared with the data set average. The fold increase is the multiplicative factor increase in risk compared with the average population risk (which is itself defined as number of failed/passed). When both the WAVE risk score and clinical impression of extubation risk were above average, 3/8 (37.5%) of extubations failed.

Table 2

Predictive performance of the Extubation Advisor in assessing risk of extubation failure with comparison to the WAVE study

Quantitative questionnaire

Data entry questionnaire

We received 52/245 questionnaires (21%) from 26 unique RTs. RTs rated the data entry process highly: 94% rated it average or better in each of the five categories (data entry process, clarity, time to complete data entry, completeness and workflow integration) (figure 3). Additionally, 83% had no technical issues using the application.

Figure 3

Respiratory therapist feedback on data entry for the Extubation Advisor on categories including technical problems, time to complete, workflow integration, completeness, clarity and data entry process.

EA report questionnaire

We received 16/48 questionnaires (33%) from 11 MDs and 5 RTs. At least 75% rated it average or better in each of the five categories (clarity, accuracy, completeness, potential impact and usefulness). MDs had more favourable ratings (82%) than RTs (60%) (figure 4).

Figure 4

(A) Intensivists and (B) respiratory therapist feedback on the Extubation Advisor (EA) report on categories including usefulness, potential impact, accuracy, completeness and clarity. AP is the rating for the average patient in contrast to the rating for the specific patient.

Qualitative interviews

We completed 15 in-person interviews (7 women and 8 men; site 1: 10, site 2: 5). None of the participants approached declined to be interviewed. We present the consolidated criteria for reporting qualitative studies in online supplemental additional file 6.

Twelve interviewed RTs had over 12 years of work experience. Nine had used EA multiple times while six 1–2 times.

Thematic analysis revealed several overlapping themes that described facilitators and barriers to EA implementation.

Mixed acceptability of EA

Seven RTs had positive opinions of EA, five were negative and three were neutral. Negative views were all from RTs with over 12 years in practice. They questioned the value of computer-generated predictive models.

My senior colleagues are against it. They feel it degrades our profession. Can this app really predict like our judgment or some other doctor’s judgment for that matter? RT4

Facilitators to implementation

RTs felt that the data entry process was straightforward, aligned with workflow and that the potential for quality improvement helped the implementation process.

Ease of the data entry process

Most RTs found the application easy to use.

It was pretty user friendly, pretty easy to input data. I can’t think of anything that could make it easier. RT11

Most did not experience technical issues, although some brought up questions regarding issues including login, internet connectivity and data population.

Workflow integration

Many RTs reported that EA could be integrated with minimal impact to their workflow, as data entry took approximately 10 min. With increased experience, RTs felt that time required could be further reduced.

Quality improvement

Some RTs felt that EA could be a valuable tool for ensuring quality during extubation decision-making.

Maybe if you’re a fence-sitter and you’re not sure. If you are new you haven’t experienced a lot of bad extubations. Maybe it would help them make their decisions. RT13

Barriers to implementation of the EA

RTs expressed concerns around job security, discomfort with being evaluated and gaps in communication during implementation.

Concerns over job security

RTs feared that lower clinical ability to predict extubation failure compared with EA could compromise job security. Recent cuts to RT positions made this a salient concern for interviewees.

We are in a situation where the RTs have gone through cuts for the past four years. They’ve cut full-time positions. And then you’re asking us how good we are at doing our job, in my opinion if you determine with the study that we’re not good, then are we keeping our jobs at the end of the day? RT1

RTs also believed the extubation decision was mostly in the physicians’ control. They felt it was unfair that only their impressions of extubation risk were evaluated.

I do find that it doesn’t matter what we think in the end, the physician is going to do what they want. RT16

Discomfort with perceived judgement

RTs were uncomfortable having their opinions compared with the application and felt it undermined the complexity of their role in extubation decision-making.

The feeling is that we’re comparing the app versus the doctor versus the RT’s opinion. If the RT’s opinion always turns out to be the worst opinion, the least useful opinion, that doesn’t look very good for us, does it? RT7

Communication gap in implementation

Insufficient communication during implementation resulted in some RTs misunderstanding the study’s purpose. This led to rumours and misconceptions about the intent of the study.

At first a lot of people didn't understand they would not be evaluated on the results of the extubation. If I say, he’s going to pass then the patient fails. I'm a bad RT. I made a bad call…so they're evaluating my knowledge. There’s no way I'm doing this study. RT3

Mixed barriers and facilitators

Training

An EA trainer gave a group presentation followed by an application demonstration for RTs. Due to varying RT work schedules, not everyone received formal training.

We do the two-day, two-night, five off. Sometimes they pick days for training and certainly none of us come in on our days off for it. I find the way they do training here, they hit very few people. RT16

Many RTs found formal hands-on training unnecessary due to the application’s ease of use.

I feel like it’s pretty straightforward. A guided video would have been helpful that way people can kind of rely or if they ever had any like questions. I think most people were able to figure it out. RT4

Data completeness

RTs felt that EA’s combination of the extubation readiness checklist, clinical impression, RSBI and WAVE risk score captured additional factors relevant to extubation risk assessment beyond essential aspects commonly considered by RTs.

It makes us do a little bit more research on our patient, which I think is great. You get to know your patient more because it’s asking you questions that are a little outside the box, which is good. RT3

Discussion

This study provides important insight into the feasibility, facilitators and barriers relevant to future implementation of the EA tool, relevant to bedside implementation of any CDS tool using predictive models. This is the first feasibility assessment of a tool that combines artificial intelligence and clinical data to assist clinicians with bedside clinical decision-making for extubation.

The predictive model showed an appropriate fold change in risk between the below and above average risk groups in line with the result of the WAVE study. The EA showed that the above average risk group was 1.37 times more likely to fail extubation compared with the average patient. However, this should be interpreted with caution given that this is a smaller study designed to determine feasibility rather than a validation study of the predictive model.

Clinical decision support tools are shown to improve care in the ICU.16 17 However, not all are successful.18–20 They can introduce concerns about the impact of technical threats or errors to patient care and safety, which emphasises the need for accurate and robust systems.21 22 The speed of the decision support is often the most important factor for clinician acceptance.23

We recognise that the EA tool cannot capture all unique clinical scenarios that influence the decision to extubate, hence the clinical judgement of physicians remains at the centre of extubation decision-making. However, the EA integrated into the ICU as a real-time, bedside clinical decision support tool will help improve the prediction and assessment of extubation decision-making. Through capturing important clinical information using the standardised extubation checklist, RSBI, clinical RRV-derived predictive model of risk and clinical impression, the EA standardises the assessment of extubation risk.

Despite affirming the potential role for the EA tool, some RTs expressed dissent that impacted implementation. This was more often expressed by experienced RTs, a finding that has been reported previously.24 The perception that RTs were being evaluated and maybe even replaced decreased the EA’s general acceptability despite overall positive ratings from both MD and RT questionnaire respondents. A lack of communication from the researchers led to unanticipated misconceptions causing many RTs being reluctant to participate. Beyond the technical training, which interviewees found straightforward, greater focus on engagement and communication with RTs about the purpose of the EA during training was highlighted as necessary to future implementation.

We identified several strengths associated with use of EA in practice. First, interviewees identified its user-friendly interface. This finding is important for ‘buy-in’, as good interface design will reduce clinical error,25 while poor usability will deter adoption.26 Second, most respondents, both in questionnaires and interviews, did not experience problems, suggesting ease of use with minimal technical training. Overall, RTs felt the workload with using the EA clinically was manageable, which mitigated a common but significant barrier to obtaining ‘buy-in’ from busy clinicians.26 Interviewees agreed that increased familiarity with the EA tool will further decrease its impact on workload.

Although RTs felt the content was accurate and useful, they cautioned against over-reliance on a standardised tool to predict failure, noting the uniqueness of each patient’s extubation and context. Like protocolisation, EA may be perceived as diminishing the importance of clinical judgement and individualised patient care.27 However, research has demonstrated that reducing practice variation with protocolised weaning can positively impact patient outcomes.28 29 Improved education to highlight the value of EA and to emphasise its role in standardising extubation assessments to assist clinicians in making individualised clinical decisions is needed.

The feeling of ownership and responsibility over the extubation process is important to RTs.24 This may explain why some feared being ‘proven wrong’ when their clinical assessment differs from the rest of the risk assessment (eg, RBSI, WAVE risk score). Improved RT ownership and education is required to ensure RTs feel empowered to share their professional assessment as intended, without fearing it would be evaluated.

Study strengths include the innovative first implementation of a predictive tool based on RRV and artificial intelligence and the mixed-methods study design. Mixed-methods approaches are invaluable for evaluating clinical decision support systems,30 helping to evaluate feasibility,31 barriers to implementation,32 intervention fidelity33 34 and acceptance.35–37 We also incorporated views from two key stakeholder groups that collaborate in weaning in the Canadian context.

A limitation is the missing MD perspective in interviews. Although questionnaires captured some input, perspectives on physician ‘buy-in’ were only indirectly assessed from RT interviews. The single centre design and low questionnaire response rate limit generalisability to other practitioners and settings. In future implementation, we hope to broaden inclusion criteria such as eliminating the need for pressure criteria, to allow for broad application of the EA to all patients for consideration of SBT.

Additionally, RT perceptions were overall less positive in interviews than questionnaires, which may suggest negativity bias in qualitative data collection.38 The low questionnaire response rate may also indicate non-response bias, with more engaged RTs being more willing to complete questionnaires. However, we note that questionnaires were sent to RTs to institutional emails which are not often checked. In a future study, we will attempt real time completion of questionnaires using non-email dependent methods.

Our results show that in future implementation of EA, emphasis should be put on providing clinicians with information that not only emphasises the use of the technology, but also encourages ownership while engaging them in appreciating the rationale, relevance and potential for the EA to impact extubation decision-making.

Conclusions

In this study, we evaluated the feasibility, facilitators and barriers of introducing and implementing EA in a multidisciplinary ICU setting. Many clinicians found it accurate, with good workflow integration, and a user-friendly interface. Although it was designed to empower RT expression of extubation risk, this proved threatening to some. Improved engagement of RTs early in the implementation process may improve buy-in for future implementation. This study also helped refine the EA’s design and with the planning of future interventional studies.

Data availability statement

No data are available.

Ethics statements

Ethics approval

This study was approved by the Ottawa Health Science Network Research Ethics Board.

Acknowledgments

We would like to thank Excel Medical Electronics for their in-kind support via the BedMaster server installation, and Dr Asher Aron Mendelson for reviewing the manuscript on behalf of Canadian Critical Care Trials Group.

References

Supplementary materials

Footnotes

  • Presented at This work was performed at The Ottawa Hospital (Civic and General Campuses), Ontario, Canada.

  • Contributors AJS: Conceptualisation, methodology, formal analysis, resources, writing—review and editing, supervision, funding acquisition. KZ: Formal analysis, data curation, writing—original draft, writing—review and editing, visualisation. CLH: Methodology, validation, formal analysis, data curation, writing—review and editing, visualisation. SS: Methodology, formal analysis, data curation, writing—review and editing, visualisation, investigation. NBS: Software, writing—review and editing, investigation. IW: Writing—review and editing, investigation. RP: Writing—review and editing, investigation. MH: Writing—review and editing. CA: Software, writing—review and editing, investigation. AF: Resources, writing—review and editing, project administration. TR: Writing—review and editing. KEAB: Writing—review and editing. AJES: Conceptualisation, methodology, formal analysis, resources, supervision, funding acquisition.

  • Funding This work was supported by a peer-reviewed grant from The Ottawa Hospital Academic Medical Organization (TOHAMO) Innovation Fund 2016-17, awarded March 2017.

  • Competing interests AJES is the Founder and Board Chair of Therapeutic Monitoring Systems (TMS); TMS is developing and commercialising waveform-based variability-derived clinical decision support tools in order to improve care for patients at risk for or with existing critical illness. Both AJES and CLH are co-inventors on patents related to physiological waveform assessment and variability analysis. All remaining authors have disclosed that they do not have any potential conflicts of interest.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.