You are here

  • Home
  • Archive
  • Volume 7, Issue 6
  • Detecting organisational innovations leading to improved ICU outcomes: a protocol for a double-blinded national positive deviance study of critical care delivery

Article Text

Article menu

Download PDFPDF

XML
Health services research
Protocol
Detecting organisational innovations leading to improved ICU outcomes: a protocol for a double-blinded national positive deviance study of critical care delivery
  1. Howard Chiou1,2,
  2. Jeffrey K Jopling1,3,
  3. Jennifer Yang Scott1,
  4. Meghan Ramsey1,
  5. Kelly Vranas1,4,
  6. Todd H Wagner1,
  7. Arnold Milstein1
  1. 1 Clinical Excellence Research Center, Stanford University, Stanford, California, USA
  2. 2 Emory University School of Medicine Medical Scientist Training Program and Department of Anthropology, Emory University, Atlanta, USA
  3. 3 Gordon and Betty Moore Foundation, Palo Alto, USA
  4. 4 Oregon Health & Science University, Portland, OR, USA
  1. Correspondence to Dr Howard Chiou; %E2%80%83%E2%80%83%E2%80%83howard.chiou{at}stanford.edu

Abstract

Introduction There is substantial variability in intensive care unit (ICU) utilisation and quality of care. However, the factors that drive this variation are poorly understood. This study uses a novel adaptation of positive deviance approach—a methodology used in public health that assumes solutions to challenges already exist within the system to detect innovations that are likely to improve intensive care.

Methods and analysis We used the Philips eICU Research Institute database, containing 3.3 million patient records from over 50 health systems across the USA. Acute Physiology and Chronic Health Evaluation IVa scores were used to identify the study cohort, which included ICU patients whose outcomes were felt to be most sensitive to organisational innovations. The primary outcomes included mortality and length of stay. Outcome measurements were directly standardised, and bootstrapped CIs were calculated with adjustment for false discovery rate. Using purposive sampling, we then generated a blinded list of five positive outliers and five negative comparators.

Using rapid qualitative inquiry (RQI), blinded interdisciplinary site visit teams will conduct interviews and observations using a team ethnography approach. After data collection is completed, the data will be unblinded and analysed using a cross-case method to identify themes, patterns and innovations using a constant comparative grounded theory approach. This process detects the innovations in intensive care and supports an evaluation of how positive deviance and RQI methods can be adapted to healthcare.

Ethics and dissemination The study protocol was approved by the Stanford University Institutional Review Board (reference: 39509). We plan on publishing study findings and methodological guidance in peer-reviewed academic journals, white papers and presentations at conferences.

  • Organisation of health services
  • Quality in health care
  • INTENSIVE & CRITICAL CARE
  • QUALITATIVE RESEARCH

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

https://doi.org/10.1136/bmjopen-2017-015930

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • This study is a methodologically innovative translation of the positive deviance approach to health services research and incorporates both qualitative and quantitative rigour.

    • A national database of intensive care unit (ICU) care in the USA is used, and the methodology is a mixed-methods approach that triangulates between qualitative and quantitative data sources.

    • The project is likely to inform the organisation of care delivery in ICUs and both positive deviance and rapid qualitative methodologies in healthcare.

    • The database is limited to ICUs with telemedicine capabilities, and case-mix adjustment using Acute Physiology and Chronic Health Evaluation IVa is imperfect. Study participants and site visitors may also not be able to accurately ascertain which innovations drive performance.

    Introduction

    Critical illness represents an enormous burden in the USA, with more than 5 million patients admitted annually to intensive care units (ICUs).1 Caring for these patients consumes a disproportionate amount of resources; despite comprising fewer than 10% of all hospital beds, ICUs account for 13.4% of total hospital costs and 0.66% of the national gross domestic product.2 This burden will likely increase with the ageing population, as both utilisation rate and the proportion of beds allocated to intensive care increase.2–5

    Yet, the quality of care delivered varies dramatically between units and hospitals. ICUs differ widely in their rates of compliance with best practices and rates of avoidable complications (eg, hospital-acquired infections).6 Risk-adjusted mortality also differs among ICUs, with studies suggesting that high-performing ICUs in the country have up to 10–12 fewer deaths for every 100 patients than the lowest performing ICUs, even after controlling for factors like discharge practices and patient demographics.7 These trends have been confirmed in more recent studies of ventilated patients.8 9

    Variations in performance are likely driven by differences in ICU organisation and practices, rather than by access to technology.10 11 Modern ICUs are more an organisational innovation than a technological one, matching a concentration of personnel and resources for any type of critically ill patient. Previous research has identified the association between organisational factors like nurse-to-patient ratios, daily care plans and usage of care bundles and  improved risk-adjusted mortality.12–14

    Unfortunately, innovations in organisation and practice are not well described in the critical care literature. Hospitals do not typically share their innovative practices with one another, and data to compare ICU performance are not readily available. Although some practices may be published, context is frequently not reported in sufficient detail to ensure successful implementation.15 All these factors obscure our ability to identify which aspects of critical care organisation and practices help drive performance.

    Positive deviance is one methodology that may offer additional insights. This approach assumes that innovations that address problems common to many organisations have already been developed and can be detected by studying positive outliers before being tested and disseminated.16 17 Originating from global health, the approach has been used successfully in a wide variety of settings to improve healthcare quality, including diabetes management in primary care practices and hospital door-to-balloon times in response to acute myocardial infarction.18 19

    However, a systematic review of positive deviance studies in healthcare found research quality to be low and there have been very few applications of the approach in the critical care setting.16 Highlighting the need for increased rigour, a previous study, which used qualitative site visits, failed to identify the differences between ICUs associated with performance.11 The goal of this research protocol is to describe our methods for conducting a positive deviance study in critical care. Specifically, we sought to identify organisational innovations in the delivery of critical care, adapting the first two steps of the positive deviance approach to generate hypotheses as to which innovations explain variation in ICU utilisation and quality of care. A secondary objective was to identify potential organisational structures, processes and contexts that may explain this variation. Through these aims, we hope to detect innovations in intensive care and support an evaluation of how positive deviance and rapid qualitative inquiry (RQI) methods can be adapted to healthcare.

    Methods and analysis

    Conducting a positive deviance study requires four steps: (1) identify outliers within an area of interest, (2) use qualitative approaches to generate hypotheses to explain their performance, (3) test hypotheses in a larger sample and (4) disseminate evidence about best practices.20 21 Our strategy uses a blinded, retrospective approach in the two first steps. We analysed a national database of ICUs to develop a study cohort of five positive outliers and five comparator ICUs. This quantitative phase will be followed by in-depth qualitative work at these ten sites, where we will build comparative case studies on their innovations and themes.

    Quantitative phase: identifying outliers

    Data source

    We used data from the Philips eICU Research Institute (eRI) database, containing over 3.3 million patient records from over 50 health systems from 2003 to 2015. All ICUs in the database have implemented the Philips eICU telemedicine system. Data for ICU admissions include vital sign measurements, quality metrics, medication orders and patient laboratory values. All the ICUs in the study were given an opportunity to opt out of the study, and the protected health information of individual patients was not included. The database includes data from over 400 hospitals; as of 2014, there were 5686 acute care hospitals in the USA, all of which had at least one ICU.22

    ICU cohort selection

    Inclusion criteria included all hospital units that contributed data to the Philips eRI database between 2013 and 2015. We excluded the hospital units that did not participate for all 3 years and self-identified step-down or intermediate care units. To minimise variation from small sample sizes, we also excluded low-volume ICUs, defined as ICUs with fewer than 300 discharges per year. The final cohort included 276 ICUs that cared for a total of 3 70 278 patients over 3 years. These ICUs form a geographically diverse sample of ICUs with eICU capabilities.

    Outcome measurements

    Primary outcomes included mortality and length of stay for patients admitted to the ICU, since these parameters reflect both ICU quality and utilisation. While mortality rates are generally low in critical care and thus insensitive to use in comparisons,23–25 rates of deaths are sufficiently high enough among ICU patients to be used as a quality indicator.7 As patients may be transferred elsewhere in the hospital as death nears,26 ICU patient mortality rates were calculated using deaths that occurred both in the ICU (in-ICU mortality) and after transfer elsewhere within the hospital (combined post-transfer mortality).

    The eRI database does not include any cost estimates. We used length of stay used as a proxy for resource utilisation, since up to 85% of ICU costs are explained by length of stay alone.27 In this study, we calculated a mean residual for each ICU, using the difference between observed and expected lengths of stay, as predicted by the Acute Physiology and Chronic Health Evaluation (APACHE) IVa algorithm. Patients who died before discharge were excluded. As with mortality, we calculated length of stay including only ICU lengths of stay (in-ICU length of stay) and including days after transfer elsewhere within the hospital (combined post-transfer length of stay).

    Patient cohort selection

    The variation in outcomes between ICUs is mostly dominated by those who are very healthy or very sick. For example, ICU metrics are greatly skewed by low-risk patients admitted to the ICU purely for monitoring purposes and by high-risk patients for whom death may be a likely outcome. Consequently, only patients who have a predicted risk of death between 2% and 20% were included, as predicted by the APACHE IVa algorithms.28 Patients without calculated APACHE IVa scores were excluded. These limits were based on expert consensus among clinicians who are familiar with the APACHE IVa scoring system.

    Patients transferred between hospitals were also excluded from the study. Transfer status from another institution is an independent risk factor for mortality, even after controlling for case-mix.29 Small numbers of patients transferred dramatically affect mortality rates,30–32 and transfers are excluded from the APACHE IVa models.31 In order to control for this ‘transfer bias’, we excluded all patients who were transferred from another institution. We also excluded patients with extreme outlier unit lengths of stay more than  300 days.

    Direct risk standardisation

    In order to enable direct comparison of outcomes between each ICU, direct risk standardisation was used to adjust for variations in case-mix.33 In summary, we calculated a weighted average for each outcome variable using two percentage point increment risk groups based on APACHE IVa-predicted ICU mortality (eg, 2%–4%,… and 18%–20%). The weights were equal to the proportion of the number of patient records within each risk group. Weighted average mortality rates and lengths of stay were calculated for all patient records for each individual ICU. ICUs with less than 300 patient records for those within the 2%–20% APACHE IVa-predicted mortality were excluded to eliminate extreme variations due to small sample sizes.

    Bootstrapped variance and percentile CIs

    As risk adjustment was performed using direct risk standardisation, all adjusted outcome variables were weighted means. Unlike the arithmetic mean, no analytical analogue of the SE exists for weighted means.34 Therefore, we estimated CIs through bootstrapping.35 36 All outcome variables were calculated for each ICU, using 5000 resamples with replacement equal to the total number of patient records for each individual ICU. Variance and percentile CIs were then calculated for each ICU.

    Outlier identification and false discovery rate control

    Outlier and comparator ICUs were defined as ICUs with CIs that do not overlap with the population mean (α≤0.05). p values were generated for each ICU using a two-sided Student’s t-test and then adjusted for false discovery rate (d<0.05) using the Benjamin-Hochberg procedure.37 This process was repeated for each of the four outcome variables (ie, in-ICU mortality, combined post-transfer mortality, in-ICU length of stay and combined post-transfer length of stay) and visualised using caterpillar plots sorted by CI limits. ICUs identified as outliers on all four outcome variables were placed into respective positive outlier and negative comparator groups.

    Qualitative phase: detecting innovations

    Site selection

    Two members of our study team (HC and MR) were provided with an unblinded list of ICUs identified as positive outliers and negative comparators. A purposive sample of five positive outliers and five negative comparators were selected using a maximum variation approach based on the following institutional characteristics: (1) ICU type, (2) patient volume, (3) academic affiliation, (4) presence of intermediate care units, (5) case-mix of ICU, (6) geographic locale, (7) urban or rural and (8) health system.18 The site visit teams were then provided with a blinded list of these ICUs for recruitment. The sample size of 10 sites is based on previous research establishing 10 sites as likely to achieve thematic saturation for positive deviance studies in healthcare.21

    Site visits

    We adapted the team-based RQI methodologies used in public health and applied anthropology, which rests on building rapport quickly, triangulating across multiple sources of data and a multidisciplinary research team.38 The blinded RQI team includes a surgeon and systems engineer (JKJ), a registered ICU nurse and administrative fellow (DB) and a healthcare researcher (RP)—all trained by two applied anthropologists (HC and HK). The research team will collect and analyse three key data sources: (1) semistructured interviews, (2) unstructured observations and (3) extant data.

    The Consolidated Framework for Implementation Research (CFIR) will be used as a theoretical framework to guide both data collection and subsequent analysis.39 CFIR is a determinant framework consisting of constructs known to be associated with effective implementation and intended to guide evaluations and implementation strategy.39 40 As the CFIR constructs include interventions, individuals, organisational context and organisational processes, this framework provides both a typology and a terminology to evaluate interventions and their context.

    Semistructured team interviews and focus groups

    Bedside staff and unit managers will be recruited for interviews using a combination of key informant, snowball and opportunistic sampling.41 Recruitment will occur using a maximum variation approach, aiming to capture a wide variety of perspectives at each site from across the hierarchy, including doctors, nurses, nursing technicians and unit managers. Teams will recruit at least six to eight participants at each site, a sample size found previously to be usually sufficient for thematic saturation in healthcare positive deviance.21 42

    All interviews will be semistructured and use an interview guide that broadly addresses three key domains: unit practices and communication, quality improvement and relationships between management and frontline staff (see online Supplementary material). The interviews will seek to identify innovations in these key domains and generate testable hypotheses that may explain the variations in performance.43 All interviews will be conducted in private settings, digitally recorded and transcribed verbatim by professional transcriptionists.

    Supplementary Material

    Supplementary material 1

    Unstructured observations

    Observational data are particularly important for rapid qualitative approaches, as they provide a point of triangulation against data from interviews.44 Our strategy requires observational data obtained using ethnographic methodologies, which are designed to access the typical routines and conditions of a field site.45 Site visit teams will conduct at least 2 hours of direct observation in each ICU, including physician rounds, nursing shift changes, cardiopulmonary resuscitations and fixed observation at nursing stations and eICU command centres. Each researcher will systematically generate descriptive field notes, including observed behaviours, processes and environmental features.41 These unstructured observations also provide opportunities to build rapport and conduct informal interviews with bedside staff.

    Extant data

    Collection of contextual data is a critical component of RQI and provides an additional basis from which hypotheses can be triangulated.38 For example, site visit teams may encounter training documents, written policies, news reports or locally collected data. With permission, these data will be digitised into the research database and analysed as described below.

    Rapid continuous constant comparative analysis

    This project will adapt a team-based, continuous analysis methodology commonly used in rapid qualitative approaches.38 Considered critical to a team ethnographic approach, site visit teams will debrief as often as possible, reviewing field notes and interviews to generate potential hypotheses and innovations for each field site. The main purpose is to generate analytical field notes in a modified grounded theory approach, generating themes and causal explanations grounded in the data.46 47 While classic grounded theory emphasises a primarily inductive approach, we will include a mixed grounded theory and content analysis as typical of rapid qualitative research.38 47

    All field notes, preliminary reports, interview transcripts and any extant data are then imported to Dedoose, a qualitative analysis software designed for teams.48 All data will then be inductively coded using a combination of grounded theory and constant comparative methods, extending the formal codebook of themes identified during team debriefs. As site visit teams remain blinded to each site’s outlier status, a constant comparative method will be used to generate causal models of factors and innovations, assessing the possibility that a field site is a positive outlier or negative comparator site in turn. Additional field notes are generated in this process (‘memoing’), and a preliminary report for each site visit is generated.38

    Cross-case analysis

    All members of the study team will then be unblinded as to each sites’ outlier status, and all data sources will be analysed using a cross-case method.49 Relevant qualitative and quantitative data points will be entered into a matrix, organised by themes of interest identified during site visits and the outlier status of each site. The data will then be interrogated for patterns, themes, similarities and differences between the outlier and comparison sites. Causal models developed during the generation of preliminary reports will then be extended across multiple sites.

    Ethics and dissemination

    The study protocol was reviewed and approved by the Stanford University Institutional Review Board (reference: 39509). Verbal informed consent will be obtained from all participants, and interviews will remain confidential and de-identified. Any study findings will only be reported in the aggregate, and individual ICUs will never be identified in publications. Participating ICUs will not be disclosed their outlier status, but all publications and reports will be shared with the recruited sites. Potential innovations will also be disseminated to participants and nationally through the work at the Clinical Excellence Research Center at the Stanford University. We plan on publishing study findings and methodological guidance in peer-reviewed academic journals, white papers and presentations at academic medical conferences.

    Study status

    The quantitative portion of this study is complete. Qualitative data collection began in September 2016 and  completed in April 2017. Qualitative data analysis will be completed by September 2017.

    Discussion and limitations

    We aim to extend positive deviance methods into a national study of intensive care. By focusing only on a subset of patients who are most likely to have lengths of stay and mortality rates affected by organisational processes and practices, this study aims to detect new innovations in the delivery of critical care. These innovations can then be tested in subsequent studies and disseminated broadly if found to be efficacious.

    There are several limitations to this study. First, the database includes only ICUs that subscribed to the Philips eICU programme. However, as the main objective of this study is to identify new organisational innovations that may drive ICU performance, the fact that all ICUs in this study have telemedicine capabilities ensures a similar level of technological access. In the USA, there are few other national databases of ICU quality, and the database is likely one of the most comprehensive data sources available.

    Second, APACHE IVa is an imperfect measure of disease severity, although it remains one of the most widely used and best validated measures.50 51 As the main purpose of the quantitative portion of this project is to identify outlier ICUs, likely to harbour organisational innovations, we believe the large sample sizes in this project will also protect against this limitation.

    Finally, an intrinsic risk of the positive deviance approach is that success is dependent on the ability of either the researchers or the study participants to identify the innovations, leading to variations in outcome. Although the double-blinded nature of this study maximises our ability to correctly identify successful innovations, there is an unavoidable risk that no new innovations will be identified. Replication of positive deviance studies can also be challenging, as differing site visit teams may identify different innovations as worthwhile.

    This protocol, however, also contains several novel features to further the translation of positive deviance methods to healthcare services research. First, we are conducting the study with both qualitative and quantitative rigour, responding to previous criticisms of the method. Second, this study is the first to use a double-blinded strategy, as both study participants and site visit teams are not disclosed the outlier status of individual ICUs. These methodological innovations will allow us to evaluate the usage of positive deviance and RQI methods in healthcare and test rapid team ethnography as a research tool. Our hope is that these methodological innovations will make a significant impact in improving healthcare delivery and outcomes for critically ill patients.

    Acknowledgments

    The authors would like to thank Hilary King, Lloyd Provost, David Spiegelhalter and Scott Halpern for providing thoughtful commentary and guidance towards the development of this protocol.

    References

    1. 1.↵
      Society of Critical Care Medicine [Internet]. Critical care statistics; c 2017 http://www.sccm.org/Communications/Pages/CriticalCareStats.aspx (cited 2017 Mar 26).
    2. 2.↵
      2. Halpern NA ,
      3. Pastores SM
      . Critical care medicine in the United States 2000–2005: An analysis of bed numbers, occupancy rates, payer mix, and costs*. Crit Care Med 2010;38:65–71.doi:10.1097/CCM.0b013e3181b090d0
      OpenUrlCrossRefPubMedWeb of Science
    3. 3.↵
      2. Halpern SD ,
      3. Ubel PA ,
      4. Asch DA
      . Harnessing the power of default options to improve health care. N Engl J Med 2007;357:1340–4.doi:10.1056/NEJMsb071595
      OpenUrlCrossRefPubMedWeb of Science
    4. 4.↵
      2. Halpern NA ,
      3. Pastores SM
      . Critical care medicine beds, use, occupancy, and costs in the United States. Crit Care Med 2015;43:2452–9.doi:10.1097/CCM.0000000000001227
      OpenUrl
    5. 5.↵
      2. Nguyen YL ,
      3. Kahn JM ,
      4. Angus DC
      . Reorganizing adult critical care delivery: the role of regionalization, telemedicine, and community outreach. Am J Respir Crit Care Med 2010;181:1164–9.
      OpenUrlCrossRefPubMedWeb of Science
    6. 6.↵
      2. Pronovost P ,
      3. Needham D ,
      4. Berenholtz S , et al
      . An intervention to decrease Catheter-Related bloodstream infections in the ICU. N Engl J Med Overseas 2006;355:2725–32.doi:10.1056/NEJMoa061115
      OpenUrl
    7. 7.↵
      2. Knaus WA ,
      3. Wagner DP ,
      4. Zimmerman JE , et al
      . Variations in mortality and length of stay in intensive care units. Ann Intern Med 1993;118:753–61.doi:10.7326/0003-4819-118-10-199305150-00001
      OpenUrlCrossRefPubMedWeb of Science
    8. 8.↵
      2. Kahn JM ,
      3. Rubenfeld GD ,
      4. Rohrbach J , et al
      . Cost savings attributable to reductions in Intensive Care Unit length of Stay for mechanically ventilated patients. Med Care 2008;46:1226–33.doi:10.1097/MLR.0b013e31817d9342
      OpenUrlCrossRefPubMedWeb of Science
    9. 9.↵
      2. Kahn JM ,
      3. Goss CH ,
      4. Heagerty PJ , et al
      . Hospital volume and the outcomes of mechanical ventilation. N Engl J Med Overseas 2006;355:41–50.doi:10.1056/NEJMsa053993
      OpenUrl
    10. 10.↵
      2. Rubenfeld GD ,
      3. Scales DC
      . Organizational Change in Critical Care. In: Scales DC , Rubenfeld GD , eds. The Organization of critical care. New York: Humana Press, 2014:3–8.
    11. 11.↵
      2. Zimmerman JE ,
      3. Shortell SM ,
      4. Rousseau DM , et al
      . Improving intensive care: observations based on organizational case studies in nine intensive care units: a prospective, multicenter study. Crit Care Med 1993;21:1443–51.
      OpenUrlCrossRefPubMedWeb of Science
    12. 12.↵
      2. Sakr Y ,
      3. Moreira CL ,
      4. Rhodes A , et al
      . The impact of hospital and ICU organizational factors on outcome in critically ill patients: results from the extended prevalence of infection in intensive care study. Crit Care Med 2015;43:519–26.
      OpenUrlCrossRefPubMed
    13. 13.↵
      2. Checkley W ,
      3. Martin GS ,
      4. Brown SM , et al
      . Structure, process, and annual ICU mortality across 69 Centers. Crit Care Med 2014;42:344–56.doi:10.1097/CCM.0b013e3182a275d7
      OpenUrlCrossRefPubMed
    14. 14.↵
      2. Pronovost P ,
      3. Berenholtz S ,
      4. Dorman T , et al
      . Improving communication in the ICU using daily goals. J Crit Care 2003;18:71–5.doi:10.1053/jcrc.2003.50008
      OpenUrlCrossRefPubMedWeb of Science
    15. 15.↵
      2. Kaplan HC ,
      3. Brady PW ,
      4. Dritz MC , et al
      . The influence of context on quality improvement success in health care: A systematic review of the literature. Milbank Q 2010;88:500–59.doi:10.1111/j.1468-0009.2010.00611.x
      OpenUrlCrossRefPubMedWeb of Science
    16. 16.↵
      2. Baxter R ,
      3. Taylor N ,
      4. Kellar I , et al
      . What methods are used to apply positive deviance within healthcare organisations? A systematic review. BMJ Qual Saf 2016.
    17. 17.↵
      2. Lawton R ,
      3. Taylor N ,
      4. Clay-Williams R , et al
      . Positive deviance: a different approach to achieving patient safety. BMJ Qual Saf 2014;23:880–3.doi:10.1136/bmjqs-2014-003115
      OpenUrlFREE Full Text
    18. 18.↵
      2. Bradley EH ,
      3. Curry LA ,
      4. Ramanadhan S , et al
      . Research in action: using positive deviance to improve quality of health care. Implementation Science 2009;4:25.doi:10.1186/1748-5908-4-25
      OpenUrl
    19. 19.↵
      2. Gabbay RA ,
      3. Friedberg MW ,
      4. Miller-Day M , et al
      . A positive deviance approach to understanding key features to improving diabetes care in the medical home. The Annals of Family Medicine 2013;11:S99–S107.doi:10.1370/afm.1473
      OpenUrlAbstract/FREE Full Text
    20. 20.↵
      2. Baxter R ,
      3. Taylor N ,
      4. Kellar I , et al
      . Learning from positively deviant wards to improve patient safety: an observational study protocol. BMJ Open 2015;5:e009650.doi:10.1136/bmjopen-2015-009650
    21. 21.↵
      2. Rose AJ ,
      3. McCullough MB
      . A practical guide to using the positive deviance method in health services research. Health Serv Res 2016;2010:1–16.doi:10.1111/1475-6773.12524
      OpenUrl
    22. 22.↵
      American Hospital Association. AHA Hospital Statistics. Chicago: Health Forum, 2016.
    23. 23.↵
      2. Lilford R ,
      3. Pronovost P
      . Using hospital mortality rates to judge hospital performance: a bad Idea that just won't go away. BMJ 2010;340:c2016.doi:10.1136/bmj.c2016
      OpenUrlFREE Full Text
    24. 24.↵
      2. Girling AJ ,
      3. Hofer TP ,
      4. Wu J , et al
      . Case-mix adjusted hospital mortality is a poor proxy for preventable mortality: a modelling study. BMJ Qual Saf 2012;21:1052–6.doi:10.1136/bmjqs-2012-001202
      OpenUrlAbstract/FREE Full Text
    25. 25.↵
      2. Pitches DW ,
      3. Mohammed MA ,
      4. Lilford RJ
      . What is the empirical evidence that hospitals with higher-risk adjusted mortality rates provide poorer quality care? A systematic review of the literature. BMC Health Serv Res 2007;7:91.doi:10.1186/1472-6963-7-91
      OpenUrlCrossRefPubMed
    26. 26.↵
      2. Kozar RA ,
      3. Holcomb JB ,
      4. Xiong W , et al
      . Are all deaths recorded equally? The impact of hospice care on risk-adjusted mortality. J Trauma Acute Care Surg 2014;76:634–41.doi:10.1097/TA.0000000000000130
      OpenUrl
    27. 27.↵
      2. Rapoport J ,
      3. Teres D ,
      4. Zhao Y , et al
      . Length of stay data as a guide to hospital economic performance for ICU patients. Med Care 2003;41:386–97.doi:10.1097/01.MLR.0000053021.93198.96
      OpenUrlCrossRefPubMedWeb of Science
    28. 28.↵
      2. Corporation C
      . The APACHE IV equations: benchmarks for mortality and Resource Use (white paper). Kansas City: Cerner, 2005.
    29. 29.↵
      2. Combes A ,
      3. Luyt C-E ,
      4. Trouillet J-L , et al
      . Adverse effect on a referral intensive care unitʼs performance of accepting patients transferred from another intensive care unit*. Crit Care Med 2005;33:705–10.doi:10.1097/01.CCM.0000158518.32730.C5
      OpenUrlCrossRefPubMedWeb of Science
    30. 30.↵
      2. Kahn JM ,
      3. Kramer AA ,
      4. Rubenfeld GD
      . Transferring critically Ill patients out of hospital improves the standardized mortality ratio. Chest 2007;131:68–75.doi:10.1378/chest.06-0741
      OpenUrlCrossRefPubMedWeb of Science
    31. 31.↵
      2. Vasilevskis EE ,
      3. Kuzniewics MW ,
      4. Dean ML , et al
      . Relationship between discharge practices and intensive evidence of a discharge bias. Med Care 2009;47:803–12.
      OpenUrlCrossRefPubMedWeb of Science
    32. 32.↵
      2. Rosenberg AL ,
      3. Hofer TP ,
      4. Strachan C , et al
      . Accepting critically Ill transfer patients: adverse effect on a referral center's outcome and benchmark measures. Ann Intern Med 2003;138:882–90.doi:10.7326/0003-4819-138-11-200306030-00009
      OpenUrlCrossRefPubMedWeb of Science
    33. 33.↵
      2. Nicholl J ,
      3. Jacques RM ,
      4. Campbell MJ
      . Direct risk standardisation: a new method for comparing casemix adjusted event rates using complex models. BMC Med Res Methodol 2013;13:133.doi:10.1186/1471-2288-13-133
      OpenUrlCrossRefPubMed
    34. 34.↵
      2. Gatz DF ,
      3. Smith L
      . The standard error of a weighted mean concentration—I. Bootstrapping vs other methods. Atmos Environ 1995;29:1185–93.doi:10.1016/1352-2310(94)00210-C
      OpenUrlCrossRefWeb of Science
    35. 35.↵
      2. Wood M
      . Bootstrapped Confidence Intervals as an approach to statistical inference. Organ Res Methods 2005;8:454–70.doi:10.1177/1094428105280059
      OpenUrlCrossRef
    36. 36.↵
      2. Efron B ,
      3. Gong G
      . A leisurely look at the bootstrap, the jackknife, and cross-validation. Am Stat 1983;37:36–48.doi:10.1080/00031305.1983.10483087
      OpenUrlCrossRefWeb of Science
    37. 37.↵
      2. Jones HE ,
      3. Ohlssen DI ,
      4. Spiegelhalter DJ
      . Use of the false discovery rate when comparing multiple health care providers. J Clin Epidemiol 2008:61.
    38. 38.↵
      2. Beebe J
      . Rapid Qualitative Inquiry. 2nd ed. London: Rowman & Littlefield, 2014.
    39. 39.↵
      2. Damschroder LJ ,
      3. Aron DC ,
      4. Keith RE , et al
      . Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science 2009;4:50.doi:10.1186/1748-5908-4-50
      OpenUrlCrossRefPubMed
    40. 40.↵
      2. Nilsen P
      . Making sense of implementation theories, models and frameworks. Implementation Science 2015;10:53.doi:10.1186/s13012-015-0242-0
      OpenUrlCrossRefPubMed
    41. 41.↵
      2. Bernard H
      . Research methods in Anthropology. 5th ed. Walnut Creek: AltaMira, 2011.
    42. 42.↵
      2. Guest G
      . How many interviews are Enough?: an experiment with data saturation and variability. Field methods 2006;18:59–82.doi:10.1177/1525822X05279903
      OpenUrlCrossRefWeb of Science
    43. 43.↵
      2. Bradley EH ,
      3. Curry LA ,
      4. Ramanadhan S , et al
      . Research in action: using positive deviance to improve quality of health care. Implementation Science 2009;4:1–11.doi:10.1186/1748-5908-4-25
      OpenUrlCrossRefPubMed
    44. 44.↵
      2. Beebe J
      . Rapid Assessment Process. AltaMira: Walnut Creek, 2004.
    45. 45.↵
      2. Emerson RM ,
      3. Fretz RI ,
      4. Shaw LL
      . Writing ethnographic fieldnotes. 2nd ed. Chicago: University of Chicago Press, 2001.
    46. 46.↵
      2. Creswell JW
      . Qualitative Inquiry and Research Design. 3rd ed. Thousand Oaks: : SAGE Publications, 2012.
    47. 47.↵
      2. Bernard HR ,
      3. Wutich A ,
      4. Ryan GW
      . Analyzing qualitative data: systematic approaches Thousand Oaks. 2nd ed. SAGE  Publications, 2017.
    48. 48.↵
      Dedoose. SocioCultural Research Consultants. Version 7.0.23 [software] http://www.dedoose.com. 2016.
    49. 49.↵
      2. Yin RK
      . Case Study Research. 3rd ed. Thousand Oaks: SAGE Publications, 2003.
    50. 50.↵
      2. de Lange DW
      . The Pitfalls of Benchmarking ICUs*. Crit Care Med 2015;43:473–4.doi:10.1097/CCM.0000000000000732
      OpenUrl
    51. 51.↵
      2. Kramer AA ,
      3. Higgins TL ,
      4. Zimmerman JE
      . Comparing Observed and Predicted Mortality among ICUs using Different Prognostic Systems. Crit Care Med 2015;43:261–9.doi:10.1097/CCM.0000000000000694
      OpenUrl

    Footnotes

    • HC and JKJ contributed equally and are co-first authors.

    • Contributors JKJ, HC, MR, KV, THW, and AM contributed significantly to the initial conceptualisation of the project. JKJ and HC were responsible for the study design. The quantitative analysis was conducted by JYS, HC and THW. HC drafted the initial manuscript, and JKJ, MR, KV, BR, THW and AM provided substantial feedback for intellectual content on initial drafts. All authors approved the final copy.

    • Funding This work was supported by The Gordon and Betty Moore Foundation grant number 4719 and the Stanford University Clinical Excellence Research Center. The following authors received salary support from this grant: HC, JYS, AM and THW.

    • Competing interests None declared.

    • Ethics approval Stanford University Institutional Review Board.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Correction notice This paper has been amended since it was published Online First. Owing to a scripting error, some of the publisher names in the references were replaced with 'BMJ Publishing Group'. This only affected the full text version, not the PDF. We have since corrected these errors and the correct publishers have been inserted into the references.