Article Text

Download PDFPDF

Protocol
Evaluating the long-term effects of a data-driven approach to reduce variation in emergency department pathology investigations: study protocol for evaluation of the NSW Health Pathology Atlas of variation
  1. Craig Scowen1,
  2. Nasir Wabe2,
  3. Alex Eigenstetter1,
  4. Robert Lindeman1,
  5. Melissa Miao3,
  6. Johanna I Westbrook2,
  7. Andrew Georgiou2
  1. 1NSW Health Pathology, Sydney, New South Wales, Australia
  2. 2Centre for Health Systems and Safety Research, Australian Institute of Health Innovation, Macquarie University, North Ryde, New South Wales, Australia
  3. 3Graduate School of Health, University of Technology Sydney, Broadway, New South Wales, Australia
  1. Correspondence to Dr Nasir Wabe; nasir.wabe{at}mq.edu.au

Abstract

Introduction Variation in test ordering is a major issue in Australia and globally with significant financial and clinical impacts. There is currently a lack of research identifying and remediating variation in the use of pathology tests in emergency departments (EDs). In 2019, NSW Health Pathology introduced the Pathology Atlas of Variation that uses a data-driven tool (the Atlas Analytical Model) to investigate test order variation across New South Wales (NSW) and engage with local health districts (LHDs) to reduce variation. The objectives of this study are to evaluate whether this data-driven approach is associated with: (1) a reduction in test order variation; (2) improvements in patient outcomes and (3) cost benefits, for the five most frequent ED presentations.

Methods and analysis This is a large multisite study including 45 major public hospitals across 15 LHDs in NSW, Australia. The Atlas Analytical Model is a data analytics and visualisation tool capable of providing analytical insights into variation in pathology investigations across NSW EDs, which will be used as feedback to inform LHDs efforts to reduce variation. Interrupted time series analyses using 2 years pre Atlas (2017–2018) and 2 years post Atlas (2021–2022) data will be conducted. Study data will be obtained by linking hospital and laboratory databases. Funnel plots will be used to identify EDs with outlying pathology test ordering practices. The outcome measures include changes in test ordering practices, ED length of stay, hospital admission and cost benefits (total pathology costs per ED encounter).

Ethics and dissemination The study has received ethical approval from the NSW Population and Health Service Research Ethics Committee (reference, 2019/ETH00184). The findings of the study will be published in peer-reviewed journals and disseminated via presentations at conferences. We will also engage directly with key stakeholders to disseminate the findings and to inform policies related to pathology testing in the ED.

  • health & safety
  • protocols & guidelines
  • international health services
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Strengths and limitations of this study

  • This will be the first state-wide study to evaluate the effects of a data-driven approach to reduce variation in emergency department (ED) pathology investigations and its potential patient and cost benefits in Australia.

  • The project will establish pathology utilisation benchmarks through which EDs compare their test ordering practices relative to sites within their local health districts and similar peer groups.

  • There are many other potential influences on the test ordering practices of clinicians in ED that may limit the impact of the Atlas project on reducing variation in test ordering.

  • Although our analyses will adjust for several potential confounders available in our data sources, other unmeasured factors can potentially confound the study outcomes.

Introduction

Pathology tests are critical for patient care including for diagnosis, monitoring of treatment outcomes and disease screening.1 However, studies indicate substantial variation in the use of pathology tests in clinical practice.2 3 Variation in pathology investigations can occur when patients with seemingly similar health status (type and severity of illness) receive different pathology tests,4 raising concerns about underutilisation of potentially useful tests or overutilisation of unnecessary tests.5

Variation in healthcare delivery in general is a major issue in Australia and globally with significant financial and clinical impacts.6–9 There has been a concerted effort by healthcare services in numerous countries to investigate and address unwarranted variation through the development of healthcare ‘Atlas’ projects.10–12 However, most of these projects have been conducted in primary care or focused broadly on healthcare variation. Few projects have targeted variation in pathology testing. Although there have been some studies indicating a high level of variation in pathology test ordering practices13–15 in acute care hospitals, there is currently limited research identifying, measuring and remediating variation in the use of pathology tests, particularly in the emergency department (ED). A recent Australian study revealed significant variation in test ordering across EDs, with some EDs ordering consistently more tests that were not in line with best practice guidelines. For example, the likelihood of ordering C reactive protein (CRP) was 38.7 times higher in one of the EDs compared with the ED with the lowest CRP testing,13 suggesting potential unwarranted test ordering practices.

Unnecessary overutilisation of pathology tests can have significant consequences for patient flow in EDs, where pathology testing has a direct effect on ED length of stay (LOS).16 In one Australian study, the median LOS was shown to increase by up to 10 min for every five additional tests ordered and by up to 17 min for each 30 min increase in test result turnaround time.16 Overutilisation of pathology tests can also have a negative impact on patient care, triggering a potential cascade of additional, unjustified and even invasive investigations, which may in turn lead to potentially harmful interventions.17 18

Applying data analytics can play a key role in identifying and reducing variation in pathology investigations by establishing benchmarks and uncommon variation that can be used to inform improved test requesting patterns and drive changes.3 19 20 Recently, NSW Health Pathology (NSWHP) in collaboration with the NSW Emergency Care Institute and eHealth Integrated Care introduced a pathology atlas project (hereafter referred to as the ‘NSWHP Atlas of Variation’). It is a state-wide health system quality improvement project that uses a data analytic tool (hereafter referred to as ‘Atlas Analytical Model’) to investigate the variation that exists in New South Wales (NSW) and to engage with local health districts (LHDs) to reduce variation where it may be unwarranted. The fundamental premise of this approach is that identifying variation will promote an examination of its causes and any consequences of outlying ordering patterns. The project will use data as a catalyst to drive change in pathology ordering toward the most appropriate and efficient testing profile for a given patient scenario by allowing comparison of ordering patterns between sites.20 By identifying differences across sites with similar groups of patients, a discussion with LHDs can be initiated as to the reasons for variation in ordering, warranted versus unwarranted variation and its ultimate impact on patient outcomes and service costs.

Objectives

The NSW Pathology Atlas of Variation project will be conducted across NSW acute care settings (both EDs and intensive care units) over 5 years (2019–2024). However, this study protocol is focused on assessing the impact of the Atlas on ED pathology test ordering specifically the (1) five most frequent ED presentations and (2) results of the first 3 years of the project.

The objectives of this study are to evaluate whether a data-driven approach (ie, using Atlas Analytical Model to engage with LHDs) is associated with: (1) a reduction in test order variation; (2) improvements in patient outcomes through a reduction in ED LOS and a reduction in hospital admission rates16–18 and (3) cost benefits (total pathology costs per ED encounter).

Methodology

Study setting

The project will involve 45 major public hospital-based EDs across 15 LHDs in NSW, Australia. The number of ED presentations to these hospitals accounted for >70% of total ED presentations in NSW in 2017/2018.21 According to the NSW peer groups for public hospitals classifications,22 12 of the study hospitals are classified as A1 (principal referral), 1 as A2 (paediatric specialist), 1 as A3 (ungrouped acute—tertiary referral), 10 as B1 (major hospital group 1), 8 as B2 (major hospitals group 2) and 13 as C1 (district group 1). Principal referral hospitals are among the largest hospitals in the Australian health system providing a very broad range of services including 24-hour ED, intensive care units and several other highly specialised units. With the exception of one hospital, all principal referral hospitals in NSW are included in the project. Paediatric hospitals have comparable facilities with those of principal referral hospitals, while major hospitals (B1 and B2) are relatively large but do not provide the range of services provided by the principal referral hospitals.23

Data sources

This study will use linked and deidentified administrative data to determine the effect of the Atlas. Comprehensive data on ED presentation characteristics, test utilisation and pathology costs will be obtained by linking two databases: the NSWHP Data Warehouse and NSW Health’s Health Information Exchange (HIE).

The NSWHP Data Warehouse contains pathology order information including, but not limited to, patient demographics, medical reference number (MRN), test orders and results and times a specimen is received and a verified result made available. The data set also contains information on pathology billing or costs aggregated at the ED encounter level. The HIE contains data on ED presentations including, but not limited to, patient demographics, patient MRN, patient visit/encounter ID, ED presenting problems, triage category, mode of arrival, patient disposition and throughput information such as arrival to triage, arrival to see a nurse, arrival to see clinician and arrival to departure.

Data linkage will be a two-step process. First, the data sets will be linked via patient MRN, date of birth and ED presentation date and times. Second, all records matched in the first step will be evaluated if the pathology service date and time are occurring within ED arrival and departure time of the patient at a given hospital. The second step will ensure that the service provided is matched to the correct patient presentation.

Study population

The study population will consist of all patients who presented to the study EDs during the study period with the following five most common conditions: abdominal pain, chest pain, limb injury, respiratory disorder and fever. The choice of these ED presentations was based on the results of a pilot trial conducted in 2018 (Craig Scowen. The NSWHP Pathology Atlas of Variation: A pilot trial). According to the pilot trial, these five presentations accounted for 39% of the total ED presentations in 2018.

Study design

The objectives of the study will be assessed using an interrupted time series (ITS), a type of quasiexperimental design, using both retrospective and prospective data. It is considered the best approach to evaluate the effect of an intervention at the population level when a randomised trial is not feasible.24 One of the strengths of ITS design is that it evaluates the effect of the intervention accounting for underlying preintervention trends in the outcome variables.24 25

The study will cover a 4-year period. The NSWHP Atlas of Variation was started in 2019 and the intervention activities will occur throughout 2019–2020. However, given 2019–2020 will be an intervention period, we will exclude the 2019–2020 data from the analysis. Therefore, the study will use 2 years (January 2017 to December 2018) pre Atlas and 2 years (January 2021 to December 2022) post Atlas data.

The Atlas Analytical Model

The Atlas Analytical Model employs a Microsoft Power business intelligence data analytics and visualisation tool capable of providing analytical insights into variation in pathology testing across NSW that can be used as feedback to inform LHDs efforts to reduce variation. It allows point and click selections across various elements to create a view relevant to a specific query or group. The model can be filtered by LHD, facility, peer group, presenting problem, age, gender, mode of arrival or separation, triage category and ED LOS.

Engagement with LHDs and other key stakeholders

Figure 1 shows a framework for engagement with key stakeholders. A clinical working group consisting of pathologists, ED clinicians and analysts was formed to oversee the development, direction and communication of the Atlas data. This group recommended the analysis should be delivered in a group forum to ED clinicians, LHD executives and administrative teams in the initial phase. Through the co-design process with the NSW Emergency Care Institute, it emerged through interviews that ED clinicians did not generally spend time exploring data visualisations for insights and a desktop tool was unlikely to drive wider change in an LHD without training and a substantial time commitment. Desktop dashboards are numerous within NSW Health and while they offer autonomy, findings may become isolated with one user. This, together with the time clinicians, would need to explore the data led the clinical working group to decide that sharing the analyses in a group would be the most effective first step.

Figure 1

A framework for engagement with various stakeholders involved in the NSW Pathology Atlas project. LHD, local health district.

The main objective of the Atlas Analytical Model is to raise awareness and generate discussion within an LHD. A localised summary analysis and key findings will be shared through clinical council meetings and cross-functional, multidisciplinary groups within LHDs. This approach will be trialled to identify the most effective medium for raising awareness and ongoing engagement within each LHD to build consensus and identify project groups to lead change.

Initial meetings will be conducted with senior members of the local pathology team to explore the Atlas data and prepare a session with the LHD clinical or executive team. This process is intended to uncover local nuances and context around findings relative to sites within the LHD and similar peer groups across the state. Then, LHD sessions will be held with ED clinicians, senior executive, financial or administrative teams. After reviewing data across the LHD and peer groups, the objective and output of the LHD session will be to develop a high-level action plan to modify ordering practices where it is considered appropriate.

Statistical analysis plan

Outcome measures

The primary outcome is a change in test ordering practices over time for selected tests where high outlying test ordering existed based on the initial analyses. The list of tests where high outlying test ordering exists will be identified using funnel plots as described in the ‘Analysis’ section. The secondary outcomes include patient (ED LOS and hospital admission) and cost benefits (total pathology costs per ED encounter). ED LOS is the length of time between arrival and ED disposition. Hospital admission will be defined as ED presentations subsequently leading to admission to inpatient wards or critical care wards such as intensive care or high dependency units. An ED episode of care with admission to the ED short stay unit but without subsequent admission to the hospital will not be considered as ‘hospital admission’.

Sample size consideration

We used chest pain data (one of the five ED presentations included in this study) to estimate the required sample size. Our previous study that evaluated the variation in test ordering practices for patients presenting to EDs with chest pain showed a considerable variation in the ordering of certain tests, particularly CRP, calcium, magnesium and phosphate, coagulation studies, lipase and blood gas.13 The variation in the ordering of CRP was exceptionally high with 56% of patient presentations receiving CRP in one of the EDs compared with the population average of 12%.13 A total of 750 patients presenting with chest pain per year from each hospital would be adequate to detect a reduction in CRP testing to within 15% of the population average testing rate, assuming a two-sided test with a 5% significance level and a power of 80%. An approximate similar sample size will be required for the other four ED presentations too. Given that the actual annual ED presentations due to the five conditions are much higher than the above estimate,26 there will be adequate sample size to address the objectives of the study.

Analysis

Funnel plots will be used to identify outlying test ordering practices by focusing on the top fifteen frequently ordered tests for each of the five conditions. Funnel plots, a type of statistical process control method, are used as a graphical aid representation of variation in an indicator of interest (eg, CRP testing rate) across different practices by plotting the 95% (≈2 SD) and 99.8% (≈3 SD) control limits from the study population mean value and displaying an ‘observed indicator’ against a measure of its precision (denominator size).27 High outlying test ordering for a given pathology test is defined as the testing rate above or below the 3 SD control limits from the mean study population testing rate (ie, the NSW average). Hospitals ordering a given test above 3 SD of the NSW average can be considered as over-testing, while those ordering below 3 SD can be considered as under-testing. For tests where high outlying test ordering exists, we will compare the pre and post Atlas funnel plots to determine changes in the number of outlying hospitals (ie, above or below the 3 SD of the NSW average).

We will use two approaches to estimate the effects of the Atlas Analytical Model on each study outcome: segmented regression method and generalised regression methods. Segmented regression is one of the most commonly used approaches in ITS design. Its main advantage is that it accounts for pre-existing trends in outcome variables. As segmented regression requires the outcome data to be aggregated into appropriate time points, we will divide the data into monthly intervals (eg, monthly CRP testing rate) over the study period. Thus, before and after Atlas periods will include 24 and 36 monthly time points, respectively, (excluding the 12-month transition period) and will constitute the two segments of the regression models. Segmented regression will be used to estimate the effect of the Atlas Analytical Model in the overall population (all study sites combined) as this approach does not account for potential clustering (that is the correlation between presentations to the same site). On the other hand, generalised regression methods, use patient-level data and can adjust for clustering effect and known confounders; it will, therefore, be used to corroborate the results of the segmented regression.

Given the need for pathology tests and the patterns of testing for the paediatric population can be different from the adults, we will conduct a sensitivity analysis by separating our data into paediatric versus non-paediatric. Other sensitivity analyses will also be conducted depending on the nature of the data (eg, excluding the paediatric hospital from the analysis).

Patient and public involvement

The project uses existing administrative hospital data sets. Therefore, this project does not involve any contact with patients or the public.

Discussion

The potential significance of the study

TheNSWHP Atlas of Variation project has the potential to assist in reducing unnecessary variation in testing and promote quality use of pathology testing in NSW. The study findings will assist NSWHP in monitoring pathology services across EDs in NSW and enable ongoing improvement of healthcare delivery across the state. The project will provide evidence that could also influence policy decisions on pathology testing in EDs and have substantial financial implications for the government into the future.

Although the main purpose of this project will be for pathology service improvement in NSW, the project will serve as an exemplar for extension and development to include medical imaging and other acute care settings including intensive care units across Australia. This development has the potential to have a major impact on the measurement of diagnostic quality and the appropriateness of treatment and to shape decision-making across the health sector. The evidence generated can also be used to support and guide similar initiatives internationally.

Strengths and limitations of this study

To the best of our knowledge, this will be the first project to evaluate the effects of a data-driven approach to reduce variation in ED pathology investigations and the potential patient and cost benefits in Australia. While there have been numerous projects undertaken to reduce or rationalise test ordering, there has been little analysis of the impact of test ordering profiles in relation to patient and economic outcomes. The NSWHP Atlas of Variation is a state-wide multisite project including 45 hospitals. The project will establish pathology test utilisation benchmarks through which EDs compare their test ordering practices relative to sites within their LHDs and similar peer groups. The benchmark metrics such as test volume, number of tests per encounter, number of requests per encounter and total pathology costs per encounter will be included. Pathology utilisation benchmarking can assist in harmonising test requesting practices and improving the quality of care across sites.28 29

The main limitation of the project is that there are many other potential influences on the test ordering practices of clinicians in ED beyond the Atlas project that may limit its impact on variation in test ordering.30 31 These may need to be specifically identified and evaluated during the implementation phase of the project.32 In this study, we have only one paediatric hospital. The pathology test utilisation benchmarks that this project is aiming to establish may not apply to the single specialist paediatric hospital, which is likely to exhibit very different testing patterns compared with the general hospitals. We are aware that another quality improvement project to harmonise test requesting processes in hospital acute care is currently underway in one of the LHDs (which includes four of our study EDs). We will conduct sensitivity analyses (excluding these four sites) to check the robustness of our results. While our analyses will adjust for several potential confounding factors available in our data sources, being a type of observational study, other unmeasured factors can potentially confound the study outcomes. Therefore, any potential impact of the intervention in this project may not imply causation. From the data consistency perspective, our pilot project has identified some variations in data entry for ‘presenting problem’, one of our key variables in this project. Although most of the entries are consistent across the sites and there were only a few missing values for this variable, our pilot project has revealed some free-text entries and inconsistencies in the free-text entries of the type of presenting problem (eg, presentations with ‘abdominal pain’ were entered as ‘pain—abdominal’, ‘pain, abdominal’ or ‘abdo pain’ etc). Similar issues were identified in a previous NSW-based study.33 Our research team will work with clinicians and relevant personnel to standardise the terminologies used to describe this data field prior to conducting the analyses. It is important to remember that the 3 SD above or below the population average to identify over-testing or under-testing using funnel plots does not necessarily imply the inappropriateness of test ordering. However, the findings of this study can be used as an important first step to investigate the appropriateness of testing. EDs can use the findings of the study to determine whether context-specific reviews are required to improve their test ordering practices.

Dissemination plan

We will disseminate the findings of the project to diagnostic informatics, health service/policymakers and broader audiences through conference presentations and peer-reviewed journal publications. We will follow the REporting of studies Conducted using Observational Routinely-collected health Data statement when writing publications.34 We will also engage directly with key stakeholders including the NSW Ministry of Health, NSWHP and LHDs within NSW to disseminate the project findings and to inform policies related to pathology testing in EDs.

References

Footnotes

  • Twitter @AGeorgiouMQ

  • Contributors CS, AE and RL conceived the study. NW and MM involved in the initial drafting of the paper. CS, NW, AE, RL, JW and AG contributed to the design of the work. CS, NW, MM, JW and AG contributed to the description of statistical analysis or dissemination plans. All authors were involved in revising the paper for important intellectual content, reviewed and approved the final version to be published.

  • Funding This work was supported internally by NSW Health Pathology. There has been no significant external financial support for this project.

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Patient consent for publication Not required.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.