Objectives To systematically review the evidence base for a systems approach to healthcare design, delivery or improvement.
Design Systematic review with meta-analyses.
Methods Included were studies in any patients, in any healthcare setting where a systems approach was compared with usual care which reported quantitative results for any outcomes for both groups. We searched Medline, Embase, HMIC, Health Business Elite, Web of Science, Scopus, PsycINFO and CINAHL from inception to 28 May 2019 for relevant studies. These were screened, and data extracted independently and in duplicate. Study outcomes were stratified by study design and whether they reported patient and/or service outcomes. Meta-analysis was conducted with Revman software V.5.3 using ORs—heterogeneity was assessed using I2 statistics.
Results Of 11 405 records 35 studies were included, of which 28 (80%) were before-and-after design only, five were both before-and-after and concurrent design, and two were randomised controlled trials (RCTs). There was heterogeneity of interventions and wide variation in reported outcome types. Almost all results showed health improvement where systems approaches were used. Study quality varied widely. Exploratory meta-analysis of these suggested favourable effects on both patient outcomes (n=14, OR=0.52 (95% CI 0.38 to 0.71) I2=91%), and service outcomes (n=18, OR=0.40 (95% CI 0.31 to 0.52) I2=97%).
Conclusions This study suggests that a systems approaches to healthcare design and delivery results in a statistically significant improvement to both patient and service outcomes. However, better quality studies, particularly RCTs are needed.
PROSPERO registration number
- organisation of health services
- quality in health care
- health policy
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This is the first systematic review to provide a comprehensive and transparent synthesis of the published evidence base for a systems approach to healthcare design, delivery and improvement.
A major limitation of our study rests on the heterogeneity of the literature it seeks to synthesise, with wide variation in the settings, participants, comparators, follow-up durations and study designs.
We have conducted two exploratory meta-analyses in order to give an overview of the general direction of results, and we acknowledge that these may give artificial numerical precision which may not be warranted.
This benefit must be interpreted and applied with care because the evidence mostly comes from before-and-after study designs, with inherent confounding factors of unknown magnitude and direction.
Several included studies reported both the potential of a Hawthorne effect and the existence of other interventions at the time of their study which may have contributed to their observed outcomes.
The 20th and 21st centuries have witnessed the development of highly effective healthcare technologies, diagnoses and interventions.1 2 Nonetheless, there remains a pressing need for improvement in both the quality and safety of care delivery.3–5 This is often attributed to several factors including multimorbidity,6 the complexity of healthcare delivery7 and a variety of cultural and organisational challenges.8 9 Drawing on the experience of fields such as engineering and design a ‘systems approach’ to improvement has been advocated, that recognises the interacting components of healthcare delivery, the people involved, as well as planned, considered and adaptive iterative implementation.10–16 However, there has not been a systematic review of the evidence base for such an approach within the healthcare literature to date.
Modern healthcare systems are striving for integrated, patient-centred, effective and efficient care,17 but the lesson from engineering is that such systems do not happen by accident; they need to be planned, designed and built.18 Understanding what this process might look like has been explored with reference to the literature on patient safety,19 Human Factors and Ergonomics (HFE),20 general practice,21 the well-being of healthcare workers22 and public health.23 These reviews, while useful, are limited in their scope and employ narrow views of a systems approach.
The primary objective of this study is to review, comprehensively, the usefulness of a systems approach to healthcare improvement. There were no limits on language, participant types, outcome types or any particular healthcare domain.
Definition of a systems approach
Defining a systems approach is challenging. The approach has origins in a variety of disciplines, which have both diverged and converged over the past century. These range from mathematics to social science, and span both the physical and biological sciences.24 In order to arrive at a definition that we could operationalise for the purpose of this systematic review, the team reviewed definitions of a systems approach including Clarkson et al,10 Maier and Rechtin,25 Chen26 and the NASA systems engineering handbook.27 As a result, we developed a shared understanding of a system, at its fundamental level, as:
A collection of different elements (or things) which together produce results unachievable by the individual elements on their own.28
Our working definition of a systems approach, which has been informed by Clarkson et al,10 is as follows:
A systems approach to healthcare improvement is a way of addressing health delivery challenges that recognises the multiplicity of elements interacting to impact an outcome of interest and implements processes or tools in a holistic way.
This view of a systems approach integrates perspectives on people, systems, design and risk in a way that is applicable to healthcare systems across all scales from local service systems through to organisational, cross-organisational and national policy levels.
This systematic review was conducted and reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) standard.29
Included were published primary research studies involving any patients in any healthcare setting where a systems approach was compared with concurrent or historical comparators—usual care or a non-systems approach, which reported numerical results for both groups for any outcomes relevant to the study being conducted. Excluded were conference reports or other unpublished studies, studies without clear evidence of a systems approach being used, studies without any type of comparator or studies without quantitative outcome data for either group.
We searched the following databases with no limits on date of publication: Medline, Embase and HMIC (via OVID), Health Business Elite, PsycINFO and CINAHL (via EBSCO), Web of Science and Scopus. The search was first conducted in August 2017 and repeated on 28 May 2019. It was repeated for a second time on 24 July 2020, but we found very little additional evidence, in particular nothing that would affect the meta-analysis direction and the conclusions of the systematic review. As a result, the analysis and results presented here are based studies up to May 2019. There were no limits on language, participant types, outcome types or any healthcare domain. The full search strategy including specific search strings are provided in online supplemental file 1.
Study selection process
We used a structured, two-stage approach to determine inclusion. The first stage involved a title/abstract review of citations after removing duplicates. The second consisted of a full-text review of the 107 papers identified as potential for inclusion.
For the title/abstract review stage, three pairs of researchers looked at a third of the records each. Studies were selected for inclusion or rejection independently by each researcher, and with differences resolved first within the pair, and then within the whole team where the pair could not agree.
The full-text review stage applied the definition of a system and of a systems approach as stated above. Researcher pairs individually reviewed studies for inclusion or exclusion based on the following two questions:
Does the study identify a clear problem framed in a systems context and demonstrate the use of a systems approach, in some way? AND
Does the study have an appropriate design to address the research question?
Question one excluded any study which did not in some way demonstrate a systems approach in its formulation and/or implementation of an improvement intervention, while question two excluded all protocols, conference abstracts, systematic reviews, reviews, editorials and any paper with no primary research or no comparator arm.
Following the individual assessment, members of each pair discussed their results to arrive at a consensus on which studies to include. As a final check for all included studies, the team assessed each study against aspects of our working definition of a systems approach. Studies were assessed on a binary scale (0 or 1) as to whether they demonstrate a consideration of systems in the development of an intervention or in the implementation of the intervention, use of design and a consideration of risk. The outcome of this assessment is shown in table 1 in online supplemental file 2. A full list of excluded studies with reasons for exclusion is also provided in online supplemental file 3. Our method is also summarised using the PRISMA systematic review process shown in figure 1.
A template for data extraction was developed by the research team working through samples of the selected papers to identify relevant fields and tables appropriate to the study question. The data extraction process was designed to include an element of quality control and minimisation of researcher bias. The lead author initially extracted data from all included studies using the agreed template, with other team members each assigned a subset of these to independently corroborate.
Data were extracted into tables as listed below, and all included in online supplemental file 2. Patient outcomes and service outcomes are each separated into two tables according to study design (see online supplemental table S2–S5b). Online supplemental tables S6–S8 are the results of applying the Critical Appraisal Skill Programme (CASP)30 questions to included studies:
Study source, Country, year and aspects of systems approach.
Characteristics of studies (population and intervention).
Characteristics of studies (design, baseline type, blinding and funding source).
Patient outcomes for studies with before and after design.
Patient outcomes for studies with concurrent design.
Service outcomes for studies with before and after design.
Service outcomes for studies with concurrent design.
CASP questions for appraisal of cohort Studies.
CASP questions for appraisal of case–control Studies.
CASP questions for appraisal of randomised controlled trials (RCTs).
Examples of patient outcomes include numbers of vaccinations received, numbers of medication events and time to death. Examples of service outcomes include appointment delays, customer flows and time to treatment received. We did not include every outcome as this was impossible. We also did not use summary outcomes as this will give undue weighting to some studies compared with others. Outcomes were selected based on their relevance to the overall objective of the respective studies.
Patient and public involvement
Due to the focus of this review on synthesising evidence within the academic literature, patient and public involvement was not applicable.
Review manager (V.5.3, The Cochrane Library) was used for the meta-analyses using a random-effects model due to the heterogeneity of participants, interventions and outcome measures. Meta-analysis was conducted for service outcomes and patient outcomes separately where the categories below had the highest number of studies. Categories were before-and-after studies, studies with concurrent controls and continuous versus categorical versus time-to-event data. The highest numbers of studies for both service and patient outcomes were the before-and-after studies so this category was used in both meta-analyses. Heterogeneity was assessed using the I2 statistic, using standard thresholds. Risk of publication bias was assessed by use of a funnel plot primarily using the service outcome studies and adding the patient outcome results for studies not already in the service outcome meta-analysis.
Risk of bias for all studies was assessed by two researchers independently using CASP checklists.30 These were chosen because they have a suite of checklists appropriate for different study designs. Differences were resolved through a consensus process. The CASP checklist for cohort studies, case control and RCTs were applied accordingly. The checklists consist of 11 or 12 questions in three sections—study validity, study results and local value of results. A complete PRISMA checklist (online supplemental file 4) is also included to illustrate adherence to the review process.
Our initial search found 11 463 records published prior to August 2017 and an extended search in May 2019 found a further 3081 records. After deduplication there were 11 405 citations including two records added from personal sources. Of these, 11 298 records were excluded after the scanning process, leaving 107 full texts. Included were 35 studies, out of which 23 provided sufficient data for the two meta-analyses conducted (figure 1).
Of the 35 included studies, 28 (80%) had a before-and-after design only. Six studies had both a before-and-after and concurrent design (including two RCTs). Summary characteristics of included studies are presented in table 1. Studies excluded at the full-text review stage, with reasons for exclusion, are provided in the online supplemental file 3.
There was considerable diversity in how a systems approach was conceptualised and implemented in the included studies. This diversity in approaches may be categorised in three ways:
A comprehensive implementation of traditional tools and approaches such as Plan-Do-Study-Act (PDSA), Lean, Human Factors Engineering, WHO health systems strengthening principles, SEIPS model, Business Process Re-engineering, Structure- Process- Outcome and various combinations of these.31–40
A focus on the breadth of coverage of the intervention, involving a wide range of stakeholders from patients, communities, multiple departments including consideration of physical structures.41–55
The application of standard systems concepts such as systems thinking and complex adaptive systems theory.56–60
Almost all included studies showed a benefit for using a systems approach for almost all the outcomes. The exceptions were New et al34 (service outcome, concurrent control) and Dennerlein et al49 (service outcome, concurrent control). Most of the factors reported as contributing to success were related to people. These were expressed in the form of engaging with stakeholders, taking a team-based approach, enhancing communication, adopting a collaborative approach, patient- centredness and physician-centredness. Similarly, difficulty in measuring impact and the inability to generalise to other contexts emerged as the most significant limitations.
We included two RCTs in our systematic review. Both reported significant improvements in outcomes favouring a systems approach. Rustagi et al36 randomised 36 health facilities in Cote d’Ivoire, Kenya and Mozambique to usual care or ‘a systems engineering intervention’ stratified by country and volume. They found that antiretroviral (ARV) coverage for HIV positive women increased threefold in intervention facilities compared with control facilities while HIV-exposed infants screening increased 17-fold. Similarly, Rothemich et al54 randomised 16 practices into intervention (8) and control (8) groups to determine whether a systems approach enhances smoking cessation support in primary care practices. The study concluded that a systems approach to identifying smokers, advising, assessing readiness to quit and referral to supporting agencies, led to statistically significant increases in cessation for patients irrespective of gender, compared with traditional tobacco-use vital sign screening alone.
Exploratory meta-analysis suggests that a systems approach significantly improves both patient outcomes (n=14, OR=0.52 (95% CI 0.38 to 0.71) I2=91%) and service outcomes (n=18, OR=0.40 (95% CI 0.31 to 0.52) I2=97%).
Heterogeneity was very high. The funnel plot (figure 4) is unclear regarding publication bias. If anything, it might suggest that small studies with very positive results are missing, rather than those with null results.
It is important that the above results are interpreted with the heterogeneity and the quality of the included studies in mind. The two included RCTs both had reasonably high quality. The five cohort studies with concurrent controls varied between good and fair quality. The before-and-after studies which made up 80% of included studies varied widely in quality, ranging from good to very poor. Details of the quality assessment results are included in the last three online supplemental table S5–S7.
Our novel systematic review with exploratory meta-analyses suggests that the use of a systems approach to improving care results in significant benefits for both patient and service outcomes. There were two RCTs included that individually found statistically significant improvements in outcomes associated with the use of a systems approach. These findings, together with the observation that the majority of studies had a before-and-after design, present a challenge in interpretation of results in relation to what is usually considered good quality evidence. In addition, we observed a number of factors, which may support success in the use of a systems approach as reported by the included studies. To our knowledge, this is the first systematic review that has endeavoured to conduct a comprehensive synthesis of the evidence base for a systems approach to healthcare improvement.
This review adds to a growing number of systematic reviews apparently motivated by the desire to find evidence for what works in healthcare improvement. Narrative reviews19–23 of a systems approach in healthcare have focused on specific health issues such as patient safety, Human Factors and Ergonomics (HFE) in healthcare, primary care, well-being of health workers and public health. Though these generally demonstrate value of a systems approach, they lack a rigorous and comprehensive assessment of the evidence base for this. Other systematic reviews have been conducted on most of the major healthcare improvement methodologies including Lean,61 Six Sigma,61 62 PDSA,63 Statistical Process Control (SPC)64 and Quality Improvement Collaboratives (QIC),65 with mixed results. DelliFraine et al61 in their review of both the Lean and Six Sigma methodologies concluded that there is very weak evidence that either of the methods improves care. However, the review did not provide a meta-analysis of the studies identified and only focused on studies between 1999 and 2009, thus limiting its value. Taylor et al66 in their review of PDSA found poor compliance with the original principles of the methodology but did not aim to assess the impact of the method on outcomes. In the review of SPC, the authors found considerable benefits of using the approach to monitor and control health processes, though they acknowledge some limitations exist. Wells et al65 in their review of QICs reported significant improvements in process and patient outcomes. Their review reported outcome measures from included studies but stopped short of a full meta-analysis. Our findings are also consistent with the expectations of positive impact from the several publications that have called for a systems approach to tackling the challenges of modern health delivery systems.10–16 There is, clearly, considerable interest in assessing the evidence base of various improvement methodologies, however, existing systematic reviews have not been comprehensive enough and lack focus on patient and service outcomes.
Though the current review focuses on a systems approach to improvement, we believe this represents the most comprehensive systematic review and meta-analysis so far for evidencing the effectiveness of an improvement methodology. This is because we had no limits on date of publication, health setting, study type or participant types. We wanted the results to be relevant to a wide range of healthcare improvements contexts. However, one may object to our decision to combine very heterogeneous studies as we have done because of the differences in clinical settings and outcomes being measured. We reasoned that the results of a combined study would be more useful to the healthcare community, practitioners and policy-makers than an issue-specific systematic review. Moreover, several of those already exist, although not as rigorous. The inclusion of two RCTs in this review further strengthens the results. Though limited in number, both RCTs report statistically significant improvements in outcomes following the implementation of a systems approach.
The major limitation of our study rests on the heterogeneity of the literature it seeks to synthesise, with wide variation in the settings, participants, comparators, follow-up durations and study designs. We have sought to mitigate this using a clearly articulated definition of a systems approach, and a structured, rigorous, approach to synthesising the available evidence. We have conducted just two meta-analyses in order to give an overview of the general direction of results. We acknowledge that the estimated effect size gives an artificial precision which may not be warranted. The heterogeneity of meta-analysis results is to be expected, given the wide variation in participants, settings, interventions, comparators and outcomes. This exploratory meta-analysis can only indicate that a systems approach appears to be beneficial. This benefit must be interpreted and applied with care because the evidence mostly comes from before and after study designs, with inherent confounding factors of unknown magnitude and direction. There is also a significant risk of publication bias, and several included studies also reported both the potential of a Hawthorne effect and the existence of other interventions at the time of their study which may have contributed to their observed outcomes. The fact that we selected outcomes based on their relevance to the overall objectives of the studies included may introduce another level of bias if authors framed their objectives based on what they wanted to publish.
Implications for further research
The engineering sector is one that has excelled in the application of a systems approach.18 The experience of the Systems Engineering community is that the value of a systems approach—in terms of quality of the resulting system, reduction in cost, delivery on time, customer satisfaction—corresponds to the extent to which a project or organisation commits to the approach.67 68 This has implication for our findings in this review. It helps raise a number of questions that present opportunities for future research. For example, what are the different ways in which a systems approach is implemented in healthcare? Is there an association between the time and resource invested in a systems approach and the impact on patient and service outcomes? If so, what is the optimum level of investment?
Another opportunity for future research is a comparative review which assesses the impact of all improvement initiatives against those explicitly adopting a systems approach if more certainty of the value of the approach is desired. Given the volume of literature involved in such a comparative review, this would represent a significant undertaking. Studies are also needed that adopt better study designs such as RCTs or, if necessary, develop alternative ways of understanding and achieving sufficiently robust evidence for a systems approach to healthcare design and delivery. This is a point pertinent to all improvement efforts, where the traditional medical model of the randomised controlled trial is rarely appropriate, but the need to generate convincing evidence remains pressing.
We have argued from the start that there has been a growing recognition of the potential value of a systems approach to healthcare improvement over the past two decades. Most of this recognition has been at the policy level, involving the WHO,69 the Institute of Medicine in the USA,3 4 12 the Department of Health in the UK70 71 and more recently, through a joint initiative between the Royal Academy of Engineering, Royal College of Physicians and the Academy of Medical Sciences.10 However, to support further research and increased practice of a systems approach in health and care, policy-makers need to understand the evidence base. Though several success stories and domain-specific reviews exist, a comprehensive review of the evidence across the healthcare literature has been lacking. Our review may, therefore, become invaluable to policy-makers who have found the argument for a systems approach conceptually appealing but also desire to see the evidence of what difference such an approach can make to patient and service outcomes. In addition, the references taken individually may serve as examples of real-world applications of a systems approach to healthcare improvement.
In summary, we have argued that a systems approach to healthcare has been championed increasingly in the health and care literature and in a variety of grey literature reports and position documents. We provide the first attempt to comprehensively explore the evidence base through a systematic review and meta-analysis. The results provide reasonably clear evidence that a systems approach to addressing health delivery challenges may lead to significant improvements in both patient and service outcomes.
The authors would like to thank the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care for the East of England (CLAHRC EoE) for providing financial support for some members of the team during this work. Our sincere thanks also go to Dr Nathan Crilly at the University of Cambridge Engineering Department, and Dr Guillaume Lamé of CentraleSupélec, Paris, for their constructive review of versions of the manuscript and finally to Paul Driver of Anglia Ruskin University, Cambridge for help with improving the quality of our images.
Twitter @AlexanderDotKom, @ilk21, @eugeniaokelly
Contributors AK, TD and JW conceived the idea for the study. All authors were involved in discussions that informed the design of the study and development of the search strategy. IK conducted the database search and sourced full texts of included studies. AK, TB, JW, TD, GKK and YL did the record scanning. The full-text review was done by AK, TB, JW, TD, GKK, YL, AG, JM and EO. Data extraction was undertaken by AK, TB, JW, TD, GKK, YL, AG, JM and KK. The meta-analysis and interpretation were done by CM and initial results discussed by all authors. Qualitative synthesis of included studies was conducted by AG, AK and NB whilst quality of studies were appraised by KK, AG, AK, GKK, YL, TB, JW and JM. Manuscript writing was led by AK, TB, CM, AG, PJC, JD, EO with contributions from all authors. Final approval of manuscript has been obtained from all authors. AK is the guarantor of this study.
Funding This research was funded by the National Institute for Health Research (NIHR) Global Health Research Group on Neurotrauma using UK aid from the UK Government to support global health research, and by The Healthcare Improvement Studies Institute (THIS Institute), University of Cambridge. THIS Institute is supported by the Health Foundation, an independent charity committed to bringing about better health and healthcare for people in the UK. The views expressed in this publication are those of the authors and not necessarily those of the NIHR or the UK government.
Competing interests AK reports grants funding from The Healthcare Improvement Studies (THIS) Institute and the NIHR Global Health Research Group on Neurotrauma. JW reports grants from National Institute for Health Research and the University of Cambridge, during the conduct of the study. TB reports grants from NIHR Global Health Research Group on Neurotrauma, during the conduct of the study. TD reports grants from NIHR CLAHRC East of England, during the conduct of the study. All other authors declare no competing interests.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement All data relevant to the study are included in the article or uploaded as online supplemental information. No additional data are available.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.