Article Text
PDF
Abstract
Purpose Leading health systems have invested in substantial quality improvement (QI) capacity building, but little is known about the aggregate effect of these investments at the health system level. We conducted a systematic review to identify key steps and elements that should be considered for system-level evaluations of investment in QI capacity building.
Methods We searched for evaluations of QI capacity building and evaluations of QI training programmes. We included the most relevant indexed databases in the field and a strategic search of the grey literature. The latter included direct electronic scanning of 85 relevant government and institutional websites internationally. Data were extracted regarding evaluation design and common assessment themes and components.
Results 48 articles met the inclusion criteria. 46 articles described initiative-level non-economic evaluations of QI capacity building/training, while 2 studies included economic evaluations of QI capacity building/training, also at the initiative level. No system-level QI capacity building/training evaluations were found. We identified 17 evaluation components that fit within 5 overarching dimensions (characteristics of QI training; characteristics of QI activity; individual capacity; organisational capacity and impact) that should be considered in evaluations of QI capacity building. 8 key steps in return-on-investment (ROI) assessments in QI capacity building were identified: (1) planning—stakeholder perspective; (2) planning—temporal perspective; (3) identifying costs; (4) identifying benefits; (5) identifying intangible benefits that will not be included in the ROI estimation; (6) discerning attribution; (7) ROI calculations; (8) sensitivity analysis.
Conclusions The literature on QI capacity building evaluation is limited in the number and scope of studies. Our findings, summarised in a Framework to Guide Evaluations of QI Capacity Building, can be used to start closing this knowledge gap.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This review represents a pioneering attempt to identify efforts to evaluate quality improvement (QI) capacity building at the healthcare system level.
With the limited base of past work to draw on, we lack a shared or sufficiently broad vision of how to construct and evaluate QI capacity building efforts, and we have therefore little evidence to make judgements regarding the appropriateness of the articles identified.
The review contributes a synthesis of current practices for evaluating QI capacity building efforts and represents a starting point to help close the knowledge gap at the healthcare system level.
Introduction
Evidence over the past few decades has consistently demonstrated that low-quality care places a heavy financial and human burden on healthcare systems worldwide.1 ,2 The problem persists despite the fact that more organisations than ever before are actively engaged in quality improvement (QI) efforts.3 ,4
QI can be defined as a systematic approach to making changes that lead to better patient outcomes, stronger system performance and enhanced professional development. Improving healthcare quality requires active participation and interdisciplinary collaboration of a workforce skilled in QI, complemented by patients, families, academics and policymakers.5–7 However, evidence shows that healthcare professionals are often ill-prepared to promote QI efforts and reluctant to change.8 ,9 This gap may partly explain why QI activity does not reliably improve performance.10 ,11
A systematic approach to capacity/capability building for improvement has been identified as one of the key characteristics of healthcare systems that deliver high performance in cost and quality.12–14 QI capacity building increases the self-sustaining ability of organisations and systems to recognise, analyse and improve quality issues by controlling and allocating available resources more effectively.15 ,16 For the purpose of this study, we defined ‘QI capacity building’ as the planned development of knowledge, skills and other capabilities of a system or an organisation to improve quality.17 Following Bevan's12 definition, ‘capacity’ refers to having the right number and level of people who are actively engaged and able to conduct improvement, while ‘capability’ refers to the confidence, knowledge and skills to lead the improvement. Although we refer to capacity building throughout this article, our focus is inclusive of capacity and capability.
Even though substantial investments in healthcare quality have focused on building capacity, there is a significant research gap in terms of assessment of these efforts. In addition, numerous research studies have evaluated specific QI initiatives and programmes, but little is known about the impact of QI capacity at the healthcare system level.7 ,18 By system level, we mean the governance, leadership, resources and service delivery arrangements that together enable a health system (encompassing healthcare providers, managers and other stakeholders) to design, implement and evaluate QI activities. This ‘system-level’ definition includes national or subnational systems (such as state, provincial or regional systems depending on the jurisdiction) and can represent autonomous healthcare systems serving specific populations (such as military services) or larger healthcare organisations that provide a range of services to specified but geographically dispersed populations (eg, Kaiser Permanente).
Capacity building assessments have been largely restricted to evaluations of specific training programmes,7 rather than system-wide studies. As Shortell et al19 noted, part of the difficulty of assessing the impact of QI activity and investments1 lies in the fact that most studies focus on a single site of care, condition or process that represents only one particular organisational problem. In healthcare, the costs of poor quality and the benefits of improved care are spread among multiple stakeholders and settings, yet organisational initiatives often focus on short-term results that are within the exclusive control of a single organisation.20 Furthermore, while building QI capacity has been a key component of system transformation efforts, it generally coexists with other capacity building activities, such as leadership training and professional development, making it hard to separate out and to assess the importance of QI capacity investments. In the current context, little is known about how QI capacity can be produced most effectively and efficiently from a system-level perspective.
While there are a number of approaches for evaluating efforts in QI capacity building and training,21–23 we sought system-level economic evaluations to understand the impact of capacity building efforts on health-system performance and the associated return on investment (ROI). ROI is a simple expression of economic evaluation that is intuitive and effective in allowing estimations of the value generated from healthcare investments. The use of ROI in QI allows the comparison of multiple inputs of an intervention on a common metric (cost). By monetising benefits (better care and better health), the intervention's value can be calculated relative to cost,24 complying with broadly accepted ‘value’ frameworks, such as the Institute for Healthcare Improvement's Triple Aim.25
The purpose of this study was to identify key steps and elements that should be considered for system-level evaluations of investment in QI capacity building, summarised in a framework that can be used to guide such evaluations. Accordingly, we conducted a systematic review of the healthcare services and policy literature with the following three objectives: first, to identify system-level evaluations of QI capacity building/training; second, to identify existing evaluations of the investment in QI capacity building/training (ROI or other types of economic evaluation), even if these were at a programme or initiative level, rather than the system level; and third, to identify any other evaluations or analyses of QI capacity building that would address the purpose of our study.
Methods
We conducted a systematic review of the healthcare services and policy literature to identify two types of studies: (1) evaluations of QI capacity building; and (2) evaluations of QI training programmes. The search included the most relevant indexed databases in the field and a targeted search of the grey literature.
The following eight indexed databases were searched: MEDLINE, EMBASE, Social Work Abstracts, HealthSTAR, Health and Psychosocial Instruments, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Social Sciences Abstracts and Scopus. We used the following search terms: Quality Improvement/Assurance and Capacity Building/Assessment/Evaluation or Training Assessment/Evaluation. We included the term quality assurance (QA) to ensure that no relevant articles were missed due to imprecise index term use. The full search strategy for EMBASE is provided in online supplementary appendix 1. Given the nature of our search, we anticipated that a substantial proportion of relevant articles would not be captured by indexed sources. Therefore, an extensive grey literature search was conducted, which included: Google Scholar; direct scanning of relevant government and institutional websites; reference searches of identified articles and additional targeted searches based on research team input. The search terms used for Google Scholar were combinations of the same terms used for indexed databases, in addition to ‘healthcare/health care’. Our scan of institutional websites included 85 organisations in Canada, the USA, the UK, Australia, New Zealand, South Africa and organisations with an international mandate (full list available). All searches were completed between November 2014 and January 2015. A study investigator (GM) supported by a Research Assistant (JI) conducted all searches, screening and data extraction.
supplementary appendix
Identified articles were screened based on their title and abstract. The 143 articles identified through MEDLINE were double screened at the beginning of this process to ensure inter-rater reliability (94% agreement). This was followed by regular meetings to monitor screening criteria. All articles describing the following types of study were identified for retrieval of full-text articles: QI/QA assessments/evaluations; QI/QA training assessments/evaluations and QI/QA capacity building initiatives. All full-text articles retrieved were then double screened, applying the following exclusion criteria: QI/QA initiatives or training without an assessment or evaluation; assessments or evaluations of QI/QA initiatives not primarily focused on QI/QA capacity building; and training in areas other than QI/QA (eg, training in clinical skills). Only articles written in English were included, with no restrictions on publication date or type. Data extracted from the selected articles included study type, context, evaluation design, common assessment themes and components.
Results
A total of 1562 references were initially identified through indexed databases and an additional 663 through Google Scholar. After title/abstract screening, 65 articles were retrieved for full-text screening. Forty-five additional articles were identified through institutional website scanning and recommendations from the research team. A total of 110 full-text articles were screened, and a further 16 articles were identified through reference list searches. Ultimately, 48 articles met the inclusion criteria and were included in the study. Figure 1 presents a flow chart summarising this process.
Searching and screening process and number of articles identified.
Table 1 pairs our research objectives with the number of articles identified, and shows general characteristics of the 48 articles selected. We did not identify any system-level QI capacity building/training evaluations (ie, evaluations targeting efforts that have broad system-wide, cross-sectoral, multiprofessional focus). All evaluations identified in our search had narrower foci on specific initiatives within particular sectors, professions or programmes. Two studies included economic evaluations of QI capacity building/training, specifically evaluations of ROI, which coincided with our second research objective. Forty-six articles representing other evaluations or analyses of QI initiatives or training were identified in relation to our third research objective. A synthesis of this general evidence is presented next.
General evidence on QI capacity building evaluation
As shown in table 1, only 30 articles represented studies directly evaluating QI capacity, QI capacity building initiatives or QI training. The other 16 articles were assessments or analyses related to QI capacity building, but not direct evaluations of it (eg, inclusion of QI in curriculum guidelines for healthcare professional education or accreditation, description of QI training programmes, analysis of how to build and evaluate QI capacity). Table 2 summarises the main content of the 46 initiative-level (non-economic) evaluations included.⇓
General characteristics of studies included
Findings from included articles, organised by theme
We identified wide variation in the approach and measures used to evaluate QI capacity and programmes or initiatives to build QI capacity. While evaluations of QI training programmes are mostly focused on measuring the incremental improvement in participant QI knowledge and skills, broader evaluations of QI capacity or capacity building initiatives are mainly focused on organisational enablers and barriers, although this pattern is inconsistent. It is worth highlighting that only 9 (30%) of the 30 direct QI evaluations identified assessed the impact of QI capacity/training in terms of patient or programme outcomes (table 2).
The process of identifying the evaluation components started with the identification of all components evaluated in the 46 articles, which were grouped according to common themes. Given the diversity of approaches, we identified 17 evaluation components that fit into 5 overarching dimensions, which are presented in table 2 and figure 2. These dimensions and components should be considered for inclusion in QI capacity building evaluations, and eventually adapted to system-level QI capacity building evaluations. Figure 2 also provides examples of how these evaluation components can be used.
Framework to guide evaluations of quality improvement capacity building.
Evaluations of ROI in QI initiatives
Given the limited evidence, we paid special attention to evaluations of ROI in QI initiatives that could inform our study objectives. We used Phillips' ROI Model in Training and Performance Improvement Programs,68 a commonly referenced work in this discipline, to analyse the alignment with the two studies that evaluated ROI in QI initiatives. Table 3 compares the approaches used in these studies and identifies elements used to calculate ROI specifically in QI capacity building initiatives.
Alignment of return on investment in quality improvement capacity building assessments
The Productive Ward Rapid Impact Assessment69 represents a large-scale evaluation of investment in QI capacity building. The ROI was estimated based on case studies in nine selected hospitals in England. Although the initiative is intended to be implemented across hospitals in England's National Health Service (NHS), the Rapid Impact Assessment was limited to this initiative rather than representing a broad system-wide, cross-sectoral evaluation of QI capacity building across the NHS. A second study by McLinden et al24 depicts an ROI assessment of a QI training intervention to improve a back office process in a US hospital setting.
Drawn from shared elements, as depicted in the fourth column in table 3, we identified eight key steps in ROI assessments of QI capacity building:
Planning—stakeholder perspective: The magnitude and value of an economic evaluation will vary depending on the stakeholder perspective selected. In the Productive Ward, for instance, the analysis took a ‘public value perspective’, attempting to include all benefits and costs allocated to every relevant stakeholder.
Planning—temporal perspective: The economic evaluation may be prospective (should the programme be undertaken?), retrospective (what were the results of the programme?) or contemporaneous (should the programme be changed?). In addition, the evaluation should allow enough consideration of midterm and long-term outcomes, especially for long-term interventions.68
Identifying costs: All relevant costs to conduct the intervention or that result from the intervention should be captured, provided they are directly attributable to the intervention. The Productive Ward assessment used national and local data sources and included indepth interviews to retrieve all relevant costs.
Identifying benefits: All relevant benefits should also be identified, including monetary and non-monetary or intangible benefits. For example, the Productive Ward Rapid Assessment included: quality outcomes, productivity and efficiency outcomes, and financial benefits. Financial benefits generated by increased direct patient care time were calculated through excess bed days, length of stay, hospital readmissions, rates of staff absence and stock reduction.
Identifying intangible benefits that will not be included in the ROI estimation: Non-monetary or intangible benefits should always be estimated and reported, even if sometimes they cannot or should not be converted to monetary values and included in the ROI estimation by design.68 In the Productive Ward Rapid Assessment, patient experience, staff satisfaction and harm events, although identified, were not quantified and excluded from the ROI estimation.
Discerning attribution: Identified costs and benefits should only be included in the ROI estimation if attributable to the intervention, and in the proportion attributable to the intervention. This is possibly the most crucial step in the ROI evaluation, due to the challenges in clearly justifying attribution and the associated potential discretional effects on the estimation results. For example, in order to isolate the effect of training, McLinden et al24 asked a group of stakeholders to consider the multiple factors that could be responsible for the financial benefits and then to estimate the percentage attributable to training. Attribution of changes to the Productive Ward was also obtained from the judgement of managers involved in the implementation of the programme during the interviews.
ROI calculations: ROI is calculated as the net benefit (benefits minus costs) divided by costs.68 For the Productivity Ward, the estimated total potential economic impact was calculated by scaling up the evidence from the 9 participating hospital trusts to all 139 wards in England. The estimation was that for every £1 spent, £8.07 would be returned. McLinden et al24 reported that for every $1 invested in training, $1.77 would be returned.
Sensitivity analysis: The results of an economic evaluation are based on assumptions that bring uncertainty to the final ROI estimate. A sensitivity analysis needs to be performed in order to understand the probabilities and magnitude of the variation in evaluation results. The Productive Ward evaluation used a table of risk assessments to discuss the implications of using the wrong assumptions in the model. McLinden et al24 also explored the impact of variations in costs and benefits in the calculations of ROI.
Discussion
Research in QI capacity building assessment is limited in the number and scope of studies, as reflected in the limited findings of our systematic review. While we cast a broad net for our search, it is possible that our search strategy was not sufficiently sensitive or specific to capture all relevant QI capacity building evaluations. However, given the multiple sources searched for this review, including eight indexed databases, plus Google Scholar, Google and targeted searches of governmental and other organisational websites, recommendations from experts and reviews of reference lists of articles identified, we believe it is unlikely that we have missed a system-level evaluation.
Several studies have shown improvement in quality outcomes related to building QI capacity; however, there has not been an emphasis on understanding how much we are getting from these investments. More indepth evaluations are needed to understand when learning occurs, is applied and when it has an impact on patient care. Furthermore, existing studies have substantial variation in evaluation approaches and measures, which reflects the lack of a shared or sufficiently broad vision of how to construct and evaluate QI capacity building. This issue challenges the applicability and generalisability of evidence across care settings and jurisdictions. With a limited base of past work to draw on, we have little evidence to make judgements regarding the appropriate level of QI investments, where these investments should be directed for optimal impact, and the extent and nature of costs related to QI training and projects. Therefore, although most health systems can quantify at least some of their investments in personnel and training dollars, the ROI for QI capacity building at the system level is largely unknown.
Taken together, this review represents a synthesis of the most current knowledge on QI capacity building evaluation at organisation and programme levels that we cautiously used to highlight important elements that are relevant to the system level, and key gaps that need further attention. To guide future evaluation efforts at the health system level, we have consolidated the main elements identified in our review into a Framework to Guide Evaluations of QI Capacity Building, presented in figure 2.
The left side of figure 2 (QI efforts) shows the 5 dimensions and 17 evaluation components identified, and the arrows represent the directional effect between them. Investments in QI capacity building produce QI training and QI activity. QI training and activity generate individual and organisational QI capacity. Organisational and individual QI capacity have an impact on patient and care outcomes. The interdependence between QI training and activity, and between individual and organisational QI capacity, is represented by bidirectional arrows.
The right side of figure 2 (QI evaluations) shows how ‘evaluations of QI capacity building’ (ie, evaluations of QI training or QI programmes/intervention) typically consider characteristics of QI training and/or QI activity and evaluate their effects on organisational and individual QI capacity, and ideally also on patient and care outcomes (arrow A). ‘Evaluations of QI capacity’ may explore the effect of organisational and individual QI capacity in outcomes (arrow B), or be limited to the assessment of the level of QI capacity in an organisation, region or healthcare system (arrow C). Distinctively, ‘evaluations of ROI in QI capacity building’ should start by taking into account the investments in QI capacity building and then evaluate all five dimensions in the framework, including outcomes (arrow D). The framework also incorporates the 8 identified key steps of a QI ROI assessment, advancing Phillip's framework by focusing on QI capacity building through examples provided for each of the 17 evaluation components. These examples show how the components relate to evaluations of ROI in QI capacity building. These evaluation questions are only examples of the many aspects that need to be considered when planning and executing economic assessments of QI capacity building, especially on a large scale.
Although not specifically focused on QI evaluation, a prior systematic review by Kaplan et al70 identified contextual factors that might influence QI success which coincide with our findings, such as leadership from top management, organisational culture, data infrastructure and information systems. Subsequently, Kaplan et al71 used an expert panel to prioritise these findings in a model to understand contextual factors affecting the success of QI projects. Although they identified external factors influencing QI success, these were from the organisational perspective and not at the health system level.
The extensive use of ROI evaluations in many industries contrasts with their slow introduction in health and social care evaluations. Direct transactions between customer and provider normally help quantify value in other industries. However, third-party payment systems in the delivery of healthcare make it difficult to identify opportunities for increasing ROI.24 Another key issue is converting intangible benefits to monetary value to be included in economic evaluations, given the central importance in healthcare of non-monetary outcomes, such as patient experience or health outcomes. This is especially critical in QI at the health system level and for population health, where targeted outcomes can be as ‘non-monetary’ as wait times or quality of life and as ‘intangible’ as innovation, leadership or culture. Phillips68 notes that ‘there is no measure that can be presented to which a monetary value cannot be assigned’, yet the key issues are making credible estimates that are stable over time and at a reasonable cost. Failing to address these issues has the inherent risk of misjudging the real value of QI capacity building investments.
Isolating the effect and discerning attribution of capacity building and training interventions is challenging, even more if doing so at the system level. Typical approaches include the use of control groups and time-series analysis, techniques that are not always plausible when multiple initiatives and programmes are implemented simultaneously. Alternatively, estimation of training impact can be obtained through focus groups or questionnaires, as shown in the examples identified through this review. The important point is to always carefully discern costs and benefits attributable to the intervention. Depending on the robustness of the estimation, error adjustments should be large enough to show reliable evaluation results.68
From the findings of this review, we can conclude that there is an important gap in QI capacity building knowledge and assessment, particularly at the system level. However, the techniques and necessary expertise to start addressing this research gap exist and the necessary resources could be made available. Even based on limited experience in this field, a more extensive use of ROI or other types of economic evaluation of QI capacity building can help close this knowledge gap. After all, ROI assessments are no more than evaluations of the balance between costs and benefits, which is coincidental with widely accepted ‘value’ frameworks in health, such as the Triple Aim. Therefore, a high policy priority going forward is to broaden the vision to pursue more comprehensive system-level evaluation and monitoring of advances in QI capacity building and the impact of investments, in order to truly achieve a better healthcare system for all.
References
Footnotes
Twitter Follow Gustavo Mery @gustavo_mery
Contributors GM and MJD designed the study. GM and JI collected data and conducted data analysis. GM wrote the manuscript. MJD, GRB and AB made substantial contributions to the identification of relevant literature, the interpretation of findings and were involved in drafting the manuscript and revising it critically. All authors gave final approval to this manuscript.
Funding This project was supported by the IDEAS Initiative—Improving and Driving Excellence Across Sectors, a QI capacity building programme in Ontario, Canada, funded by the Ontario Ministry of Health and Long-Term Care.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.