Article Text
PDF
Abstract
Objectives The aim of this study is to explore the current and future state of quality measurement and feedback and identify factors influencing measurement feedback systems, including the barriers and enablers to their effective design, implementation, use and translation into quality improvement.
Design This qualitative study used semistructured interviews with key informants. A deductive framework analysis was conducted to code transcripts to the Theoretical Domains Framework (TDF). An inductive analysis was used to produce subthemes and belief statements within each TDF domain.
Setting All interviews were conducted by videoconference and audio-recorded.
Participants Key informants were purposively sampled experts in quality measurement and feedback, including clinical (n=5), government (n=5), research (n=4) and health service leaders (n=3) from Australia (n=7), the USA (n=4), the UK (n=2), Canada (n=2) and Sweden (n=2).
Results A total of 17 key informants participated in the study. The interview length ranged from 48 to 66 min. 12 theoretical domains populated by 38 subthemes were identified as relevant to measurement feedback systems. The most populous domains included environmental context and resources, memory, attention and decision-making, and social influences. The most populous subthemes included ‘quality improvement culture’, ‘financial and human resource support’ and ‘patient-centred measurement’. There were minimal conflicting beliefs outside of ‘data quality and completeness’. Conflicting beliefs in these subthemes were predominantly between government and clinical leaders.
Conclusions Multiple factors were found to influence measurement feedback systems and future considerations are presented within this manuscript. The barriers and enablers that impact these systems are complex. While there are some clear modifiable factors in the design of measurement and feedback processes, influential factors described by key informants were largely socioenvironmental. Evidence-based design and implementation, coupled with a deeper understanding of the implementation context, may lead to enhanced quality measurement feedback systems and ultimately improved care delivery and patient outcomes.
- Quality in health care
- QUALITATIVE RESEARCH
- HEALTH SERVICES ADMINISTRATION & MANAGEMENT
Data availability statement
All data relevant to the study are included in the article or uploaded as an online supplemental information. To safeguard the privacy of the participants and those involved in their services, we cannot make the transcripts available.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
Synthesis of perspectives and experiences of experts involved in measurement feedback systems.
Application of a theoretical framework to identify factors influencing measurement feedback systems.
Purposively sampled key informants may not represent all possible viewpoints and organisational barriers.
Introduction
Quality of care directly impacts the health outcomes and well-being of patients. The incidence of health service quality issues has led to the implementation of diverse measurement feedback systems over the last 25 years. Measurement feedback systems, also known as ‘audit and feedback’, involve collecting data to evaluate the healthcare delivery and identify unwarranted variation. It is intended that feedback on quality measurement data is provided to healthcare services to facilitate care improvements.1 Internationally, there are pockets of quality measurement efforts with varied purposes, methods and recipients. While healthcare professionals and health service research drive some of these measurement activities, many are externally driven and mandated by government agencies, medical professional groups and accrediting bodies.2 Such activities tend to operate as an additional reporting obligation, lacking integration with clinical care and workflow, and health professionals do not consistently receive feedback making it more challenging to realise the benefits of quality measurement.3 4
The 2012 Cochrane review of audit and feedback found that the quality improvement method demonstrated modest improvements. However, the effectiveness was variable and dependent on factors, including the feedback recipients’ baseline performance and how the feedback is provided.5 Despite data collection and technology advances over the last decade, the translation and impact of measurement feedback systems on patient outcomes and quality of care remain inconsistent.5 6 In 2019, Grimshaw et al suggested that future research should focus on how, when and why measurement feedback systems are most effective,7 as the literature often lacks implementation context and feedback processes.6 8 Recent research has focused on specific elements of measurement and feedback, which may contribute to their effectiveness, including the data source of quality measurement,6 health professional perceptions and psychological elements of receiving feedback,9 10 the use of theory in the design of feedback11 12 and computer-supported feedback.13 14 While this work has explored measurement feedback system design and how it is received, additional implementation barriers may prevent the uptake of even the most evidence-based interventions.
The fidelity of measurement feedback system implementation suggests that a gap remains between current systems and best practice. Furthermore, there is limited evidence on how to achieve best practice and in which contexts it is most viable.10 15 16 There is a need to understand the implementation context to enable the translation of evidence, including both barriers and facilitators.17 Given the limited insights produced by measurement feedback system literature, differences in the published studies may be due to distinctions in the context of these interventions. To bridge this gap, the Theoretical Domains Framework (TDF) has been applied across a wide range of healthcare settings to better understand how interventions can be effective with consideration of the socioenvironmental implementation context and to inform future interventions.17 18 The TDF includes a set of 14 domains covering psychological and organisational theory and could synthesise a broader set of external and internal health service factors, which affect measurement and feedback.
Guided by the TDF, this study aimed to identify factors influencing measurement feedback systems that involve data collection, analyses and providing insights back to health professionals on the quality of care delivered and patient outcomes. These factors included barriers and enablers to effective design, implementation, use and translation into quality improvement. This research study used qualitative interviews of key informants in quality measurement, feedback and improvement. These interviews allowed exploration of the measurement feedback system design and implementation factors not necessarily covered in published materials to inform considerations for future measurement feedback system design and implementation.
Methods
Design and setting
This qualitative study used key informant interviews of participants with identified expertise in quality measurement, feedback and improvement. This methodology was selected as it focuses on the knowledge of the expert and allows an in-depth description of context and practices, experiences in quality measurement and feedback, and perceptions of ideal future systems.19
Interview process
The interviews were semistructured with an approved interview guide (online supplemental material file 1) and a responsive interviewing style.20 The interview script was limited to <15 questions to allow in-depth discussion in four key areas; (1) the informants’ organisational context, (2) quality monitoring activities at the organisation, including barriers and enablers, (3) the impact on care delivery and patient outcomes and (4) an ideal quality monitoring system. Key informants focused responses on their professional experiences with one or more specific measurement feedback systems. All interviews were conducted by videoconference between 25 June 2019 and 2 September 2019 by two trained and experienced qualitative researchers (CD and AJ). All interviews were audio-recorded.
Supplemental material
Sample and recruitment
A purposive sampling strategy was used to recruit key informants.21 22 Key informants were defined as international leaders in quality measurement, feedback and improvement. An initial environmental scan was conducted to identify key organisations and individuals who met the inclusion criteria. Potential key informants were identified based on diversity of expertise and affiliations, position held within a relevant organisation, years of work experience and recent publications in quality measurement, feedback and improvement. A deidentified list of potential key informants was presented to the research team for feedback on participant roles and experience distribution. Key informants were recruited by email invitation, along with a participant information statement and consent form. If there was no response to the initial email, one follow-up email was sent. Key informants were recruited until preliminary analyses suggested that key themes had reached saturation.23 24
Patient and public involvement
Patients and/or the public were not involved in the design, conduct, reporting or dissemination plans of this research.
Data analysis
Two researchers conducted the analysis (CD and KS) of transcribed audio-recordings using a deductive coding approach according to the 14 domains of the TDF, then grouped by subthemes generated within each domain. One researcher (CD) coded all transcripts and the second researcher (KS) independently analysed a random sample of transcripts (n=3, 19%).25 The inter-rater reliability between researchers’ coding was calculated, providing a percentage of agreement.26 27 The two researchers met to discuss coding and the generation of subthemes, and any discrepancies were discussed to establish consensus. Coding and validity checks were performed, including an analysis of transcript data relevant to the informants’ demographic characteristics to identify any disconfirming evidence between informants from the same country. One researcher (CD) then generated statements representing the specific beliefs for each subtheme, capturing key informants’ core thoughts of multiple utterances. Each belief statement provides detail about the perceived role of the domain in influencing behaviour.18
The Consolidated criteria for Reporting Qualitative research (online supplemental material file 2) was used to report essential research processes.28 A description of the results is presented with a semiquantified approach to highlight how widespread a particular belief is among the informants.29 30 The following criteria were applied to identify the most relevant theoretical domains and subthemes: (1) relatively high frequency of specific beliefs and themes, (2) presence of conflicting beliefs and (3) evidence of strong beliefs that may affect the target behaviour.17
Supplemental material
Results
Summary of interviews and key informants
A total of 17 key informants participated in the study of the 26 invited experts. Eight invited experts that did not participate did not respond to the initial email or follow-up. Participating key informants included experts with primary positions as clinical (n=5), government (n=5), research (n=4) and health service (n=3) leaders. Almost all informants (n=15) held secondary positions that combined these roles or had extensive prior experience. For example, five informants were specialist clinicians with health service leadership positions with academic affiliations. Informants were from Australia (n=7), the USA (n=4), the UK (n=2), Canada (n=2) and Sweden (n=2). Most informants worked in public healthcare or government settings (n=10), while other informants worked across both public and private healthcare settings (n=4) or exclusively private healthcare organisations (n=3). Informants were mainly male (n=10), although an even ratio of male and female experts was originally recruited. The interview length ranged from 48 to 66 min. Key informants had professional experiences with multiple measurement feedback systems at various levels, from local health services to nationally coordinated systems, including voluntary and mandatory participation and various clinical contexts.
TDF domains and subthemes
Informants provided numerous examples of existing measurement feedback systems and their impact on behaviour change, highlighting their value in delivering high-quality care. However, all informants noted that implementing measurement feedback systems is highly complex, and multiple nuanced internal and external factors affect their success. These factors were identified in coding 505 utterances into 12 of 14 TDF domains. Inter-rater reliability across the interviews coded by two researchers had ‘almost perfect agreement’ (83.8%) across the 12 coded domains. The domains environmental context and resources, memory, attention and decision-making, and social influences were the most populous domains. A total of 38 subthemes and corresponding belief statements were developed across the 12 domains represented in table 1. For clarity, the relevant illustrative quotes for each subtheme are displayed in italics within online supplemental material file 3. The findings of this study have been organised into 11 considerations for future measurement feedback systems, where results have been described in further detail.
Supplemental material
Frequency of domains and subthemes
Financial investment in technological infrastructure and interdisciplinary human resources
Informants regularly cited an underestimation of financial resources as a significant barrier. Interdisciplinary human resources were considered essential to develop, implement and sustain measurement feedback systems. Several contributing roles to an interdisciplinary team included multidisciplinary health professionals, data specialists (ie, coders, data scientists), technology specialists (ie, engineers), health service management, quality improvement officers and researchers. However, some informants identified financial barriers to engaging interdisciplinary teams, for example, the difficulty retaining skilled data scientists in healthcare when private industry salaries are much higher. Some informants expanded on other resourcing issues in hospital quality assurance departments where valuable quality measurement expertise exists, but capacity beyond external reporting requirements is often limited. Other informants highlighted the importance of resources to rectify variation, noting that mobilisation of resources often limits quality improvement to change care delivery.
System-wide collaboration, organisational commitment, ownership and accountability
Social and interpersonal relationships were noted as influential in the design, implementation and sustained use of a measurement feedback system. Collaboration between all levels of healthcare across government, professional societies, industry, health service leadership and health professionals was noted as essential to measurement feedback systems. Informants acknowledged that measurement feedback systems are predominantly government led, but advised that a collaborative ‘top–down–bottom-up’ approach would drive an ideal system. Informants also noted that organisational commitment to quality improvement is critical to long-term success, particularly in areas when improvement is not immediately apparent. Leadership, ownership and accountability were commonly addressed. A sense of ownership was reported as a reflection of legal responsibility and health professional morale to deliver best-practice care.
Clinically relevant, patient-centred measurement and feedback
Informants emphasised that health professionals and patients should determine the measurement focus in an ideal system. Many informants believed that the development of measurement frameworks in the future should be patient centred. Informants noted the importance of clinically meaningful measurement and shared examples of identifying such measures. Furthermore, informants recommended that quality measures are actionable where health professionals influence the outcome. One informant provided an example of developing patient-centred measurement using patient-only focus groups to identify priority outcomes and clinical consensus of process measures linked to identified patient outcomes.
Use of routinely collected data and linkage methods in quality measurement
Data sources for quality measurement were frequently discussed. Many informants criticised the quality and completeness of electronic medical records (EMRs) and clinical registries. However, few informants reported high levels of coverage and high-quality data in their relevant registries. Some informants praised the usefulness of structured EMR data in measurement but acknowledged that most information is collected as unstructured clinical notes. There were synonymous views on lacking data accessibility and interoperability. EMR vendors were described as a barrier to data access; therefore, manual extraction remains the predominant collection method for clinical registries. A lack of interoperability between EMRs and other information management systems was a reported barrier to consolidating data needed for measurement. Although issues surrounding routinely collected data were raised, informants shared the belief that automated extraction and use of EMR data would reduce the burden of data collection in an ideal measurement feedback system. Informants also noted that this would require significant investment in technical infrastructure and clarifying data security and ownership to develop, store and maintain such datasets.
Transparency of data sources, quality and use in analyses
Informants recommended building trust in quality measurement by engaging health professionals in transparent and open dialogue about the data quality and use of data in analyses. One informant provided an example of presenting data to health professionals during the measurement development stage to seek feedback. Another emphasised the value of health professionals challenging quality measures to improve the validity of feedback. Furthermore, health professional engagement was acknowledged as critical to sustained buy-in and translation into improvement.
Review and harmonisation of quality measurement sets
Informants discussed how external factors influenced measurement focus, including mandated government reporting, financial resources and the availability of existing data. These factors contributed to misaligned priorities, lack of clinical relevance and reduced feedback utility. Informants shared the belief that the selection of quality measures is commonly based on the convenience of data already collected and stored, typically administrative claims data and government-mandated reporting data. However, a disconnect between government priorities for reporting and relevance to improving clinical practice was frequently discussed. Informants stressed that existing measures should be routinely reviewed for relevance and impact and harmonised with other measurement requirements to reduce measurement burden and cognitive overload.
Digital tools for the provision of timely, interactive and digestible feedback
Informants highlighted that health professionals receive very little feedback on their care delivery and should be systematically provided to front-line health professionals to ensure it is not siloed in administrative departments. Informants also noted that feedback is often significantly delayed (ie, 2–4 years after the clinical episode), reducing the relevance of data, and recommended that feedback is closer to the point of care to increase impact. Informants acknowledged that health professionals are time-poor and required to juggle competing demands. Therefore, feedback should be digestible and personalised to provide a reference point using peer benchmarking. Informants suggested using technology such as clinical dashboards and interactive data visualisation techniques to display quality measurement insights and described several examples of dashboard development efforts.
Quality improvement culture supporting non-punitive feedback
Fostering a positive quality improvement culture was the second most frequently discussed subtheme. Two subthemes within the emotion domain included health professionals’ fear of judgement and a lack of trust. Informants believed that it is vital to foster a non-judgemental environment where quality measurement acts as a screening tool for reflection rather than a punitive tool for judgement.
Knowledge, skills and capacity of health professionals
Informants discussed the skills and capacity of health professionals to use feedback while balancing the demands of day-to-day care delivery. One informant underscored that the skillset needed for translating measurement data and implementing quality improvement strategies differs from delivering care. Furthermore, transitioning from the mindset of individual patient consultations to the analysis of cohorts of patients can be difficult. While most of these informants highlighted a need to build the skills and capacity of health professionals to interpret and action quality measurement data, one informant believed that an unreasonable onus is put on health professionals to develop such statistical capabilities to analyse and interpret data.
Quality improvement translation supports
Informants recommended assigning health professionals to champion quality measures and establishing team meetings to review feedback and promote an understanding to contribute to achieving quality improvement. Informants also considered dedicated translation officers that interpret measurement data, investigate variation and lead team quality improvement meetings to disseminate information and translate findings into appropriate improvement strategies.
Discussion
Statement of the principal findings
This manuscript presents the experiences and opinions of leaders in healthcare quality measurement, feedback and improvement underpinned by a theoretical framework to systematically identify factors influencing measurement feedback systems. There was significant consensus among experts highlighting important considerations and a unique perspective on how future systems can be improved. Using the TDF, subthemes related principally to environmental context and resources, memory, attention and decision-making, and social influences. The TDF highlighted that while there are some modifiable factors in the design of measurement feedback systems, significant socioenvironmental factors influence their translation into improved care and patient outcomes. The identified factors surrounding health professionals and their engagement with a measurement feedback system are specific to the implementation context. These factors should be considered in optimising future quality improvement interventions. Based on the results from this study, the following set of 11 considerations have been developed for future design and implementation of measurement feedback systems:
Financial investment in technological infrastructure and interdisciplinary human resources.
System-wide collaboration, organisational commitment, ownership and accountability.
Consumer engagement in the development of patient-centred measurement frameworks.
Health professional engagement in clinically relevant measurement and feedback.
Use of routinely collected data and linkage methods in quality measurement.
Transparency of data sources, quality and use in analyses.
Review and harmonisation of quality measurement sets.
Digital tools for the provision of timely, interactive and digestible feedback, that is, dashboards.
Quality improvement culture supporting non-punitive feedback.
Knowledge, skills and capacity of health professionals in local quality improvement.
Quality improvement translation supports, that is, clinical team quality improvement meetings, translation officers.
Interpretation within the context of the wider literature
It is well-documented that the impact of measurement feedback systems is variable.5 6 These interviews highlighted that while there is a lack of published evidence providing consistent and definitive correlations to improved quality of care and patient outcomes, rich information exists in expert knowledge. The frequency of socioenvironmental subthemes presented in this study may explain some variation between published interventions. This study generates new knowledge on how to apply theory to explore a wider set of factors affecting measurement feedback systems. One other study by Tuti et al11 used the TDF in a systematic review analysis of seven audit and electronic feedback trials and found the most frequently coded domains in the intervention arm were knowledge, motivation and goals, and social influences. This study similarly found social influences as the third most coded domain. However, environmental context and resources and memory, attention and decision-making were coded considerably more, which did not feature highly in the review analysis. The review did not identify any studies whose interventions targeted social/professional role and identity or emotion. Both of these domains which focus on the health professional were identified in this study. The differences between the review’s findings and this study may be due to the limited implementation context provided in published intervention studies, increasing the value of theory-informed qualitative evaluations.31 32
A recently published review of individual and system competencies within learning health systems identified consistent themes with this study, such as health professional knowledge and skills, leadership and teamwork, technological skills, data science, infrastructure and standardisation, integration of data, and culture.33 Learning health systems are a concept of self-monitoring and improving performance through continuous learning cycles supported by people, policy and processes. Measurement and feedback is an essential component of a learning health system, and therefore it is encouraging that the findings of this study have incorporated all competencies in the review. Furthermore, the 11 considerations for future design and implementation of measurement feedback systems highlighted in this manuscript are aligned with literature related to patient prioritisation of measurement focus,34 quality improvement culture, collaboration and leadership,35 36 health professional knowledge and skills37 and the use of dashboards in feedback.31 38 39 This study articulates complementary considerations for measurement feedback systems to the literature, including financial investment in technological infrastructure and interdisciplinary human resources, translation support roles for quality improvement and the need to work with EMR vendors to support use of routinely collected data.
Strengths and limitations
This study has achieved the synthesis of key informant interviews with perspectives and experiences of experts involved in quality measurement and feedback. The data demonstrated relevance to the TDF and identified commonalities irrespective of geographic location, deeming the findings transferrable to various healthcare contexts. The research methodology has limitations. By purposively sampling experts as key informants, the selection of key informants may only be representative of some viewpoints, particularly as experts are likely to be employed by organisations with fewer barriers to quality improvement, that is, organisations with a strong improvement culture. While the response rate for participation in this study was considered acceptable, non-respondents restricted the geographical diversity, health service leaders and allied health professional representation of the study sample. Non-respondents included additional experts from the countries covered in the sample, as well as Asian and Middle-Eastern countries, thus potentially limiting the global applicability of these findings. It may be helpful to understand the prioritisation of TDF domains by expert type but that is impossible to uncover in a sample of this size with generalisable results. Although there were some noted differences between perspectives of clinicians and government leaders, it is challenging to associate views with that population unless interviewing more key informants of each type. The TDF also has limitations in that it does not specify relationships and patterns between domains. Additional focus group discussions could further clarify these relationships. Lastly, some utterances could have been coded to multiple TDF domains, but the coders assigned each to a single domain. This may have reduced coding frequency in some domains with secondary relevance.
Implications for policy, practice and research
The insights presented in this manuscript contribute to progressing the science of quality measurement, feedback and improvement in an area where traditional studies are often not replicable and diminished by the influence of the implementation context. This qualitative study provides important and contextual information on barriers and enablers to effective measurement feedback systems and how they could be improved. In particular, the 11 considerations presented in this manuscript can be used to guide health service frameworks for measurement feedback systems and prioritisation of barriers and enablers to be addressed. Although informants believed quality measurement is valuable to healthcare quality, many shared challenges in demonstrating success and is an area of needed improvement in health services research. The study findings have practical implications for future measurement feedback systems implementation. The applicability to 12 TDF domains demonstrates that simply measuring quality and producing reports is inadequate. Quality of care delivery is more than a process but a highly-complex human issue influenced by socioenvironmental factors. There is a need for adequately resourced, evidence-based approaches to promoting the implementation and evaluation of measurement feedback systems supported by theoretical frameworks.
Conclusions
Multiple factors influence measurement feedback systems; thus achieving an optimal system requires a multifaceted approach. Local assessment of barriers and enablers is needed to identify factors salient to the implementation context and to develop appropriate implementation strategies. In an ideal system, routinely collected data would be automatically extracted into consolidated purpose-built clinical quality registries near real-time. This data would be used to measure clinical care selected by patients and health professionals to ensure that any insights derived are patient-centred and relevant for improving care delivery and outcomes. Digital tools would highlight meaningful measurement insights and visualise them in a digestible and interactive format. Timely access to these digital tools would be available to front-line clinical staff and executive leadership. These insights may be personalised to the individual and teams delivering care and benchmarked appropriately to provide context. Quality improvement translation resources would support the interpretation of feedback in a non-punitive, collegiate environment led by trained health professionals and organisational leaders. This approach may enhance quality measurement and feedback systems and ultimately improve care delivery and patient outcomes. Furthermore, appropriate evaluation to effectively capture potential improvements to care delivery, patient outcomes and cost-benefit will have broader implications for healthcare systems and society.
Data availability statement
All data relevant to the study are included in the article or uploaded as an online supplemental information. To safeguard the privacy of the participants and those involved in their services, we cannot make the transcripts available.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants and was approved by the study was approved by University of Sydney Human Research Ethics Committee (USYDHREC 2018/422). Participants gave informed consent to participate in the study before taking part.
Acknowledgments
The research team would like to thank the key informants for their time.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @CandDonn, @shalinivinod13
Contributors CD developed the search strategy with the guidance of AJ, TJS, SV and PH. CD and AJ conducted the interviews. CD and KS completed the interview analysis. CD drafted the manuscript with contributions and review from AJ, PH, TJS, SV and KS. CD is the guarantor of this work and the conduct of this study, had access to the data, and controlled the decision to publish.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.