Article Text
PDF
Abstract
Objectives Explore how patient-reported experience measures (PREMs) are collected, communicated and used to inform quality improvement (QI) across healthcare settings.
Design Systematic review.
Setting Various primary and secondary care settings, including general practice, and acute and chronic care hospitals.
Participants A full range of patient populations from (children through to the elderly) and staff (from healthcare practitioners to senior managers).
Methods Scientific databases were searched (CINAHL, PsycINFO, MEDLINE and Cochrane Libraries) as was grey literature. Qualitative and quantitative studies describing collection of PREM data and subsequent QI actions in any healthcare setting were included. Risk of bias was assessed using established criteria. Of 5312 initial hits, 32 full texts were screened, and 11 were included.
Results Patient experience data were most commonly collected through surveys and used to identify small areas of incremental change to services that do not require a change to clinician behaviour (eg, changes to admission processes and producing educational materials). While staff in most studies reported having made effective improvements, authors struggled to identify what those changes were or the impact they had.
Conclusions Findings suggest there is no single best way to collect or use PREM data for QI, but they do suggest some key points to consider when planning such an approach. For instance, formal training is recommended, as a lack of expertise in QI and confidence in interpreting patient experience data effectively may continue to be a barrier to a successful shift towards a more patient-centred healthcare service. In the context of QI, more attention is required on how patient experience data will be used to inform changes to practice and, in turn, measure any impact these changes may have on patient experience.
- PRIMARY CARE
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This review draws together emerging evidence on the use of patient-reported experience measures for quality improvement in healthcare settings in the USA and Europe.
It is a relatively new field, so there were a limited number of studies eligible for inclusion.
Some relevant interventions may not be published or publicly available, for instance being held by National Health Service (NHS) Trusts.
None of the included studies reported formal quality improvement methods.
More research is needed in order to get a better understanding of the use of patient experience data in quality improvement.
Alongside measures of clinical and safety outcomes, patient experience is increasingly recognised as an important indication of the quality of healthcare provision in Western societies and is frequently cited in national and international health policy.1 ,2 Healthcare organisations in England are required to review patient experience as part of quality and performance reporting and proportions of service funding in some areas are contingent on achieving improvements in patient-reported experience.3 Aside from wanting to provide a healthcare service that considers patient needs, the use of patient experience perspectives in healthcare may also have other benefits. For example, a recent systematic review4 found that higher levels of positive patient experience were associated with higher levels of patient safety and clinical effectiveness across disease areas, study designs and self-reported and objective outcomes (eg, mortality, greater adherence to treatment recommendations and lower use of additional healthcare, such as repeat hospitalisations and overuse of primary care).
Over the past 10 years, there has been increased attention to patient satisfaction and experience, and all healthcare organisations in England are required to conduct some form of patient satisfaction survey.5 Patients' experiences of healthcare services are increasingly visible to the public through online resources such as NHS Choices. The Friends and Family Test was implemented throughout the National Health Service (NHS) in 2015. However, there is debate over the reliability, validity and usefulness of these data6 and a lack of clear evidence of how to make use of such data to meaningfully inform improvements to the quality of care.7–9
The collection and use of patient-reported experience measures (PREMs) for the purposes of quality improvement (QI) has become part of a relatively recent move towards more holistic, ‘patient-centred’ provision of care in a range of countries.4 However, there is little evidence to show that collecting patient experience data leads to sustained improvements in patient satisfaction within healthcare settings.10 In order for patient experience data to be a meaningful source of service change within healthcare settings, there is a need to examine organisational factors that promote meaningful use.8 ,11 ,12 A particular difficulty is enacting and measuring change at all levels of the organisation with few studies exploring links between organisational levels. Barriers around an organisation's ability to use patient experience data include time and resources available for staff to collect and analyse data;12 ,13 it remaining a priority over time; clear structural plans for use in QI;8 ,11 ,12 and staff ownership of data and freedom to set directions for QI.9 ,14
Although some studies suggest such data can be used to contribute to practice change in the USA15 and Canada,16 evidence of positive use in England is still very limited.7 ,9 This may be partly due to a lack of consistency in measurement of patient experience and variation in the ways in which these data are used across healthcare settings.17 Additionally, while these surveys can indicate a general level of satisfaction with healthcare, they offer little in the way of an understanding about why patients may be satisfied or not with the services they receive, and therefore may be of limited use in determining where QI might have most impact. There are also indications that clinical or administrative staff may lack the required skills and expertise in QI implementation to be able to effectively use patient experience data to identify areas for improvement, as well as design and evaluate the impact of interventions.18
In times of financial constraints, healthcare authorities and managers are compelled to find the most cost-effective and efficient means of providing high-quality healthcare services to patients. QI programmes have been cited as a useful means of doing this.4 ,17 However, what is unclear from the current research literature is what the optimal ways of collecting PREMs for the purposes of QI might be and how information collected might best be fed back to clinicians. We also know little about the barriers and facilitators that need to be considered to ensure that the time and money spent collecting PREMs is used to its full capacity.
A number of authors have reported that merely providing feedback on patient experience surveys is not enough to effect change in the quality of healthcare practice and that surveys per se should not be seen as QI tools.14 ,19 A review of the extant literature is clearly needed to explore the ways in which patient experience data are collected and used in healthcare settings and the facilitators or barriers to their use.
Aim of the present research
The aim of the present research was to systematically review the extent of use of PREMs to inform QI across healthcare settings and the barriers and facilitators experienced in collecting and using PREMs for QI. Specifically, two research questions were addressed:
How are patient experience data collected, communicated and used for QI in healthcare settings?
What are the main barriers and facilitators to using patient experience data for QI in healthcare settings?
Methods
Search strategy
A protocol was developed for the present research and an initial scoping of the literature was conducted to assist in refining the protocol and research questions and to identify any existing studies that fit the inclusion criteria and we would expect to be retrieved by the search strategy. This scoping search identified inconsistencies in the indexing of relevant literature, which led us to construct a broad ‘low threshold’ search strategy. Subsequently, a full search was conducted in four databases: CINAHL, PsycINFO, MEDLINE and the Cochrane Libraries (Central and Reviews) alongside an internet search for any relevant grey literature and hand-searching of reference lists of relevant papers.
Searches were conducted for papers published from 1990 to May 2015.i Owing to the diversity in use of terms surrounding patient experience and QI,20 a range of combinations of search terms for keywords used in the papers found through the scoping exercise were used (see online supplementary appendix 1). For this review, patient experience was defined as identified from a systematic review of definitions: “The sum of all interactions, shaped by an organization's culture, that influence patient perceptions across the continuum of care,”21 while QI referred to the application of “interventions to improve the effıciency or effectiveness of a program, process or organization” which may include reducing inefficiency and error.22
Supplementary appendix
Inclusion criteria
Studies were deemed eligible for inclusion if they were based in any healthcare setting. We also included studies that covered the full range of patient populations from children to the elderly. Both qualitative (eg, interview and case study) and quantitative studies (eg, randomised controlled trial (RCT), survey and cross-sectional) were considered for inclusion (also see online supplementary appendix 1).
Search flow
The present research adhered to the guidelines presented in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2009 checklist.23 Based on best practice guidelines,24 ,25 the review followed the stages described in figure 1 below.
PRISMA flow chart.
The initial search returned 5312 papers, which were imported to Endnote for screening; after removing duplicates 5226 papers were retained. To ensure reliability, two authors (HG and AC) screened ∼20% of the titles and abstracts against the inclusion criteria with 93% agreement. After discussing inconsistencies, the same two authors completed the screening of the rest of the titles and abstracts with 98% agreement; the few inconsistencies were resolved through discussion. This initial screening led to the inclusion of 32 papers. The main reason for exclusion of papers at this stage was that the papers did not report on the use of PREMs for QI—in some cases, QI projects were reported on but they were not based on any patient experience data. Moreover, a large number of studies were editorial commentary or theoretical or narrative overviews of PREMs and/or QI rather than empirical studies. These papers often sought to argue in favour of the use and collection of PREMs data for QI and to lay out the potential benefits of doing so for patients and staff. This would suggest that the study of improvement in healthcare from a patient experience perspective is an emergent field with further research needed to develop a comprehensive body of evidence to suggest what approaches are most effective.
Full-text articles were retrieved and assessed for inclusion by the first author (HG), of which 11 were retained for final inclusion. The main reason for exclusion at this stage was that papers stated that they were reporting on QI in the abstract but did not give any details of QI approaches; some papers presented overviews of QI approaches used in healthcare settings, but these were not detailed enough to extract useful information for the purposes of this review.
A random selection of full texts were screened by the second author (AC) who also extracted data from these papers, so that forms could be compared for agreement/disagreement and to ensure that all relevant information was captured and referenced. Data analysis involved the comparison of included studies and extracted data. As all but two of the studies used qualitative methods, a narrative synthesis was deemed most appropriate for this review. Synthesis was developed through examination of relevant themes, according to the two research questions of the review, within and across the studies.26
Risk of bias assessment
We chose a tool that would allow for all included studies to be assessed using the same criteria, regardless of the particular methods used, for a more consistent risk of bias assessment across studies. Hence, the assessment criteria developed by the Evidence for Policy and Practice Information and Co-ordinating (EPPI) centre were used27 as these criteria are based on the rationale of qualitative research. These seven criteria are listed in box 1 and were applied to each individual study included in the review, with each study being assigned a rating of A to D as shown in table 1. Similar appraisal systems have been used in previous qualitative synthesis studies in other areas.28
Risk of bias assessment (adapted from Rees et al27)
1. Theoretical framework and/or literature review. Does the study provide an explanation and justification for the focus of the study and methods used? How does the study link to an existing body of knowledge?
2. Does the study have clearly stated aims and objectives?
3. Is the context and setting of where the research was developed and carried out explicitly described?
4. Does the study provide adequate details of the sample used in the study? This should include as much detail as possible about the sociodemographic details and any salient factors that are relevant to this area of study, so that an accurate judgement of the quality of sample selection can be made.
5. Does the study include a clear description of the methodology used, including data collection, choice of collection tools or approaches, and analysis used?
6. Is there evidence that the researchers tried to establish the reliability and validity of the data analysed?
7. Are sufficient original data examples (eg, direct quotations) reported to assist the reader in making judgements on the interpretation of that data? Can a clear connection be seen between the data and the interpretations of that data and the conclusions drawn?
Summary of included studies
A=no, or few, flaws;
B=some flaws;
C=significant flaws which may affect the validity of the study findings;
D=untrustworthy findings or conclusions.
Results
Study characteristics
Of the 11 studies included in the final review, 3 were based in general practice10 ,15 ,29 and the rest were in either acute or chronic care hospital settings.5 ,30–36 Five were from North America,10 ,15 ,30–32 five from the UK5 ,33–36 and one from the Netherlands.29 All included studies were based on adult populations under healthcare services for physical (as opposed to mental) illnesses, although population characteristics were not generally given within studies as they focused primarily on clinician responses to and use of experience data rather than the results of such data.
Nine of the 11 studies were qualitative investigations of the collection and/or use of PREMs for QI, or the barriers and facilitators to implementing QI initiatives based on patient experience feedback.5 ,10 ,15 ,30–34 ,36 Of these nine, six were qualitative follow-up studies with staff on the methods they used to collect, understand and use patient experience survey data.10 ,15 ,30–32 ,34 Three studies described an evidence-based co-design (EBCD) approach to collecting patient experiences and determining areas for QI projects.5 ,33 ,36 Two studies29 ,35 used an RCT to measure differences in QI work across groups depending on the type of feedback and support given.
The most common means of collecting patient experience data was through postal questionnaires.10 ,15 ,30 ,31 ,34 ,35 This is despite many authors acknowledging that clinicians and ward staff generally find qualitative comments more interesting and relevant than numerical data.18 However, qualitative data were seen to be more problematic than survey data as it required more time and expertise to collect and analyse.33
Table 1 below outlines the details of the QI initiatives described and measured in each of the included studies. The QI intervention labels are modified from those described in Ovretveit37 and include clinical intervention—changes to how patients are treated; behaviour—changes to clinician behaviour or thinking; service delivery—changes to processes such as appointment bookings or teamwork among staff; service infrastructure—changes to support systems and physical structures; and patient education—changes to information given to patients about their treatment and/or care.
How PREMs data are collected, communicated and used for QI in healthcare
In 7 of the 11 included studies, PREMs were initially collected through questionnaires,10 ,15 ,29–31 ,34 ,35 with 6 of these being national, regional or otherwise standardised surveys for use across different geographical areas.10 ,15 ,30 ,31 ,34 ,35 These surveys were created and tested by outside organisations for use within healthcare settings and tended to have little or no staff input in their development. One study35 tested the use of a national survey that was then modified to the ward under study but kept close to the original in the number of items used and the domains included.
Three further studies used EBCD to gather patient experience data.5 ,33 ,36 This approach uses video-recorded patient narratives of their experience, which were then shared with staff at collaborative meetings with patients. Action groups discussed issues raised within these narratives and identified directions for improvement and change. Within this approach, there was no ongoing measurement of patient experience per se, but patient participants reported back to the group on the effects of changes they had seen and experienced.
The final paper32 described a patient shadowing approach, which involved an empathetic observer recording a patient's experiences and producing a report and flow chart to highlight areas that were in need of attention. The report was presented to clinicians, and potential target areas for improvement were identified and discussed. The use of this report and the effects of improvements identified were not detailed.
The method of communicating data to staff or collaborative groups was influenced by the type and means of data collection: (1) a written data report that could include percentages of satisfaction from survey items and graphical displays of survey data,15 ,29–31 ,34 (2) video narratives followed by clinician–patient discussion groups with the EBCD approach,5 ,33 ,36 or (3) collaborative discussions between staff groups or patient and staff groups.10 ,32 ,35 Some of the collection procedures used a mixture of communication approaches; for example, the ward-specific survey35 included a staff meeting to discuss the written report produced to communicate statistical data findings.
None of the studies included in this review reported the use of formal QI methods of data collection, analysis or reporting such as Total Quality Management (TQM), Continuous Quality Improvement (CQI), Business Process Reengineering (BPR), Lean thinking and Six Sigma.38 Generally, organisations reported using questionnaire data, not collected or analysed by themselves, to both identify areas in need of improvement and to assess the impact of changes made.10 ,15 ,30 ,31 ,34 Most studies (9 out of 11)5 ,10 ,15 ,30–34 ,36 used qualitative rather than statistical methods to measure PREMs' impact on QI and one paper noted that, while staff reported having made effective improvements, the authors found it difficult to identify clearly what those changes were or the actual impact they had.35 The two studies that measured changes in PREM questionnaire results after QI interventions29 ,31 reported no statistical improvements.
Organisations using questionnaire data as a benchmark of patient experience tended to focus on the lowest scoring items, or areas that staff had already identified as needing improvement. In many cases, staff reported using patient experience data not only to identify areas for improvement but also as a means of validating existing improvement efforts.15 ,29–31 ,34 ,35 Across all studies, changes to administrative practice (eg, appointment management) and patient education (eg, producing discharge materials, medication guides) were the most commonly targeted areas. Least common were structural changes, clinician behaviour or communications. Overall, the more successful uses of PREMs data in terms of improvement in patient experience were those that involved small, incremental changes that did not require changes in clinician behaviour.31 ,33 ,36 The EBCD approach appeared to generate more improvement efforts than questionnaire-based PREMs data but effects of QI interventions were not measured or reported on. The study on patient shadowing32 did not measure any changes made based on reports or flow charts created through observation.
Barriers and facilitators to using PREMs for QI in healthcare
Table 2 below outlines the barriers and facilitators identified in each of the studies included in this review. Five of the studies included reports of barriers and/or facilitators to using PREMs data for QI.10 ,30 ,31 ,33 ,34
Barriers and facilitators identified by individual studies
Barriers associated with data collection or use included a lack of understanding and expertise, poor specificity of results, and the timing of feedback. National surveys in particular were criticised by clinical staff as lacking specificity, with findings not applicable to, or relevant for, their own organisation or setting. Despite most survey results being analysed by outside organisations and presented to healthcare staff using non-technical language, many primary studies reported a need for staff training in data analysis and statistics to facilitate full understanding and use of results.10 ,30 ,31 ,34
Organisational barriers were most common and included a lack of time or resources to collect, analyse or act on data.30 ,31 ,33 ,34 Competing priorities, such as financial goals and the number of patients that needed to be seen, were reported to be additional barriers in two studies,10 ,33 which prevented staff from spending the time necessary to fully engage in QI initiatives. Further barriers were an organisational culture or staff resistance to QI improvement initiatives and a lack of engagement or support for change from management.33 Survey results that were not frequent or timely were seen as a barrier to making and sustaining successful improvements.10 ,31 ,34
Organisational facilitators included working in a culture supportive of improvement, change and patient views.10 ,30 ,33 ,35 Management support and encouragement were seen to be key in facilitating staff motivation and engagement with improvement.10 ,30 ,33 Allowing dedicated time for staff to discuss results and plan improvements was also seen as a factor for successfully implementing improvements.10 ,36
For the EBCD approach, patient involvement in identifying areas for improvement and ways to enact them was seen as key to its perceived success. The co-design nature of collecting and using data allowed for a collaborative relationship between staff and patients to develop. The regular meetings promoted a sense of community and allowed both groups to see the connections between experiences and changes.5 ,33 ,36
The ward-specific survey35 was reported to be useful in facilitating a sense of ownership over improvement actions in nursing staff. Dedicated meeting time to discuss survey results and improvements helped to reduce scepticism about the relevance of results. Where national surveys were considered high quality and well validated, this was reported to facilitate clinician support and helped foster a patient-centred culture.31 ,34 For some organisations, using national publicly reported surveys acted as an incentive to use PREMs results for QI.10 ,31 ,34
Discussion
The aim of the present research was to systematically review the extent of use of PREMs to inform QI across healthcare settings and the barriers and facilitators experienced in collecting and using PREMs for QI. The most common form of collection of PREMs in healthcare settings was questionnaires, often administered by outside agencies with limited organisational support to interpret and make use of the results. There was limited evidence of these being used to support QI, perhaps because staff found the data too far removed from their day-to-day services10 or because they did not measure domains felt to be important,18 and/or because they found it a challenge to translate findings into actionable QI interventions.30 Whether patient feedback pertained to organisational or more specific clinician experiences appeared to make little difference in the ultimate use of PREMs to initiate QI interventions, although this was not a particular focus of any of the studies included in the present review. The most popular areas for improvement were in changes to processes for admissions or waiting times and producing educational materials for patients. These were chosen as relatively easy targets that did not require any major structural or practice changes.10 ,30 ,31 The least common areas to be targeted were those that involved upgrading infrastructure or changing clinician behaviour. Approaches more prominently involving patient voice and co-designed with patients5 ,32 ,33 ,36 seemed to be the most acceptable to practitioners, although even here evidence of impact was limited. The identified barriers and facilitators were broadly similar for all approaches, with barriers including a lack of time, resources and expertise in data analysis and QI. Findings from this review suggest that staff are interested in gaining a better understanding of using PREMs for QI, as they frequently cited a lack of expertise as one of the main barriers. Without structured training across healthcare settings for all relevant staff, this may continue to be a barrier to a successful shift towards a more patient-centred healthcare service.
Limitations should be considered when interpreting the findings of this review. As the field is relatively new, there was not a large body of published literature on QI linked to patient experience data in healthcare. It is possible that publication bias means that there is more unpublished evidence of non-significant or negative findings and that many more interventions take place within healthcare settings with findings held locally for use by the Trust Board, but that the design or outcomes of these interventions are not published or otherwise publicly available. Nevertheless, this review included published grey literature in an attempt to overcome these biases and include findings from some of these interventions. In addition, despite the included studies varying in their research designs and methodologies, common themes across studies were identified which provide useful information for future investigations of how to collect and disseminate PREMs data for use in QI.
There is some evidence4 that more positive experiences of healthcare can have tangible benefits for patients, which in turn helps to reduce the burden on overstretched services. In recent years, all healthcare organisations within England have been required to collect data on patient experiences, but the particular expertise needed to be able to conduct effective and meaningful data collection, analysis and interpretation appears not to have been provided to any great extent. This can be seen from clinician and staff reports that, while often they believe patient experience reports are important in their organisations, they also state that they have neither the time nor the expertise to use these data to any great effect.
In contrast to previous research which solely examined the associations of patient experience,4 ,6 we examined the methods for use of patient experience data for QI. It is concerning that none of the healthcare settings included in these studies employed any formal methods of QI, such as TQM or CQI,38 to identify targets or monitor progress. This may be due to the reported lack of expertise in QI and a lack of confidence in interpreting patient experience data effectively.10 ,30 ,31 ,34 Being able to measure and demonstrate the impact of QI interventions is an important aspect of this work and is likely to be a requirement for securing funding for interventions in the future. Reports or impressions from staff alone,35 where staff reported multiple improvements but the authors were unable to identify the actions taken or changes to patient experiences, are not as useful as objective assessments or measurement of outcomes. Practitioners in some studies reported that they were unsure what to do with PREMS data and were not given any guidance on how to implement improvement in work where needed.10 ,30 ,31 ,33 ,34
There is scope for further research based on the findings from the present review. Specifically, there is a need to find means of collecting patient experience data that are most likely to be accepted and used by clinicians. Studies report that qualitative data appear to hold more meaning to staff, but are time consuming and there is no evidence that they lead to tangible improvements in services. The acceptability of survey data to clinicians varied across studies. None of the included settings used formal QI procedures or means of measuring change. This makes it difficult to compare improvements across settings, which is necessary if this field is to move forward, and is likely to become a requirement of funding commissioners.
Our findings show that there is no single best way to collect or use PREM data for QI in healthcare, but they do suggest some key points to consider when planning such an approach. In healthcare settings generally, patient experience data are most commonly collected through surveys and used to identify small areas of incremental change to services that do not require a change to clinician behaviour. In the context of QI, more attention is required on how PREM data will be used to inform changes to practice and, in turn, measure any impact these changes may have on patient experience. Importantly, sufficient resources and leadership support are needed for the collection and consideration of data, which need to be relevant to the local context. Formal training is also recommended, as a lack of expertise in QI and confidence in interpreting patient experience data effectively may continue to be a barrier to a successful shift towards a more patient-centred healthcare service.
Acknowledgments
This is an independent report commissioned and funded by the Policy Research Programme in the Department of Health. The Child Policy Research Unit (CPRU) is funded by the Department of Health Policy Research Programme. The authors thank the members of CPRU: Terence Stephenson, Catherine Law, Amanda Edwards, Ruth Gilbert, Steve Morris, Helen Roberts, Cathy Street and Russell Viner. The authors would also thank Lisa Arai for helpful comments on an earlier draft of the paper.
References
Footnotes
Funding Department of Health.
Disclaimer The views expressed are not necessarily those of the Department of Health.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.
↵i Subsequent to our scoping of the literature—the purpose of which was to inform our search strategy for the systematic review and to assess the volume of research on measuring, analysing and using patient experience data for the purposes of QI interventions—it became clear that little had been published prior to 1999. In order to keep the scope of the review as broad as possible and so that any relevant literature would not be missed, the authors reached agreement through discussion that a 25-year span would be sufficient to ensure that empirical papers likely to meet our inclusion criteria would be discovered.