Article Text
PDF
Abstract
Objectives The Friends and Family Test (FFT) is commissioned by the National Health Service (NHS) in England to capture patient experience as a real-time feedback initiative for patient-centred quality improvement (QI). The aim of this study was to create a process map in order to identify the factors that promote and limit the effective use of FFT as a real-time feedback initiative for patient-centred QI.
Setting This study was conducted at a large London NHS Trust. Services include accident and emergency, inpatient, outpatient and maternity, which routinely collect FFT patient experience data.
Participants Healthcare staff and key stakeholders involved in FFT.
Interventions Semi-structured interviews were conducted on 15 participants from a broad range of professional groups to evaluate their engagement with the FFT. Interview data were recorded, transcribed and analysed for using deductive thematic analysis.
Results Concerns related to inefficiency in the flow of FFT data, lack of time to analyse FFT reports (with emphasis on high level reporting rather than QI), insufficient access to FFT reports and limited training provided to understand FFT reports for frontline staff. The sheer volume of data received was not amenable to manual thematic analysis resulting in inability to acquire insight from the free text. This resulted in staff ambivalence towards FFT as a near real-time feedback initiative.
Conclusions The results state that there is too much FFT free text for meaningful analysis, and the output is limited to the provision of sufficient capacity and resource to analyse the data, without consideration of other options, such as text analytics and amending the data collection tool.
- health informatics
- health policy
- health & safety
- quality in health care
Data availability statement
Data sharing not applicable as no data sets generated and/or analysed for this study. Recruitment began before 1 January 2019, therefore no data sharing plan is included.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
This study reveals for the first time the complex stakeholder interaction involved to process the Friends and Family Test.
Stakeholder interviews strengthened the value of using associated free-text data and applying innovative approaches for real-time application.
This was a single site study with a small sample size, which may not be representative of other hospitals in England.
Opinions from staff such as healthcare assistants, administrative staff and student nurses would have been valuable.
Background
Over the last decade there has been a steadily increasing focus on collecting patient experience data in real time or near real time, with the aim of using it to improve care delivery. A real-time approach to collecting data are anticipated to increase the chance of feedback being put to effective use as staff have a greater sense of ownership of the results; the data are more recent and have the potential to be more granular.1 2 In the English National Health Service (NHS), near real-time feedback is collected via a national policy directive, the Friends and Family Test (FFT).3 In addition to the Likert scale response, there is also a free-text option. This enables organisations to identify the ‘why’ behind responses, providing a richer, more nuanced picture of patient experience. A review conducted by NHS England3 showed FFT’s capability for delivering real-time feedback was found to be a particular strength for its use in local service improvement. In response the FFT has gone through a development process with changes effective from April 2020. One area of focus is encouraging NHS providers and commissioners to actively generate insight from the free-text portion of FFT feedback and use it to drive changes locally and in near real time.
Despite a growing emphasis on gathering feedback in near real time, the effectiveness of this approach for driving service improvement remains largely under-researched.2 4 Kasbauer et al5 evaluated the barriers and facilitators of real-time feedback, relating to technology, volunteer and staff engagement. However, the study focused on older patients, specifically those aged 75 and above, and feedback was elicited using a bespoke survey. The present study is uniquely different from previous research in that it evaluates barriers and facilitators that are specific for FFT as a near real-time feedback initiative. To address this and the recent national policy changes on FFT reporting, there are two aspects that need to be understood; first, how FFT as near real-time feedback initiative cascades in a secondary care setting, and second the level of engagement from key stakeholders, in particular frontline staff. This is because the success of any survey approach for generating improvements in patient experience requires staff engagement and their involvement in interpreting and using the results for quality improvement (QI).1 4–7 Therefore, the aim of this study was to create a process map in order to identify the factors that promote and limit the effective use of FFT as a real-time feedback initiative for patient-centred QI. The objectives were to; explore staff engagement of FFT in a secondary care setting; and categorise the responses according to ‘factors that promote’ and ‘factors that limit’ effective use of FFT.
Methods
Study design
Semi-structured interviews were used, which suited the exploratory aims of the study.
Setting
This study was conducted at a large London NHS Trust. The Trust caters for a population of approximately 1 000 000 people across five sites. Services include accident and emergency, inpatient, outpatient and maternity, which routinely collect FFT patient experience data. FFT data are collected via tablets, kiosks, short message service (SMS) and paper/cards. The outpatient department employ SMS, and inpatient employ tablets as the main mode of FFT administration.
Sampling and recruitment
Through purposeful sampling we began by identifying staff within the patient experience team followed by lead nurses and junior staff in each of the four services. Purposeful sampling is a technique widely used in qualitative research for the identification and selection of information-rich cases for the most effective use of limited resources.8 This involves identifying and selecting individuals or groups of individuals that are especially knowledgeable about or experienced with a phenomenon of interest.9 Participants who met the following criteria were identified; (i) direct or indirect involvement in patient care, (ii) satisfactory overview of patient experience feedback and (iii) current of previous engagement with patient feedback data including FFT.
Data collection
An invitation letter and a participant information sheet were emailed and hand delivered to all participants. Written informed consent was obtained prior to interview participation. The interview guide and topic list were designed based on the work engagement model.10 Topics included perceived resource and capacity, perceived usage, interoperability and the perceived impact of FFT. Interviews conducted in the hospital premises took 30–60 min. Due to the semi-structured nature of the interviews, other questions emerged from dialogue and these were followed-up as an iterative process. For the purpose of open discussions, any information that the participant wanted to retract was deleted from the transcript. No demographic information was collected aside from the role of each participant. The interviews were transcribed verbatim and double-checked for inaccuracies. To aid trustworthiness of data collection, the first author checked accuracy against interview audio-recordings, participants were asked to review the transcript for their interview and any sensitive comments were retracted prior to analysis.
Data analysis
Transcripts were transferred to NVivo (QSR International) where they were analysed using applied thematic analysis.11 Thematic analysis of interview data were undertaken following the ‘framework method’12 and commenced after the first interview. Framework method is a transparent and iterative process of analysing qualitative data. It allows the researcher to incorporate both deductive and inductive codes which was appropriate for this study where specific questions in relation to effectiveness were identified a priori, but experiential aspects were not. It involved five iterative stages of analysis: familiarisation, identifying thematic framework, labelling, charting and mapping and interpretation.
During familiarisation with data, the transcripts were read several times and both initial deductive and inductive codes were identified. Deductive codes originated from questions related to the topic guide. The conceptual framework was developed and discussed with the coauthor prior to the next phase of analysis. Associated keywords, for example, for resource could indicate positive emotions or expressions such as ‘easier to understand’, associated keywords for a demand could indicate negative emotions or expressions such as ‘there is no time’. We also made notes if such comments were made in relation to one role or if these affected other roles. An open-coding strategy was used whereby descriptive codes were attached to participant quotations, staying close to participant wording. One quotation could contain multiple codes. Coding was performed manually by the first author. Peer checking was employed to aid credibility and confirmability of data analysis, whereby two transcripts were open-coded by a second author (KF). Differences in coding or interpretation of the thematic framework were resolved by discussion between the authors.
During the labelling phase, the thematic conceptual framework was applied to the entire data set to ensure total coverage and further developed through the iterative process if new areas were identified. Charting is a process for summarising and synthesising the data to facilitate identification of thematic links and was conducted using a thematic matrix. The final phase is mapping and interpretation in which the final categories and their relationships and interactions are described. This process was facilitated through diagrammatical representations of the themes and critical discussion with the research team (MK, KF, DM, SH-W and EM) to ensure themes were comprehensive and enhance the depth of analysis. Disagreements were resolved through peer debriefing until clarity and consensus were obtained.
Process mapping—visualising the flow of FFT
We summarised data from the semi-structured interviews to create a process map that demonstrates in detail how FFT data cascades from the point of collection, analysis, to dissemination. The process map also depicts the interaction of stakeholders involved and how FFT reports are processed as a near real-time initiative. By creating a process map, we are better equipped to understand what happens to FFT, where the process and organisational problems are and identify areas for improvement.
Patient and public involvement
Working as a collaborative group enabled shared decision-making, with patient and public involvement and engagement (PPIE) at key stages throughout the project moulded the project to be patient-centric. In addition to a lay representative who was part of the steering group, we presented our proposal to the Research Partners Group (RPG) at the Imperial Patient Safety Translation Research Centre. The RPG positively impacted our research project, we learnt about patient and public involvement and the value of it and RPG members also benefited from their participation. Using this approach we noted that there was equality of legitimacy and value in inputs from all those involved, whether suggestions entail large-scale or small-scale changes. During the initial stages where our protocol was being refined, feedback from all individuals from the PPIE group was invaluable.
Results
Thirteen participants were interviewed initially and analysed. Once the data appeared to have reached close to thematic saturation, two further interviews were conducted and analysis confirmed thematic saturation had been reached.13 Mean interview time was 33 min (18–62). Table 1 indicates the characteristics of the participants, their professional background and the healthcare service division they represented.
Characteristics of the staff interviewed (n=15)
Process map of FFT feedback as a real-time initiative
With the interview data we created a process map that demonstrates the complex nature of stakeholder interactions with FFT reports as it cascades from collection to dissemination (impact) (figure 1). The diversity of stakeholders involved included information governance team, data outsourcing team, patient experience team, divisional managers and frontline staff.
This demonstrates the complex flow of Friends and Family Test as a real-time feedback initiative and the stakeholders involved as the feedback cascades down. The division comprises of surgery and cancer, medicine and integrated care, women’s and children and clinical support and private patients. A&E, Accident and Emergency.
We provide a descriptive summary of the process map of FFT data as depicted in figure 1. Data from all four care settings is collated and sent to central business intelligence department where due diligence is carried by the information governance team. A mandatory report is then sent to NHS England at monthly intervals. This report is not sent out to frontline staff. Once the feedback is released by the information governance team, the raw data are then sent to an external provider who assists in analysis and building visualisations and reports. The reports are presented in a traffic light format based on the response to the FFT question, that is, FFT score. In addition, the number of responses and free-text data are available to view. No further analysis on the free-text data are conducted. Any amber and red reports are flagged by the patient experience team which triggers an action plan by the appropriate ward manager. The reports in theory are accessible to all staff but access is not mandatory. At monthly intervals each divisional lead gathers the data from the FFT reports to create another entirely separate report for Trust board meetings. The patient experience team are tasked to assist with these regular reports, and also ensure that any feedback is acted on. This highlights the unstructured route of FFT feedback, how ownership of the FFT reports changes at each time point and the delay in providing FFT reports to the frontline staff. Despite the flow not being streamlined, all four care settings follow the same sequence of steps from collection to dissemination (impact).
Identified themes
During the interviews, the participants were very clear about what they perceive as factors that promote and limit the effective use of FFT as a near real-time feedback initiative. They were generally able to elaborate clearly and consciously on the causes and effects of these factors. Subsequently, many separate barriers and facilitators were categorised in four main themes (figure 2) as described below.
A summary of key factors that promoted and limit the effectiveness of FFT as a near real-time feedback initiative based on four key themes. FFT, Friends and Family Test.
Capacity and resource
There was a lack of capacity and resource within the organisation to enable regular and consistent FFT collection. Specifically, frontline staff described having very limited time available to engage with the FFT collection, as they were usually too busy to remember to collect FFT feedback. To address this, the patient experience team introduced designated staff or ‘champions’ and volunteers. However this was done on an ad hoc basis and prioritised following a mandate by the medical directorate when response rates dropped below the national average. There were also concerns about the use of digital tools used to collect FFT data due to the lack of availability of devices and issues with connectivity. A portion of FFT surveys were therefore being completed on paper and transferred onto a digital format. Participants felt that improving the digital infrastructure could subsequently enable redeployment of staff to improvement projects rather than spending time manually uploading FFT data. One participant said, ‘It is not good use of their time, we should take that resource and get them [staff] out on the wards doing some improvement work’.
Usage
Staff highlighted several factors which had an impact on the use of FFT reports. First, FFT data were held in various formats, that is, unprocessed, formatted for NHS England, summarised for divisional reporting, analysed through outsourcing and presented via a visualisation tool. The tool was only accessible with an individual login and once training had been completed. The number of lead nurses who had access and training on the visualisation tool was higher compared with frontline (junior) staff. We found that there was a lack of access to the visualisation tool for frontline staff and this was exacerbated by the ad hoc training. There were delays in creating FFT reports for frontline staff despite the near real-time capability, due to the number of stakeholders involved in handling the FFT data as depicted in figure 1. Additionally, participants felt that the use of FFT reports fell short because ‘there are no real sanctions for FFT and patient experience’, and ‘some of them [staff] have so much to do’ therefore, ‘it’s something that gets forgotten’. Therefore, managers had to take initiative to implement the FFT results, but this was not a priority as they spent their time preparing reports for divisional and trust wide meetings, where ‘FFT data were very rarely looked at’. Furthermore, the sheer volume of free-text data received at the end of the month was not amenable to manual thematic analysis. We also found that the overall FFT score did not change much per month, and ‘not subtle enough to pick on variations’. Only services or wards highlighted as ‘red’ (traffic light rating scale) on their FFT question score were followed-up. One participant expressed that the main FFT question should change, ‘maybe it’s not the right question, but it’s the question we’ve got and we have to deal with’.
Interoperability
Despite the best attempt to ensure FFT data were interoperable, FFT data were loosely triangulated with other quality and safety metrics, that is, presented as a report at the executive quality committee. The biggest component of the report is the safety aspect, ‘we don’t spend a lot of time on the FFT section as the month on month variation in the FFT score is negligible’. The outsourced visualisation reporting tool lacked satisfactory user experience and quoted as being ‘clumsy’. One participant explained, ‘if the FFT reports were presented in such a way that services could learn from each other, we can pre-empt problems in other areas’. Of note, other patient feedback reports were reviewed in more detail such as the Adult Inpatient Survey14 as the results are presented in a way that is understandable. FFT reports were not linked to other sources of patient feedback held within the organisation.
However, ward managers printed FFT reports by offering a static dashboard, summarising progress and areas for improvement at-a-glance. These types of reports were also included in ward accreditation programmes and used as part of revalidation.
Impact
In the action planning process, staff found free-text comments written by patients more meaningful compared with the FFT score. Seeing patients’ own comments brought the experiences to life for frontline staff and added a ‘sense of urgency’ to address them in improvement efforts. However, due to the sheer volume there was a desperate need to consider automation in the form of text classification and sentiment analysis in the hope that the insights from free text are not neglected and continue to have impact. Staff called for more flexibility in the timing of FFT collection as it was conducted mostly on discharge. One participant mentioned that ‘patient experience shouldn’t start when they are being discharged, it should start when they are being admitted’. Evaluations made from FFT reports were primarily used for internal benchmarking and comparisons with other Trusts, but did not result in local improvements. When the nursing directorate intervened, their experience and expertise allowed for improvements to be made locally driven by frontline staff. It is important to note that most frontline staff lacked formal QI training. Once trained in the outsourced visualisation tool, frontline staff were independently able to identify areas that required attention by understanding trends and using word clouds generated from the free-text comments.
Discussion
We highlighted the factors that promote and limit the effectiveness of FFT as a near real-time feedback initiative and demonstrated in detail the complexity involved in processing FFT data within the organisation (figure 2). The process map (figure 1) highlighted the number of various stakeholders involved resulting in lack of ownership and the inconsistency in FFT reports hindering FFT-driven improvement efforts. Data from the qualitative interviews revealed several concerns highlighted by staff based on four themes; capacity and resource, usage, interoperability and impact of FFT. We discuss why this can impede effective use of FFT as a near real-time feedback initiative and investigate the literature for strategies that healthcare providers could consider deploying to increase staff engagement and thereby improve patient experience.
Recent studies have emphasised the preconditions for highly engaged staff, which include meaningfulness of work,15 sustainable workload,15 accountability,16 opportunities for learning and development,16 17 strong leadership,16–18 involvement in decision-making17 and relatively flat hierarchies.17 There is evidence19 that suggests that staff struggle to translate data into action: ‘perceived barriers included a lack of knowledge of effective interventions, and limited time and resources’. Frontline staff are focused on their current patients. This is in direct contrast to the focus of the hospital management who produced FFT reports based on the experience of previous patients who were cared for weeks before. At board level, the focus was on monitoring of FFT response rate and no accountability for lack of FFT-driven improvement. In fact, frontline nursing staff, of which majority lacked access to the visualisation software, were at the bottom of the hierarchy for viewing FFT reports. This disconnect offers some explanation for the lack of engagement of staff with FFT and was one of the main reasons that staff were ambivalent to FFT as a real-time feedback initiative. Sheard et al20 identified that there is a lack of staff ownership of patient feedback and this most often pertains to staff flux or demoralisation with action plans failing to be initiated. They demonstrated that when staff sought to make improvements from patient feedback, changes to the structures or processes of the individual ward on which they worked, this often led to success.
A systematic review21 noted that despite the FFT policy mandate, the particular expertise needed to be able to conduct effective and meaningful data collection, analysis and interpretation appears not to have been provided to any great extent. This can be seen from clinician and staff reports that, while often they believe patient experience reports are important in their organisations, they also state that they have neither the time nor the expertise to use these data to any great effect.21 This barrier has been highlighted in our study as well as previous studies,4 19 22 23 calling for a need for staff training in data analysis and statistics to facilitate full understanding and use of results particularly if data are outsourced. Our findings revealed that frontline staff that were critical in championing and implementing improvement work when given the right training and opportunity. For, for example, staff recommended making the FFT reports printable by offering a static dashboard, summarising progress and areas for improvement at-a-glance and in near real-time. However, any improvement programme introduced in other services could not be shared widely, resulting in repetition and inefficiencies.
There should be an organisational emphasis where patient experience data collected has the ability to be meaningfully used by frontline staff. Sheard et al24 made recommendations to facilitate healthcare organisations to change the way patient feedback is used, by tackling both macro-level structural/organisational factors and micro-level factors surrounding how individuals interact with patient experience data. An organisational strategic focus that prioritises utilisation over collection, and ensuring data are relayed to staff by patient experience teams in an accessible, straightforward and engaging manner, coupled with staff training that encompasses both quantitative and qualitative analytical techniques and QI methodologies.
Transforming culture by embracing frontline staff
Organisations need to improve their understanding of how frontline staff can use FFT data for QI; what motivates them to get involved in improvement; what helps or hinders; and what can be done to make FFT reports more convincing, credible and practically useful. Senior leadership should give staff a voice and play an active role in supporting staff in addressing system problems and delivering change through genuine sharing of responsibility.25
Ipsos-MORI raise three critical issues for real-time data driven service improvement, all of which have a bearing on how effective real-time data are for key stakeholders, and all of which would benefit from further research: practicality of implementation, quality of data collected and organisations ability to translate data into action.26 They also highlight key issues for healthcare services to focus on: ensuring the patient experience data are as granular and real time as possible, combining this with qualitative and other data sources, producing data reports that are accessible and focus managers attention on areas of improvement, implementing real-time data as an organisational rather than technical exercise and actively bringing staff on board to champion and use the data to improve patient experience. Explaining the benefits of FFT to staff and dealing openly with issues of scepticism and resistance to change will increase the likelihood of success.6 7 Indovina et al27 showed that real-time experience reporting, coupled with staff education and coaching, improved satisfaction of inpatients. Similar findings were reported,28 augmenting the need to cultivate a culture that promotes staff communication and engagement. Furthermore, evidence suggests there is a relationship between staff well-being and (a) staff-reported patient care performance and (b) patient-reported patient experience. So, where patient experience is low, so too is staff well-being and vice versa.29
Improving reporting by addressing clinical analytics
The need for advanced analytical capability in the NHS is growing and faces the same cost pressures that impact all other realms of service development and QI. Recent literature5 reveals some of the barriers and facilitators associated with technology driven real-time data collection. The main barriers were related to familiarity, connectivity and positioning, which was similar to our findings. Another barrier noted in our study was the lack of awareness of end users’ individual values and needs. As a result these technologies either fail to be used at all or adaptations are made to fit them into pre-existing workflows that were not considered a priori. Therefore, assessing user insights and acceptance during the development and testing phases, and delivering technical support and versatility to data collection approach5 is likely to improve the likelihood of meaningful implementation and uptake.
Analysis of free-text comments was challenging due to time and resource constraints, and prone to delays which resulted in outdated information. A semi-automated process to rapidly identify and categorise comments from free-text responses may overcome some of the barriers encountered with manual extraction and long processing times. Patient experience themes and sentiment can be extracted from free-text comments, highlighting areas of concerns and providing the context and details required for staff to rapidly learn and act on patient feedback,28 thereby addressing the FFT redevelopment programme requirements. By investing resources in building the capacity to innovate and develop clinical analytics within the organisation will not only improve services but also build a foundation of technical knowledge in the organisation and create a culture that promotes innovation. Sheard et al24 state that ‘if patient experience feedback is to be valued, then it should stop being viewed as the poor relation to patient safety and finance while simultaneously—and concertedly—moved outside the remit of being badged as a problem for corporate and shop floor nursing to solve’.
Limitations
Participants in the study were directly involved in FFT reporting, however, a broader sample of staff with little-to-no FFT involvement such as healthcare assistants, administrative staff and student nurses would have been valuable. They are in fact the coal face in delivering the patient experience, so they need to be represented in order to understand how FFT can be used as a real-time feedback initiative. Moreover, this was a single site study with a small sample size, which may not be representative of the UK as a whole.
Conclusion
The use of staff ‘FFT champions’, supplementary free text, visualisation tool and enhancing ward accreditation using FFT reports promoted the use of FFT in some care settings. However, the unstructured flow of FFT data from collection, analysis to dissemination failed to align with real-time reporting aspirations and timely interventions. This was exacerbated by lack of ownership and accountability, training and access to FFT reports, resulting in staff ambivalence. The results also demonstrate that there is too much FFT free text for meaningful analysis, and the output is limited to the provision of sufficient capacity and resource to analyse the data, without consideration of other options, such as implementing text analytics on free-text FFT data and developing versatile and targeted point of care FFT collection.
Data availability statement
Data sharing not applicable as no data sets generated and/or analysed for this study. Recruitment began before 1 January 2019, therefore no data sharing plan is included.
Ethics statements
Patient consent for publication
Ethics approval
This study received ethical approval from North East—Tyne and Wear South Research Ethics Committee, 17/NE/0306.
Acknowledgments
We thank the healthcare staff and key stakeholders for their involvement in this study. This work is supported by the National Institute for Health Research (NIHR) Imperial Patient Safety Translation Research Centre. Infrastructure support was provided by the NIHR Imperial Biomedical Research Centre.
References
Footnotes
Twitter @MusKhanbhai, @kelseyflott
Contributors Conception and design of the work: MK, KF and EM. Data collection: MK and KF. Data analysis and interpretation: MK, KF, SH-W and EM. Critical revision of drafts for important intellectual content: MK, KF, DM, SH-W, RK, AD and EM. Final approval of the version to be published: MK, KF, DM, SH-W, RK, AD and EM. MK acting as guarantor.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review Not commissioned; externally peer reviewed.