Objectives We aimed to evaluate the impact of patient and public involvement (PPI) at each stage of the research cycle in a dementia research programme.
Design We used monitoring forms to record the impact of the research programme’s PPI at different stages of research and qualitative interviews with all participants to evaluate the impact of PPI.
Setting We evaluated Research User Groups (RUGs—older people with dementia and care partners) which were established to provide PPI support for the research programme in multiple European sites.
Participants We purposively sampled RUG members (n=34) and researchers (n=13) who had participated in PPI activities. Inclusion criteria for the study were: (a) RUG members who had participated in the research awareness training and in PPI activities and had the capacity to consent; (b) researchers who involved RUGs in their work.
Results Impact on the research: changes to the study conduct were made as a result of the feedback from RUGs. These included prioritisation of clinical recommendations, the wording of study information and recruitment materials, the content and layout of the user interface for a computerised memory test, interpretation of intervention results and advice on dissemination avenues. Impact on RUG members: they reported that involvement had given them a sense of purpose and satisfaction. Their perception of health research changed from being an exclusive activity to one, which lay people, could have meaningful involvement. Impact on researchers: PPI was a new way of working and interacting with PPI members had given them insight into the impact of their work on people living with dementia.
Conclusions PPI can have a substantial impact on dementia research and the people involved in the research. To justify the time and expense of PPI, the advantageous practical impacts of PPI should be systematically recorded and consistently reported.
- qualitative research
- public health
This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
Patient and public involvement (PPI) monitoring forms allowed systematic monitoring and evidencing demonstrable impact.
Research User Groups (RUGs) had limited control over what level or type of PPI activities they were tasked with.
We interviewed researchers involved in a project that contained a strong element of PPI.
Interviews conducted by PPI coordinators may have biassed RUG participant responses.
The RUG could have included people from more diverse backgrounds.
Recognition that considering the views of end-users in health research improves the relevance and quality of health research has raised the profile of patient and public involvement (PPI).1–13 PPI reflects the growing democratisation of health research and the importance of research impact. PPI ensures that research questions, conduct and outcomes are relevant and meaningful to patients and the public.11 14–18
In countries such as the UK, PPI is required by many funders,19 as well as an ethical obligation.20 Researchers are encouraged to involve patients and the public at all stages of the research cycle.21 However, substantial challenges with the conception and implementation of PPI are still evident.22–25 Although many health research studies acknowledge the beneficial impacts of PPI,23 various reviews highlight that the reporting of PPI is often of poor quality and evidence of impact is weak.23–28 Thus, there is a clear need to demonstrate the impact of PPI on the research process more rigorously. Detailing the contribution made by PPI enables learning how to do PPI effectively and how to identify key indicators of impact.29 30 In the context of the international multi-site dementia research programme, SENSE-Cog (www.sense-cog.eu), our aims were to: (a) identify the impact of PPI at each stage of the research cycle31; (b) provide a model for recording the impacts of PPI; and (c) explore the experiences and perceptions of people with dementia, their care partners and dementia researchers of PPI in the research programme.
Context of the SENSE-Cog PPI work
SENSE-Cog32 is a five‐year (2016–2020) European research programme investigating the combined impact of ageing-related hearing and vision impairment on mental well-being. It involves the exploration of longitudinal datasets to understand the interplay of hearing, vision and cognitive impairment,33–35 the validation of assessments adapted for people with dementia and hearing and/or vision impairment,36 and the evaluation of a home-based hearing and vision support intervention in improving quality of life for people with dementia.37–39 As part of this programme, a dedicated work package on PPI40 was embedded as a cross cutting theme (figure 1) to involve older people with dementia and care partners. This involved establishing Research User Groups (RUGs) in Manchester (UK), Nice (France), Nicosia (Cyprus) and Athens (Greece) for PPI at each stage of the SENSE-Cog programme. Each RUG consisted of seven to nine RUG members, who were people with dementia and their care partners. RUG members were provided with research awareness training (RAT; outlined elsewhere39) to equip them with the skills and knowledge to guide their involvement in the research and to enable them to contribute to the research programme. Each RUG site was managed and supported by a local PPI coordinator in each of the four sites.40 We reported all features of the PPI process according to the GRIPP2 short form recommendations30 (see online supplemental file 1) in this paper, and the full protocol involving the RUGs is published elsewhere.40
The published protocol40 outlining the PPI activities pertaining to the entire SENSE-Cog programme, was established by the RUGs at the outset of the research programme. Additional PPI activities were added at different stages on an ‘as needed’ basis by local RUGs and to include local contextualised content for the PPI activities. These activities were coordinated both centrally by the lead PPI coordinator based in Manchester, as well as locally by the site-specific coordinators. For the basic programme-wide PPI activities, the central PPI coordination team, RUG and cross-site researchers worked together to agree on the key objectives, formulate the PPI questions, and develop and revise the materials for the PPI activities. The central PPI coordinator supported the local RUGs and their PPI coordinators to implement and shape the activities in a manner that was appropriate and relevant to their local needs.
Patient and public involvement
A member of the Manchester RUG provided PPI on this paper to ensure it was informed by the perspectives of a person with experience of involvement in health research. PPI input was obtained by a face-to-face consultation. The RUG member first reviewed the structure of the paper, including a rationale, the introduction, background and methods sections. The RUG member suggested the use of diagrams to illustrate the main findings and contributed ideas on how to categorise the findings. We acted on this by using a table to illustrate the key PPI activities in this paper. Second, the RUG member reviewed and commented on the draft paper. The RUG member supported the findings and approved the final paper. PPI in this paper provided the perspective of a person with experience of involvement in health research. The RUG member requested not to be named on this paper.
Monitoring and evaluating the impact of the SENSE-Cog PPI programme
We used two methods to monitor and evaluate the PPI work package within SENSE-Cog programme, (a) PPI monitoring forms (see online supplemental file 2), throughout the research programme; and (b) qualitative interviews conducted towards the end of the programme.
For all PPI activities, local PPI coordinators kept monitoring logs of the feedback from RUG members at each site (figure 2). Types of feedback sent back to the research teams included: the wording of an adapted rating scale evaluating support care needs; the wording of participant information sheets; content and presentation of the intervention manual; aspects of the intervention itself (a multi-module intervention involving hearing and vision rehabilitation supported by a therapist); adaptation of a hearing aid guide for person with dementia (PwD) for the intervention manual; confirmed the researchers’ analysis and refined the coding framework (table 1). The completed PPI monitoring logs from the sites were sent to the central PPI coordinator in Manchester for recording purposes, and were then forwarded to the relevant work package research team for comments who recorded the actions taken to address the feedback on the monitoring form. All RUGs were given updates on the progress of the overall SENSE-Cog programme every 18 months to inform them of new developments and outputs and for the RUGs to witness how their feedback had shaped different aspects of the work. The PPI monitoring forms were examined by the central PPI coordinator and impacts and justifications for no action following RUG recommendations extracted and itemised in a table format (table 1).
Participants for the qualitative substudy
Participants for the impact evaluation were purposively sampled from two of the PPI working groups:
(a) RUG members (n=34) who had participated in the RAT and in PPI activities and had the capacity to consent. Establishing whether potential participants with cognitive impairment can give informed consent to was assessed by staff trained, further details on the assessment of capacity are available elsewhere40; (b) researchers who involved RUGs in their work.
We aimed for a sample size of six RUG participants in each RUG site and six researchers across all sites to ensure representation and was determined by data saturation, the point when no new information was obtained. We examined the coding framework data, the characteristics of the participant quotes and the findings, and decided to end the at the point when no new information emerged but included three more interviews to check that data saturation has been reached.41 42 Thirty-four RUG members were invited for interviews. Twenty-three RUG members responded and participated in the interviews. Eighteen researchers were approached for interviews, thirteen researchers responded and participated in the interviews. A total of 36 interviews were conducted (table 2).
Procedure for qualitative substudy
We wrote two interview schedules, one for RUG participants (see online supplemental file 3) and one for researchers (see online supplemental file 4). The interview schedule for RUG participants was designed to understand their experience as a RUG member, participating in PPI activities and the perceived impact of their input on the research process. The interview schedule for researcher participants was developed to explore the researcher’s view of PPI, their experience of working with RUGs and their perceived impact of PPI on the research.
Interviews with RUG participants, lasting about an hour each, were conducted between October 2018 and October 2019 by local PPI coordinators in each site. The interviews were conducted separately or in pairs as preferred by the PwD and care partners. Interviews with researcher participants were conducted between January 2019 and September 2019 by the central PPI coordinator (JM) in English, either by phone of face-to-face.
Interviews with English speaking participants were audio-recorded and then transcribed verbatim. Interviews in Greek and French were transcribed verbatim in the local language and translated by the local PPI coordinators. Personal information was removed from transcripts.
Qualitative data analysis
Researcher and RUG data were pooled across sites and analysed. We undertook thematic analysis43 to draw out the main characteristics of RUG members and researchers experiences. JM undertook the primary analysis, reading transcripts numerous times to develop open codes. Various levels of open coding took place from line-by-line coding. JM and SP independently examined the data to identify themes and reach consensus on interpretation. Themes were then assigned codes according to the Framework method44 using NVivo software V.11 (QSR International, Doncaster, Australia). This method allows the data to be structured into a chart with columns and rows aiding the identification and grouping of text into relevant parts.
The coding framework for the RUG participants was emailed to local site PPI coordinators who were asked to provide input on a draft of the coding framework. Any additional themes identified by coordinators were added to the list of themes. Datasets were then reanalysed according to the final coding framework. We stopped data collection when no new information was being generated from analysis.41
According to the Mental Capacity Act (2005),45 people should be assumed to have capacity unless otherwise demonstrated. All researchers involved in this research completed training in assessing capacity in research. Participants were provided with easy access user-friendly versions of the participant information sheet and consent form and enough time to think about whether to participate in the study. Consent was assessed on an on-going basis. Participants provided informed written consent prior to the interview. Further information on consenting procedure is available elsewhere.40
Impact of PPI on research
PPI resulted in specific changes at all stages of the research from identifying and prioritising the research agenda to dissemination (table 1). In some cases, changes recommended by PPI were not implemented. Reasons that changes could not be implemented included scientific or regulatory requirements (eg, not being able to make changes to validated psychometric tests, not being able to supply clinical diagnoses on the basis of tests given for a research study, not being able to shorten the participant information sheet due to ethical governance requirements) or limited financial or personnel resources (eg, being able to produce a study newsletter only four times a year rather than more frequently).
RUG members’ perspectives
Four themes were identified: facilitators to contributing, meaningful PPI, personal gains and perceived impact of involvement on research. In the following quotes, the site is reported first (ie, ‘Manchester’, ‘Nice’, ‘Nicosia’ or ‘Athens’). ‘RUG (number)’ refers to the individual RUG member being quoted. ‘PwD’ and ‘care partner’ refers to a person with dementia or a care partner for someone with dementia, respectively.
Facilitators to contributing in PPI
Participants discussed the benefits of group work. Group work was viewed as peer support for participants, which was seen as a valuable way to help understand each other’s circumstances and needs to support each other. Participants reported that they had gained new knowledge and discussed the benefits of receiving the RAT for the role, highlighting that the structuring of the training alongside the involvement tasks as factors that helped them to contribute to research. RUG participants also reported that the support provided by the local PPI coordinators made them feel that they were making a valuable contribution. A participant commented on the importance of the ‘human relationship’ (Athens, RUG4, care partner) with the coordinator.
Interaction with researchers and having researchers listen to RUG member’s opinions to was reported as an important aspect. Receiving feedback was also highlighted as an important factor in letting RUG members know that their contribution was having a tangible impact:
I feel that our communication was quite satisfactory and apart from that we had feedback. We said something and they ‘listened’ to it and we move forward, they tell you a little more, something like that …. Yes, for sure. Otherwise, I wouldn’t be here today. It’s a good feeling, to be heard. You gain this feeling that you are not alone and that there are people that care for you, and that what you say can go further (Athens, RUG5, care partner)
Personal benefits of PPI
Face-to-face group meetings provided RUG members the opportunity to share issues with others in similar position. RUG members reported that the experience personally benefited them in terms of gaining new knowledge, interacting with others, giving advice and feeling that they were making a worthwhile contribution. RUG participants felt their involvement exceeded their expectations, due to continuous involvement and training in research awareness:
Throughout the meetings we felt that what we learned is very important for the lives of individuals. I felt the role was more than was described (role of RUG member), because we participated (learned) in many task trying to find solutions that we have to resolve. (Athens RUG6, PwD)
Perceived impact of involvement on research
RUG members talked about their involvement in dissemination activities, particularly in countries other than the UK. RUG members talked about how providing their perspective provided researchers with a better understanding of the impact of dementia that informed research decisions. RUG members emphasised their contribution in making written materials more accessible for participants and suggesting changes in the interventions and tools developed by the research team. RUG members reported that incremental changes informed by PPI could make a difference to practical outcomes:
Well it’s just one factor in the complicated chain of conditions that lead to the outcome. So it’s not going to have any direct impact on any individual people, but every time a member of the group makes a contribution, it’s adding a little bit more information to the whole package of information that’s going to improve the lives of the end users in the long term. (Manchester, RUG1, ex-care partner)
Four themes were identified: limited experience of PPI, acceptance and importance of PPI, beneficial impacts of involvement and challenges. These are presented in table 3, in the quotes, the site is reported first (ie, ‘Manchester’, ‘Nice’, ‘Nicosia’ or ‘Athens’) then the researcher number. ‘PM’ refers to project managers.
PPI is proposed as an important means of enhancing the relevance and impact of research.21 46 However, while the descriptive literature on PPI is growing, there is still little evidence for how PPI actually impacts on research and there has been no clear methodology for recording impact.22–24 27 Here, we reported a method for capturing the impact of PPI in a multi-centre European research project, outlining the impact of PPI at each stage of the research process. We also provided qualitative perspectives of RUG members and researchers concerning their views on the impact of involving people with dementia and care partners in research.
A number of key changes were made to different aspects of the research programme at various points following PPI advice. PPI contributed substantive input to intervention formation, design, recruitment, interpretation of results and dissemination of the research. However, some PPI feedback could not be acted on for the reasons outside the research team’s control due to resource restrictions, lack of time or compromising validated psychometric measures. These restrictions on implementing PPI advice are similar to restrictions reported elsewhere.15 26 Without ongoing advice from the RUGs to the research teams, researchers may not have made appropriate changes on many aspects of the research design to take account of the experience of living with dementia. In addition to concrete changes in the research following RUG input, RUG members reported various personal benefits in contributing to research, including feeling that they were making a useful contribution to research. Researchers reported that the RUG members input gave them an important personal perspective on the relevance and impact of their research for people living with dementia.
RUG interview data were collected by the coordinators who were themselves SENSE-Cog project researchers, rather than by an independent researcher. Data collection by SENSE-Cog project researchers may have resulted in responses being overly positive.43 44
Although research involvement opportunities for RUGs were advertised via contact with local charities, social organisations and support groups, we received few queries about involvement and no queries from anyone from minority ethnic communities. The RUG groups may not have reflected the diversity within the general community, and this may have limited the perspectives that RUGs were able to offer and of the impact of PPI in this project. Using local community leaders or organisations to access more diverse communities could have increased recruitment from minority groups.47–49 Future research could explore views of older adults from minority ethnic background and diverse communities in PPI.
For our qualitative work, we only sought the opinions of researchers who were involved in a project that contained a strong element of PPI. The researchers in this project may have been more favourably disposed to PPI than European researchers in general. Another limitation is that RUGs were not involved in the initial conception of the study at the grant application stage. Ideally, PPI should occur from the point of conception of a research programme and around setting research agendas. Due to time and resource constraints, and possible extra burden we did not involve RUG members in data collection although they were involved in the development of the interview questions.
It is appropriate to highlight the issue of the power imbalance13 between researchers and RUGs. The RUGs had limited control over what level or type of PPI activities they were tasked with in the research programme. In addition, RUG members had limited influence on what feedback were acted on, as they were dependent on researchers listening to their input and acting on their feedback. We provided RUG members with notes of the PPI activities to acknowledge their contribution and provided written responses from the researchers on what actions were taken because of the RUGs contributions.
PPI in research has potential benefits, but it does require substantial resources that could otherwise be spent elsewhere. Although this paper highlighted the impact of PPI on the research and documented positive experiences of the RUG members and researchers, we cannot be sure of the impact of these changes on research outcomes (eg, if fewer participants would have been recruited if recruitment materials had not been altered following PPI input). An understanding of the benefit and cost utility of PPI would require a controlled evaluation (eg, recruitment with vs without PPI input on recruitment materials). There is some evidence from controlled evaluations that PPI does have tangible impact: a systematic review on the impact of PPI on enrolment and retention in clinical trials reported that PPI considerably improved the likelihoods of a patient joining a trial compared with recruitment strategies that did not include PPI.18 However, because recruitment strategies that had used PPI also included non-PPI components, it was difficult to establish the contribution of PPI versus other factors.
The overall strength of this study is the demonstrable impacts of PPI on research and the benefits reported for both researchers and RUGs interactions highlight gains in experiential knowledge and learning.
This study provides a methodology for recording the impact of PPI in multi-centre health research using systematic ongoing monitoring and feedback throughout the research programme, followed by detailed qualitative exploration of PPI contributors’ and researchers’ views. To justify the time and expense of PPI, the impacts and benefits of PPI should be systematically recorded and reported for all health research that makes use of PPI.
Further information about SENSE-Cog is available in online (www.sense-cog.eu).
The authors would like to acknowledge the people working with us as part of Research User Groups and SENSE-Cog colleagues coordinating the Research User Groups in the UK, France, Cyprus and Greece.
Contributors IL, PD, KL, BS contributed to the conception of the study. JM, PD and SP contributed to the design of the study. JM wrote the first draft and was involved in the editing of the manuscript, which was critically reviewed by IL, KL, PD and SP and approved the final manuscript.
Funding This protocol paper is part of Work Package 5 of the SENSE-Cog project, which has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No. 668648. Piers Dawes, Suzanne Parsons and Bella Starling are supported by the National Institute for Health Research Manchester Biomedical Research Centre.
Disclaimer The funding body has no part in study design, data collection, data analysis, data interpretation or manuscript writing.
Competing interests None declared.
Patient consent for publication Not required.
Ethics approval Ethical Approval was approved by the University of Manchester’s Research Ethics Committee UREC: 2017-0627-2142. Additional ethical approvals were sought and obtained for each study site; in Nicosia approved by the Cyprus National Bioethics Committee, in Nice by ‘Comité de Protection des personnes Sud Est I’, and in Athens by Ethical Committee of Health Sciences and Scientific Committee of the Eginition Hospital of the National and Kapodistrian University of Athens.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Interview data from this study are personally identifiable for RUG members and researchers, and are therefore not available for data sharing.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.