Objectives This mixed-method process evaluation underpinned by normalisation process theory aims to measure fidelity to the intervention, understand the social and structural context in which the intervention is delivered and identify barriers and facilitators to intervention implementation.
Setting RETurn to work After stroKE (RETAKE) is a multicentre individual patient randomised controlled trial to determine whether Early Stroke Specialist Vocational Rehabilitation (ESSVR) plus usual care is a clinically and cost-effective therapy to facilitate return to work after stroke, compared with usual care alone. This protocol paper describes the embedded process evaluation.
Participants and outcome measures Intervention training for therapists will be observed and use of remote mentor support reviewed through documentary analysis. Fidelity will be assessed through participant questionnaires and analysis of therapy records, examining frequency, duration and content of ESSVR sessions. To understand the influence of social and structural contexts, the process evaluation will explore therapists’ attitudes towards evidence-based practice, competency to deliver the intervention and evaluate potential sources of contamination. Longitudinal case studies incorporating non-participant observations will be conducted with a proportion of intervention and usual care participants. Semistructured interviews with stroke survivors, carers, occupational therapists, mentors, service managers and employers will explore their experiences as RETAKE participants. Analysis of qualitative data will draw on thematic and framework approaches. Quantitative data analysis will include regression models and descriptive statistics. Qualitative and quantitative data will be independently analysed by process evaluation and Clinical Trials Research Unit teams, respectively. Linked data, for example, fidelity and describing usual care will be synthesised by comparing and integrating quantitative descriptive data with the qualitative findings.
Ethics and dissemination Approval obtained through the East Midlands—Nottingham 2 Research Ethics Committee (Ref: 18/EM/0019) and the National Health ServiceResearch Authority. Dissemination via journal publications, stroke conferences, social media and meetings with national Stroke clinical leads.
Trial registration number ISRCTN12464275.
- clinical trials
- stroke medicine
This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
Strengths and limitations of this study
A mixed-methods theory-driven process evaluation will generate detailed findings to assist in interpreting the results of a pragmatic, multicentre individual patient randomised controlled trial of a complex vocational rehabilitation intervention, which crosses the work/health divide.
This is one of the most comprehensive multisite, multicomponent, multistakeholder perspective process evaluations embedded in a stroke rehabilitation trial, involving detailed assessment of implementation fidelity, therapist competency to deliver the trial intervention, contamination logging and exploration of social and structural influences on intervention provision in poststroke rehabilitation services.
Longitudinal case studies with intervention and usual care will capture participant experiences of providing and experiencing the intervention including those of employers.
The COVID-19 pandemic limited researcher access to direct observation of face-to-face intervention delivery and employer interactions with stroke survivors in each site. Integration of interview data from different participant sources, including stroke survivors and carers, occupational therapists and employers with available observational data are planned to address this limitation.
Approximately 100 000 people in the UK suffer from a stroke every year,1 and around one in four are of working age.2 Returning to work after a stroke is a major goal for stroke survivors, contributing to social identity, emotional and financial well-being and conferring a sense of purpose and has benefits for the individual, the individual’s family and the economy.3 Despite this, only half of working age stroke survivors make a successful return to meaningful work, and they are two to three times more likely to be unemployed 8 years after their stroke than the general population.1 Although impairments in the stroke survivor’s physical, cognitive and communication abilities can affect this,4 5 social and environmental factors such as personal and employer beliefs and attitudes, job type and organisation size and the benefits system also play an important part.6 7
Vocational rehabilitation (VR) is defined as whatever helps someone with a health problem to return to, or remain in, work and includes both work and work-related education.8 It involves helping people find work, helping those who are in work but having difficulty as well as supporting career progression in spite of illness or disability. The primary aim is to optimise work participation.9 Existing research suggests that VR may help stroke survivors return to their previous job or find new work,10 11 however, trials to date involve small samples in non-UK settings.
RETurn to work After stroKE (RETAKE) is a multicentre individual patient randomised controlled trial, which aims to determine the clinical and cost-effectiveness of an Early Stroke Specialist Vocational Rehabilitation (ESSVR) intervention in addition to usual National Health Service (NHS) rehabilitation on stroke survivors’ return to work (RTW) at 12 months postrandomisation, compared with NHS rehabilitation alone.12 Acceptability and utility were assessed in a feasibility trial.13 ESSVR combines conventional occupational therapy (OT) with case coordination. The intervention commences within 2 weeks of randomisation and lasts up to 12 months postrandomisation. It is intended for delivery in the community as often as required by individuals, as determined by a stroke specialist OT with additional VR training. ESSVR includes the following: (a) assessing stroke impact on the person and their job, (b) educating individuals, employers and families about stroke impact on work and strategies to lessen impact (eg, memory aids, fatigue management), (c) work preparation, including opportunities to practice work skills and (d) liaison with employers to plan and monitor a phased RTW (see online supplemental appendix 1). The target number of participants for the trial is 760 participants (420 ESSVR and 340 usual care) from 20 UK hospitals and linked early supported discharge/community services. The RETAKE trial and embedded process evaluation commenced in June 2018 and will complete in March 2022. This period includes a funder approved extension of 7 months necessitated by an unplanned pause in recruitment during the COVID-19 pandemic.
Failure to implement evidence-based stroke rehabilitation interventions in clinical practice may result in unnecessary suffering and disability.14 15 Trialists must consider future implementation in the real world when designing clinical trials, paying particular attention to the context for intervention delivery and factors likely to influence its uptake and use.16 This is especially true for trials of complex rehabilitation interventions, which comprise multiple interacting components and target a number of different organisational levels, making them particularly challenging to implement. An embedded process evaluation provides for an in-depth exploration of factors influencing the implementation of complex interventions.
The Medical Research Council (MRC) argue for a systematic approach to designing and conducting process evaluations, drawing on clear descriptions of intervention theory and the identification of key process questions.17 Mixed methods approaches to process evaluation are increasingly common and consistent with the MRC framework’s emphasis on exploring and understanding the important relationship between context, mechanisms and implementation. Theory-driven process evaluations are recommended alongside complex intervention trials to measure what is delivered. These measurements include fidelity (whether the intervention was delivered as intended), dose (the quantity of intervention implemented) and ‘reach’ of interventions to understand how the intended audience interacts with the intervention.17 Fidelity data are necessary to interpret intervention outcomes, but despite an extensive literature supporting its importance, fidelity is commonly under-reported in studies of complex rehabilitation interventions. While most trials of VR have not raised particular concerns about fidelity, ESSVR in the RETAKE trial is an example of a particularly complex intervention that crosses organisational boundaries, involves interactions between multiple stakeholders, is highly individually tailored and requires behavioural change by the patient, their family and employer. Therefore, in the process evaluation for the RETAKE trial, we have included specific methods to measure fidelity. Alongside a focus on fidelity, in-depth qualitative exploration of participants’ experiences of an intervention and of the social and structural context, in which an intervention is provided, are essential elements of process evaluation of complex interventions. This ensures any adaptations made to tailor intervention to the individual and/or differing contexts, which might undermine fidelity can be evaluated. Understanding and reporting how the intervention (including training and support, communication and management structures) is delivered is important for replication in clinical practice.17 Such evaluation aims to reduce the chance of discounting effective interventions (type II error) or erroneously attributing outcomes to treatment effectiveness, when interventions are not delivered as intended (type III errors).18–21 The approach is designed to improve trial design and knowledge translation interventions enhancing clinical implementation and reducing research waste.22 23
This paper reports the protocol for the process evaluation embedded in the RETAKE trial.
Aims and objectives
To measure fidelity to the ESSVR intervention and understand, the social and structural context in which the intervention is delivered and identify factors, which may influence the quality of implementation.
Fidelity measurement and competency assessment will
Ascertain intervention dose.
Describe content of usual care and ESSVR.
Describe levels of adherence to the ESSVR intervention.
Understand the delivery of usual Care and ESSVR.
Determine OTs competency to deliver ESSVR.
Social and structural context will include
Describe participating sites.
Understand professionals’ experiences of being trained to deliver the intervention.
Understand experiences of delivering the intervention.
Understand the social and structural factors, which support or act as barriers to the implementation of the intervention.
Understand participants’ experience of being supported to RTW after stroke.
Identify potential contaminants.
Embedded theory-driven mixed-methods process evaluation incorporating qualitative and quantitative methods. The process evaluation will draw on the intervention logic model developed by the Trialists (figure 1) and will be underpinned by normalisation process theory (NPT), an implementation theory built on four constructs (coherence, cognitive participation, collective action and reflexive monitoring) each informed by four components.24 NPT will be used in the development of data collection tools (interview topic guides and observation checklists (see table 1)) and as a sensitising lens in qualitative data analysis and interpretation. NPT constructs will underpin the process evaluation and provide insights into the implementation and integration of the intervention into participating stroke services. This will include how the intervention is received, understood, implemented and how it could be normalised into the current healthcare system.
Column 3 of the logic model identifies the core components of the ESSVR intervention. A more detailed description of the development and feasibility testing of the ESSVR intervention have been published previously.13
In addition, the Conceptual Framework for Implementation Fidelity (CFIF) (figure 2) will guide collection and analysis of quantitative data.25 The CFIF outlines the components and variables that make up and affect intervention fidelity and explains how they relate to each other. Adherence includes content and dose (frequency, coverage and duration) of the delivery.25
Stroke survivors that meet the following criteria for inclusion in the RETAKE trial will be eligible to participate in the process evaluation:
Admitted to hospital with new stroke (all severities).
In work at stroke onset (including self-employed, paid or voluntary).
Willing and have capacity to provide informed consent to participate in the study.
Have sufficient proficiency in English to contribute to the data collection required for research.
Potential participants who do not intend to RTW will be excluded. Potential participants with a transient ischaemic attack will be excluded.
Inclusion criteria for carers of potential participants:
Nominated carer of consenting participant.
Willing and have capacity to provide informed consent to participate in the study.
Have sufficient proficiency in English to contribute to the data collection required for research.
Potential participants will be provided with an information sheet and be provided the opportunity to ask questions of a researcher prior to consent. Written informed consent will be obtained from all participants. When a participant is randomised to the case study element, a researcher will contact the participant to gain consent for interview and observations. Consent will be reaffirmed at the start of interviews. This process will be the same for carer, employer, OT and NHS staff interviews. For employer interviews, additional consent to contact the employer will be requested from the case study participant before the employer is contacted. OTs who will deliver the ESSVR intervention and mentors supporting these OTs will be recruited prior to intervention training. NHS staff involved in the management, commissioning or delivery of stroke rehabilitation in each site participating in the RETAKE trial will be recruited.
For professional and patient interviews, as far as possible, we will use a purposive sampling strategy to ensure diversity in terms of geographical location (eg, urban vs rural centres), level of staff seniority and participant sociodemographic variables (including gender and socioeconomic status). See table 2 for the timepoints at which data collection is planned.
Patient and public involvement statement
Stroke survivors are involved in all stages of the research cycle.
Design and development
Two stroke survivors are coapplicants on the grant and assisted in identifying the research questions, designing the study and developing the trial protocol.
Two patient and public involvement (PPI) are members of the Trial Steering Committee, and two are members of the Trial Management Group. Additionally, our RETAKE PPI group, which has six members, meets quarterly. Examples of the work achieved by the PPI group to date are:
Helping define the primary outcome and defining ‘voluntary work’, which is included in the definition of the primary outcome.
Evaluating all patient-facing materials, including aphasia friendly recruitment material.
Codevelopment of interview topic guides for trial participants and occupational therapists.
Overcoming problems with recruitment. For example, resources and narratives to assist recruiters in approaching people with severe stroke.
Assisting in the design of new materials to promote follow-up, for example, including a ‘patient journey leaflet’ and Thankyou cards.
Helping reduce the length of follow-up questionnaires.
Advising on communicating with participants during the pandemic.
Changes to the Excess Treatment Cost payment models during trial caused problems for the study. One PPI member wrote directly to Directors of the National Institute for Health Research (NIHR), NHS England, Health and Social Care and the leads for the NIHR Clinical Research Network to explain the impact that these changes on the trial. She received a prompt response, which was extremely helpful to the research team. This has assisted us in explaining the new system to clinical colleagues and researchers in the Trusts.
Codevelopment of a trial website and trial newsletters.
A draft report on the process evaluation findings will be presented to the PPI group for their consideration and comments prior to submission of the final report to the funder and as part of planning publications and dissemination. The PPI group will be involved in writing up and presenting study findings.
The process evaluation will employ qualitative and quantitative methods to address the research questions. Table 2 illustrates the relationship between the process evaluation aims, research questions, data sources and data collection methods. The following section describes each data source in more detail.
Intervention content case report forms
To check on fidelity in terms of (early) intervention within 2 weeks of recruitment, initial session case report forms (CRFs) (one per participant) record the intervention start date and whether this occurred within 8 weeks of stroke. Participant summary CRFs record the number of sessions attended out of those proposed and whether there was an agreed ending for the OT-led RTW support. To ascertain intervention dose and describe intervention content, data will be extracted from intervention CRFs for all participants (see table 3). Therapists record each intervention session against predefined components, on the Intervention content CRF.13 These data will be used to identify which components of the intervention were delivered, to what extent therapists adhered to the intervention process described in the RETAKE manual and to what extent participants adhered to the intervention. For case study participants only, content data will be cross-referenced with the OT’s clinical case notes and additional data extracted to explain how the RETAKE intervention interacts with usual care and other services such as employment services. Participants’ consent includes permission for members of the trials team to access their therapy records.
Describing usual care
To describe the content of the intervention and of usual care, resource use questions pertaining to participants’ use of health and social care services over the previous 3 months will be completed by all participants at 3, 6 and 12-month postrandomisation as part of follow-up. This data will be used to describe the content of usual care, and in case study, participants (n=38) will be triangulated with therapists’ clinical notes and participant interview transcripts.
To assess implementation fidelity, a range of data collection methods informed by the CFIF will be used (see table 3).25
Therapist competency assessment
Following attendance at a 2 day, manualised face-to-face training session with VR expert trainers and again at refresher training 6 months later, retake OTs competence will be assessed using OTs written responses to questions based on vignettes depicting novel RTW after stroke scenarios. Model answers developed by the training team will be used to measure competence using criteria based on knowledge of the intervention process (40%), clinical reasoning (50%) and written communication (10%). Scores will be mapped to a rubric identifying OTs as highly competent (≥70%), competent (50%–69%) or needing additional support (≤49%) (see online supplemental appendix 2). In addition, as mentors meet with mentees on a monthly basis, informal monitoring of OT competency can occur. If required, action can be taken to addresses issues of concern identified by mentor or mentee. After 12 months of delivering the intervention, RETAKE OTs competence will be reassessed by evaluating the intervention delivered in a random selection of completed intervention case records (one participant per RETAKE OT) against the trainer’s expert opinion. The trainer will review the selected case records against the intervention mechanisms identified in the logic model and confirm whether the intervention delivered is consistent with the intervention that would have been delivered by the trainer as an expert RTW-related OT.
A fidelity checklist based on the RETAKE intervention logic model (see figure 1) and RETAKE intervention process and components will be applied to complete case records (Content of Intervention CRFs, RETAKE OT case notes and Initial Session CRFs) from a random selection of stroke participants randomised to receive the RETAKE intervention (one per treating RETAKE OT). This will be used in measuring adherence to the RETAKE process and identifying factors affecting adherence.
Mentor interviews and records
Following training, each treating OT will be assigned a mentor with extensive knowledge and experience of VR. Mentoring will take place monthly via teleconference in small groups (four to six therapists) and serve as an intervention implementation support mechanism. RETAKE OTs will be able to discuss any difficulties they are experiencing, ask questions and share best practice with other OTs and their mentor. This process will also facilitate communication between the trial team and enable barriers to implementation and contamination risks to be reported. Key discussion points will be recorded by mentors using a mentoring record form for each session. These records, along with all email correspondence between mentor and mentees, will be collected for qualitative content analysis.
Semistructured interviews will be conducted by two research assistants (SC and KC) with all mentors (n=6) to explore their experiences of supporting RETAKE OTs to deliver the intervention, and ascertain their views of organisational, social and other factors contributing to or affecting delivery of the intervention.
Social and structural context
To describe participating sites and identify potential contaminants, sites will be asked to complete a questionnaire by telephone at three time points; prior to recruitment, halfway through, and at the end of the intervention period. This will contribute to understanding contextual influences through capturing data on existing stroke care pathways and resources (including staff and services) available for supporting participants in a RTW. It will also identify potential contamination risks associated with proposed or planned VR service developments or changes in practice that may influence trial outcomes.
To understand OT’s experiences of being trained to deliver the intervention, a research assistant (RC) will observe up to four training sessions delivered by the training team. A checklist will be developed using NPT constructs to guide observations. Non-participant observations aim to identify; whether therapists understand the intervention and their role in implementation, whether they think the RETAKE intervention can be integrated into existing practice and any contextual factors affecting the trial.
To describe adherence to the intervention, a researcher will observe up to three sessions for each case study participant in the intervention and usual care arms of the trial. Non-participant observations will be conducted using prompts for structured observation and unstructured field notes.26 Participant selection for inclusion the case study element is described below.
Interviews with occupational therapists
Semistructured interviews will be conducted by a research assistant (RC) with a minimum of one OT per site following their initial RETAKE training to explore their experience of training, the mentoring process and their confidence in intervention delivery. OT’s views of the intervention, barriers and facilitators to implementation and any organisational or social factors impacting on delivery will also be explored. Interviews will take place following training and be repeated at two additional time points: mid-way through the RETAKE intervention delivery and at the end of the study.
Longitudinal case studies will be used to map the care received by RETAKE and usual care participants to develop a more detailed understanding of participants’ (stroke survivors, carers, employers) and RETAKE OTs experiences of support for RTW. A 5% subset of participants from both arms of the trial (total n=38) will be randomly selected and invited to participate in the case study element of the process evaluation:
Case study interviews
Semistructured interviews will be conducted by two research assistants (SC and KC) with case study participants at three time points: 3, 6-month and 12-month postrandomisation, about their experiences and views of and adherence to the RETAKE intervention and the support they received to RTW. The case study participants’ carers (if nominated), their employers (where participant consent is obtained) and the OTs providing support for RTW will be interviewed.
NHS staff interviews
To further understand the social and structural factors which influence the implementation of the intervention, interviews will be conducted with up to two (n=34 in total) NHS staff involved in the management, commissioning or delivery of stroke rehabilitation within each trial site. Participating staff will be chosen using a mixture of purposive and snowball sampling. This will be based on a full range of trial sites, staff knowledgeable about the implementation of the intervention at their site, and staff knowledgeable about the decision-making process relating to wider roll-out.
Additional participant interviews
An additional random 5% of study participants will be invited to participate in semistructured interviews at the end of the intervention period. These interviews will explore participants’ experience of the intervention as well as their perceptions and experiences of returning to work.
All qualitative interviews will be conducted using a topic guide informed by NPT. Examples of question topics and how they relate to the four NPT constructs are shown in table 1. Topic guides will be presented to the RETAKE PPI group for comment prior to use. All interviews will be audio recorded and transcribed in full.
The dose, duration and frequency of the ESSVR intervention will be calculated using data from completed CRFs in combination with NHS therapy records. The total time spent delivering the ESSVR intervention (face to face and non-face to face contact (liaison with the patient, employer and other stakeholders by letter/phone), administration and travel) will be identified. Details relating to the content of intervention sessions will be extracted to identify whether core components of ESSVR were delivered as intended (ie, as specified in the intervention manual and logic model). Associations between therapist attributes, contextual factors and intervention fidelity (measured by deviations from the RETAKE core process) will be explored using regression models. Analysis will be conducted using SPSS (V.21.0 for Windows).
Describing usual Care
Data regarding rehabilitation delivered in usual care will be extracted from resource use data in the follow-up questionnaires and from NHS therapy records in case study participants randomised to usual care. These data will be used to inform the cost of usual care for the economic evaluation and describe and understand usual care provided during stroke rehabilitation in inpatient and community services.
Quantitative analysis of these data will be conducted using SPSS (V.21.0 for Windows). Analysis of usual care data obtained from NHS therapy records is described below.
Inductive (thematic analysis) and deductive (informed by NPT) approaches will be used to guide data analysis and interpretation. Observational and interview data will be transcribed verbatim and uploaded into QSR NVivo software for management. Descriptions of usual care in NHS therapy records, observational field note data, including researcher reflections and interview data, will be analysed thematically.27 Framework analysis will be used with the case study data. For each participant, the interview data will be coded in NVivo and then imported into a framework matrix for comparison both within the individual case (comparing views of stroke survivor, carer, OT and employer) and across cases and sites. Analysis will proceed iteratively with data collection to determine whether data saturation has been achieved; researchers will draw on the RETAKE logic model (figure 1). Throughout the qualitative analysis, NPT will be used as a sensitising framework.
Analysis of each qualitative data set will be conducted independently and then jointly by at least two study team members (SC, KC, KP) to corroborate themes and discuss any discrepancies. It will follow a standard inductive approach of data familiarisation, line-by-line coding and development of broad themes. Themes will then be mapped to NPT constructs as part of development and refinement of broader conceptual explanatory categories. Researchers will keep a set of interim summary notes documenting any reflexivity points and connections between the data with NPT and the logic model, to aid analytical discussions with the wider process evaluation team. Iterative testing of interpretation will occur through discussion with and feedback from the PPI group and discussions within the research team.
Synthesis of qualitative and quantitative data
During the RETAKE trial, the qualitative and quantitative data generated as part of the process evaluation will be independently analysed by the process evaluation team and the Clinical Trials Research Unit, respectively. Data related to intervention fidelity and description of usual care will be synthesised at the conclusion of the trial. We will review and compare findings from related data sets, identify areas of agreement and disagreement and develop explanations for the findings. Synthesis of findings from both the quantitative and qualitative data generated will contribute directly to the overall evaluation and explanation of the outcomes of the RETAKE trial.
Ethics and dissemination
Ethics approval has been obtained through the East Midlands—Nottingham 2 Research Ethics Committee (reference 18/EM/0019) and the NHS Research Authority. The procedures for gaining informed consent have been detailed above. Dissemination will be via journal publications, stroke and rehabilitation-focused conferences, newsletter articles, social media, presentations to clinicians and stroke survivors and meetings with national clinical leads for the Stroke Plan and the NHS Plan.
Patient consent for publication
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Deceased Professor Walker has now retired and requested not to be included in the publications
Contributors KAR, CM, AF, AB, RJOC, MW and CW conceived the study. KAR, DC and CM designed the process evaluation. KAR, CM, DC, SC, KC, JH, JP and KP operationalised the process evaluation protocol. KR, JP, and JH designed the intervention. AS has the role of trial sponsor. IH, RB, and AFa devised the data management and statistical analysis plan. JS and JM acted as PPI collaborators to support plans for trial design/delivery, management, and dissemination of trial findings. VM and SH have responsibility for management of the trial. KAR, SC and DC drafted the manuscript; all other authors read and approved the final version.
Funding This study is funded by the NIHR HTA programme (ref: 15/130/11). The views expressed herein are those of the authors, not necessarily the NIHR, the Department of Health and Social Care or the NHS.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.