Introduction Treatment fidelity is a complex, multifaceted evaluative process which refers to whether a studied intervention was delivered as intended. Monitoring and enhancing fidelity is one recommendation of the TiDIER (Template for Intervention Description and Replication) checklist, as fidelity can inform interpretation and conclusions drawn about treatment effects. Despite the methodological and translational benefits, fidelity strategies have been used inconsistently within health behaviour intervention studies; in particular, within aphasia intervention studies, reporting of fidelity remains relatively rare. This paper describes the development of a fidelity protocol for the Action Success Knowledge (ASK) study, a current cluster randomised trial investigating an early mood intervention for people with aphasia (a language disability caused by stroke).
Methods and analysis A novel fidelity protocol and tool was developed to monitor and enhance fidelity within the two arms (experimental treatment and attention control) of the ASK study. The ASK fidelity protocol was developed based on the National Institutes of Health Behaviour Change Consortium fidelity framework.
Ethics and dissemination The study protocol was approved by the Darling Downs Hospital and Health Service Human Research Ethics Committee in Queensland, Australia under the National Mutual Acceptance scheme of multicentre human research projects. Specific ethics approval was obtained for those participating sites who were not under the National Mutual Agreement at the time of application. The monitoring and ongoing conduct of the research project is in line with requirements under the National Mutual Acceptance. On completion of the trial, findings from the fidelity reviews will be disseminated via publications and conference presentations.
Trial registration number ACTRN12614000979651.
- clinical trial
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
The Action Success Knowledge (ASK) trial’s fidelity protocol is a pragmatic, considered evaluative process, based on the National Institutes of Health Behaviour Change Consortium.
The fidelity protocol sets out the checks and balances in place to monitor and enhance fidelity within delivery of a complex intervention.
One limitation of the ASK fidelity protocol and checklist is that it primarily focuses on how the provider administers treatment to participants with aphasia, as they are the main focus for change.
Treatment receipt and enactment are embedded within the design of the ASK intervention (as demonstrated by the use of accessible aphasia-friendly materials, personalised goal-setting and follow-up telephone reviews), but they are not directly assessed.
Treatment fidelity is a complex, multifaceted evaluative process. The aim is to increase scientific confidence in the findings of behavioural intervention studies by monitoring and enhancing the reliability and validity of the intervention(s) delivered.1 2 For studies of interventions for health-related behaviour change, a high level of treatment fidelity is a marker of quality, indicating that end users (ie, researchers, clinicians, healthcare providers and patients) can have confidence in the study’s findings.3 Monitoring and enhancing treatment fidelity forms part of the TiDIER (Template for Intervention Description and Replication) recommendations, a consensus reporting checklist aimed at enhancing the complete description of interventions.4 Without treatment fidelity, conclusions cannot be drawn about treatment effects, leaving open the possibility for type I, type II or type III errors.1 3 5 Within a therapy trial, a common component of fidelity monitoring is to evaluate and enhance the extent to which the therapy provider administers the treatment as planned and how competently treatment is delivered.6–9 The purpose of this paper is to describe the development of a fidelity protocol for the Action Success Knowledge (ASK) study, a current cluster randomised trial investigating an early mood intervention for people with poststroke aphasia (an acquired language and communication disability). The protocol for the ASK trial has been published elsewhere.10 Data collection is currently ongoing; the results of the fidelity monitoring will be reported at the end of data collection, as part of the trial results paper.
Despite the methodological and translational benefits, fidelity monitoring has been used inconsistently within health behaviour intervention studies,11 and no single method has been widely adopted across studies.12 Assessing treatment fidelity is challenging in the context of complex behavioural treatments.13 Such behavioural treatments tend to be situated within an interaction between the therapy provider and the participant and involve a degree of customisation to suit the needs of the individual patient.8 14 Interpersonal interaction is an important contributing factor in delivering a high-quality treatment but creates challenges in training and monitoring treatment administration.13 The need for individual tailoring of the treatment to meet the needs of the patient creates high potential for variation in how the treatment is administered, particularly when the treatment is administered by different providers and across research sites.15 In response to these challenges, a comprehensive framework was developed by a panel of experts who formed the Treatment Fidelity Workshop as part of the National Institutes of Health Behaviour Change Consortium (NIH BCC).1–3 The NIH BCC framework expands on previous concepts of treatment fidelity to provide five domains of guidance for researchers within the broad field of health behaviour treatment (table 1).
Brief background of fidelity monitoring in aphasia treatment studies
A review of 149 aphasia treatment studies published between 2002 and 2011 found only 14% of aphasia therapy studies reported treatment fidelity.5 A more recent review16 of 42 aphasia randomised controlled trials (RCTs) published since 2012 found that 21% reported on treatment fidelity processes and one article addressed all receommended treatment fidelity components.17
Given the relative scarcity of fidelity monitoring inaphasia treatment studies and in the related fields of psychological treatment,5 18 it was necessary to develop a novel fidelity protocol and tool for the ASK study. The NIH BCC fidelity framework was used to inform and develop a fidelity protocol for the ASK study (table 1).
The ASK trial
The ASK study is an ongoing Australian-based multicentre, cluster-randomised controlled trial with a target recruitment n of 344, funded by the National Health and Medical Research Council (for more information, see the trial protocol10). The experimental treatment targets psychosocial well-being and was developed in recognition of the high incidence of depression in this population, estimated to be 62%–70% and higher than in the general stroke population without aphasia.19 The nature of aphasia makes it difficult for individuals to access usual care psychological treatments, which are typically ‘talking therapies’.20 Speech pathologists are well-placed to provide treatment, but they report feeling uncertain about how to do this.21 The ASK trial trains speech pathologists to deliver either a psychosocial treatment (experimental treatment) or an information-focused treatment relating to secondary stroke prevention (attention control treatment).
Theoretical underpinnings of the ASK interventions
The ASK experimental treatment draws on previous research which explored what factors influence living successfully with chronic aphasia.22 23 Themes identified as being key to living successfully with aphasia included: engaging in meaningful activities; having sufficient social support and positive adaptive strategies. These themes served as a search strategy to identify relevant psychosocial interventions that could be adapted for the population of people with aphasia. That is, the psychosocial intervention literature was explored to identify which interventions mapped onto the themes previously identified as influential in living successfully with aphasia. Those psychosocial interventions with the highest available evidence were incorporated into modules within the ASK experimental intervention. Thus, the ASK experimental intervention is underpinned by theories of positive psychology24 social problem-solving,25 adult learning26 and the life participation approach to aphasia.27 28
The attention control treatment is a secondary stroke prevention information programme that draws on previous research with patients with stroke.29 30 The intervention is underpinned by theories of adult learning,31 the health belief model32 33 and self-efficacy theory.32 This specific intervention was chosen as an attention control as it can be provided in a similar dosage and format to the ASK intervention. Furthermore, the content of a secondary stroke prevention information programme is relevant to all patients with stroke. The materials were adapted in line with aphasia-friendly formatting principles.34 The provision of secondary stroke prevention information has had no known demonstrated effect on the primary outcomes used in the ASK trial.
A central component of both the experimental and attention control interventions are the specific approaches of Supported Conversation for Adults with Aphasia (SCA).35 SCA acknowledges the ‘success’ of a conversation is not wholly dependent on the severity or type of individual’s aphasia; rather, the behaviours of the communication partner can influence the success of the conversation. As such, SCA sets out specific strategies for communication partners (such as family and health professionals) to implement in conversation with the individual with aphasia. These strategies are implemented by those around the individual with aphasia to support message in (understanding), message out (expressing a thought or opinion) and checking that the family member or health professional understood what the individual with aphasia intended (clarification).
The need to develop a specific fidelity protocol for the ASK study
In usual clinical care, speech pathologists do not typically lead administration of either of these treatments adding to the importance of monitoring fidelity within the ASK trial. Fidelity criteria were developed for both the experimental and the attention control treatments, and the same level of monitoring and feedback occurs for both arms. The rationale for each aspect of the ASK fidelity protocol is described below. The study design and training components of the fidelity protocol are monitored separately to the treatment fidelity of sessions delivered by therapy procedures. Criteria for monitoring the fidelity of each treatment session were developed to determine whether treatment fidelity was achieved by each provider, and across participants (tables 2 and 3).
Within the current paper, Section 1 will describe the fidelity processes within the ASK trial and how these processes map onto the NIH BCC framework. The fidelity criteria used to monitor therapy delivery for both the experimental and attention control arms are included (tables 2 and 3). Section 2 will focus specifically on fidelity monitoring of therapy delivery. Here, we describe the procedure for reviewing intervention sessions, such as the procedure for therapy providers to submit audio-video recordings. Figure 1 provides a schematic of this procedure.
Overview of the ASK treatments
The ASK treatments consist of weekly face-to-face sessions (minimum of 3, maximum of 8 sessions) followed by monthly telephone reviews until the participant reaches 12 months poststroke. The treatment is delivered by a speech pathologist who has been trained in the trial protocol and the study treatment. The first face-to-face treatment session focuses on goal-setting, when the therapy provider works with the participant with aphasia and their family member to generate relevant goals. Goal development is supported by the Goal Attainment Scale36 and written in a way that is accessible for individuals with aphasia. After the goal-setting session, the therapist is able to use the goals to a) tailor the content of the treatment modules to the participants’ specific needs and b) help the participants to reflect on any perceived changes during the course of the treatment. Within the experimental treatment, homework tasks are agreed at the end of each session, with the aim of supporting the participant to enact the strategies outside of the treatment sessions. Homework tasks are not prescribed in the attention control intervention. The content of the intervention modules are subject to ongoing investigation within the trial; to reduce the risk of unblinding assessors and/or participants the content of treatment will not be discussed in detail in this paper. For further details regarding the protocol of the trial, see Worrall et al.10
Development of fidelity processes within the ASK trial
The experimental ASK treatment was piloted in a Phase I feasibility study of n=9 participants.37 This development work led to production of a standardised manual for the experimental treatment detailing the theoretical basis of the treatment; supporting information for each module and plans to monitor treatment fidelity, protocol deviations, data management and safety. Clusters (health service districts) provide one trial treatment to minimise potential for contamination between the treatments. Further information can be found in the protocol paper.10
Treatment dose was specified a priori including the length of each treatment session, the number of face-to-face sessions and telephone reviews, and maximum duration of the treatment time. Participants across the experimental and attention control arms receive the same dose and frequency of treatment. A minimum and maximum treatment dose was set to allow for individual preferences of the participants with aphasia (minimum of 3, maximum of 8 face-to-face weekly sessions). Following face-to-face treatment, participants receive monthly telephone reviews. Participants are recruited and complete baseline assessment within 6 months poststroke, and treatment ceases at the participant’s 12-month anniversary of their stroke. For all sessions (face-to-face and telephone), treatment dose within and between groups is monitored via data records submitted by therapy providers. Any changes in dose are recorded as a ‘protocol deviation’, for example, a missed session due to participant illness. While family members are also recruited as participants and are invited to attend the treatment sessions, the treatment dose relates specifically to the participants with aphasia, as they are the primary targets of the treatments.
Standardised manuals and materials were developed for each role within the trial, that is, blinded assessor, recruiter and therapy provider. All roles are carried out by speech pathologists who have been trained by members of the trial management team, who are qualified speech pathologists. The assessor manual emphasises the theoretical underpinnings of each of the assessments, instructions on administration, skills required and data management and safety. Blinded assessors are usually contracted as casual employees nominated by the principal investigator at each site as suitable for this role.
Within both arms, the treatment manuals contain information on the theoretical underpinnings of the treatment as well as information on each module, data management and safety. Therapy is provided either by a private therapist employed as a casual employee of the trial, or by speech pathology staff at the participating clusters. In the case of the latter, suitable therapists are identified as those whose usual duties bring them into contact with patients with aphasia for screening, recruitment and administration of the study treatment.
Training for therapy providers across the trial arms is standardised, ensuring that all providers have access to the same content and amount of training. Training is carried out in person or via video conferencing with small groups of therapists, for the duration of 3 hours (for assessors or recruiters) or 6 hours (for therapy providers). Online refresher/booster training is available to all staff; this can be requested by the staff member at any time. Typically, staff access booster training following a period of leave. For therapy providers, the aim of the training programme is to ensure that staff understand the trial procedures, the rationale and theory behind the treatment, and how to tailor the treatment to suit the needs of an individual participant while not deviating from the essential components of the treatment.
Fidelity of treatment delivery is defined as the extent to which the therapist administers the treatment as planned and the competency with which the treatment is delivered.6–9 According to Bellg et al,2 assessment of treatment delivery includes monitoring and improving how therapy providers a) deliver only the target treatment (differentiation); b) acquire and maintain the required skills set (competency) and c) deliver the intended treatment components (adherence). ASK therapy providers receive a treatment manual which details the essential components of goal-setting as well as each of the treatment modules. Therapy providers are encouraged to tailor treatment to the individual needs of the participant, for example, through setting personalised, meaningful goals. In the experimental treatment, this tailoring is essential in order to encourage and support the participant to make changes in their behaviour, for example, shifting from potentially negative coping strategies to positive strategies. In the attention control treatment, tailoring ensures participants receive information that is relevant to them and at a pace that is acceptable.
Fidelity of treatment delivery is monitored using video recording of treatment sessions; recording is considered the ‘gold standard’ for monitoring treatment fidelity.2 All assessment and treatment sessions within the ASK trial are video recorded. Each therapy provider submits the audio-video recording of his/her first goal-setting session and first intervention module for assessment by the ASK fidelity monitor. This ensures that any deviation from the treatment is identified early by the fidelity monitor and steps taken to improve adherence to the target treatment. Providers’ behaviours are assessed in relation to content components and competency skills; see the ‘Procedure for monitoring and enhancing treatment delivery’ section. Due to their familiarity with the interventions and as they are unblinded as to the randomisation allocation of each site, the fidelity monitor role is carried out by one of two trial managers. As noted, the trial managers also delivered training to blinded assessors, recruiters and therapy providers within the trial.
Care is taken to engage the providers in the fidelity process and to be sensitive to any potential negative emotions they may feel when receiving feedback on their delivery of the study intervention. During the initial training session, the rationale for video recording treatment sessions is explained along with an outline of the process of fidelity checking including who will watch the video, how it will be assessed and what will happen if criteria are not met. This discussion highlights the importance of mutual respect, transparency and building trust. Therapy providers give their written consent to be video recorded. Before administering their first treatment sessions, therapy providers receive the treatment delivery fidelity criteria. This serves to further emphasise the essential components of the treatment and aims to reduce any provider anxiety relating to what the integrity monitor is assessing in the video recordings of treatment.
This category of fidelity focuses on whether the treatment was accurately ‘received’ by the participant (p. 8, 3). Borrelli points out that even if a treatment has been well-designed and competently delivered, the treatment will be ineffective if the participant is unable to understand or use the new skills. The target population of the ASK trial is people with aphasia and their family members. Aphasia can affect understanding, both through auditory and written modalities. Treatment receipt was not directly assessed; rather, it was embedded in the development of the treatment and materials, as well as provider training and treatment delivery. A number of steps were taken to support participants’ comprehension: the treatment materials were developed based on aphasia-friendly guidelines including simplified language, larger font, key words in bold, use of photographs/diagrams/pictures to support the written text and blank space.38 39 The treatment materials were developed by the research team before any participant commenced treatment. Essential criteria in the treatment fidelity checklist (tables 2 and 3) relate to the provider using techniques to support the participant with aphasia to understand (message in) and techniques to support the participant to ask questions, or express their thoughts or feelings, or to demonstrate their interpretation of the conversation (message out). These techniques draw on an evidence base of supported conversation,35 and are essential to ensure that the treatment is being delivered in an accessible way to participants with aphasia.
Enactment of treatment skills
While Bellg et al 2 acknowledge that treatment enactment is difficult both to conceptualise and to implement, they argue there is an important distinction between ‘what is taught (treatment delivery), what is learnt (treatment receipt) and what is actually used (enactment)’ (p. 450). Within the ASK study, enactment of treatment skills is not directly assessed or monitored but is supported in various ways.
For both the experimental and attention control interventions, the therapy provider and participants work together to generate person-specific, meaningful goals. Regular goal review throughout the intervention means that participants can report on how they are progressing and/or update their goals as needed. Goal-setting is assessed by the fidelity monitor and feedback given regarding the appropriateness of goals; goal achievement is not assessed within the fidelity monitoring process.
For both the experimental and attention control interventions, enactment is supported via follow-up telephone sessions for both arms. After attending face-to-face sessions with the therapy provider, participants receive monthly telephone calls from the therapy provider until the 12-month anniversary of their stroke. The purpose of these calls is to reflect on progress related to the participant’s goals, check if the participant requires more information or discussion in relation to these and to explore any obstacles to enacting the skills taught in treatment. The telephone sessions are audio recorded and clinicians submit data on the nature of the discussion and length of the call.
For the experimental intervention only, it is an essential fidelity criterion that therapy providers set homework with each participant. The homework task is set at the nd of each session, reflecting the discussion that took place in the session. The homework task builds on the most recent therapy session and/or generates opportunities for the participant to practise a skill outside of the clinic. In the following session, the therapy provider reviews the homework with the participant. Reasons for non-completion of homework are explored, as these can indicate a lack of understanding or obstacles to implementing the learnt skills. The therapy provider’s behaviours (ie, setting and reviewing homework tasks) are monitored within the fidelity review; the participant’s completion of the tasks as directed is not directly assessed in the fidelity process.
Procedure for monitoring and enhancing treatment delivery
All assessment and face-to-face treatment sessions are video recorded. Each therapy provider submits his/her first goal-setting session and first module for assessment by the ASK fidelity monitor. Dependent on resources within the trial, later sessions are selected at random for fidelity checking to ensure skills have been maintained. In particular, assessing the first goal-setting session and the first module ensures that any deviation from the treatment is identified early by the fidelity monitor and steps are taken to improve adherence to the target treatment. The fidelity monitor watches 100% of the recordings of the goal-setting and first modules; this allows the fidelity monitor to get a broad sense of the session, rather than simply focusing on the frequency of specific behaviours.
Providers’ behaviours are assessed in relation to content components and competency skills, and given a score based on whether these behaviours were evident in the recording (see table 2 for the fidelity criteria relating to the goal-setting session, and table 3 for criteria relating to the intervention modules). Once fidelity assessment is complete, the fidelity monitor provides written and verbal feedback to the therapy provider regarding whether they met fidelity criteria and on specific aspects of the sessions. If fidelity criteria are met, the therapy provider continues to video record all subsequent treatment sessions. If the therapy provider does not meet fidelity criteria for the first goal-setting or module sessions, he/she is provided with written and verbal feedback and coaching. Subsequently, the therapy provider is required to submit his/her next session to ensure that the necessary changes have been incorporated. This cycle of submitting a video recording and receiving feedback and coaching continues until the therapy provider meets the fidelity criteria.
Development of a tool to assess treatment delivery fidelity within the ASK study
The treatment fidelity criteria were developed from the proposed active ingredients of each treatment module. These criteria reflect the content components and the competency skills necessary to deliver the target treatment (see table 2 for fidelity criteria relating to the goal-setting session; see table 3 for fidelity criteria relating to the treatment modules). Inter-rater reliability is currently being investigated as the trial continues to collect data.
Adherence to content components
The key content components of the treatment were identified for each module (goal-setting module and the therapy modules) based on the underlying theory and evidence.
Adherence to competency skills
Competency skills were defined as the communication strategies used by the provider to support the participant with aphasia to understand the information and to express their thoughts, feelings and questions. Identification of these skills was informed by the extensive literature on supported communication, that is, there is strong evidence to suggest the skill of the communication partner can have a substantial impact on the communication ability of the person with aphasia.40 41
Overall marker of quality of the treatment session
All behaviours (content and competency) are categorised as either ‘essential’ or ‘desirable’. To demonstrate fidelity to the intervention protocol, therapy providers must demonstrate 100% of all essential behaviours. Additionally, providers must also demonstrate a minimum number of desirable behaviours (ie, considered high quality but not essential). Feedback on all behaviours is provided to the therapy providers. Evaluation of adherence to treatment content and competency components involves scoring occurrences of behaviours; a binary system of ‘present’ (1 point) or ‘absent’ (0 points) is applied and the scores counted. Scores for essential and desirable behaviours are combined to give a total maximum score for each session; a higher score is interpreted as a marker of quality of the treatment session. The threshold is set at 100% adherence to essential criteria and an overall minimum score of 80% adherence, that is, therapists need to demonstrate at least 80% fidelity to the protocol to be considered adherent. A score below the minimum results in the conclusion that the therapist has not met the treatment fidelity criteria for that session.
Within health-related behavioural treatment studies, it is crucial that treatment fidelity is monitored, enhanced and reported in order to increase the power to detect treatment effects and to increase confidence in the study’s findings. Monitoring treatment fidelity is still relatively rare in the field of aphasia treatment studies, with a previous review indicating that 14% of aphasia studies reported fidelity5, with a more recent review indicating that 21% of aphasia RCTs reported fidelity. The fidelity protocol and checklist developed within the ASK trial provides a useful template for other aphasia and psychosocial treatment studies. One limitation of the ASK fidelity protocol and checklist is that it primarily focuses on how the provider administers treatment to participants with aphasia, as they are the main focus for change. While family members are recruited as participants and attend intervention sessions, their participation is not included in the current fidelity protocol. Yet, their support is likely to be influential in how the individuals with aphasia receive the treatment and embed the intervention into everyday life. Future studies of intervention involving participants with aphasia and family members could consider adding specific fidelity criteria that relates to the family member. A further limitation is that, while treatment receipt and enactment are embedded within the design of the ASK intervention (ie, accessible aphasia-friendly materials; personalised goal-setting; follow-up telephone reviews), there is no direct assessment of treatment receipt or enactment. In particular, treatment enactment is challenging to directly assess and may rely on self-report from study participants. Future aphasia studies could directly assess treatment receipt and enactment, for example, review whether participants felt they had achieved their goals or collect participant self-reports on whether the strategies learnt in therapy had become embedded into their daily life.
Patient/ and/or public involvement
Patients and/or public were not involved in the development of the ASK fidelity tool.
Ethics and dissemination
The monitoring and ongoing conduct of the research project is in line with requirements under the National Mutual Acceptance with the submission of progress reports, safety and/or adverse event reports and amendments. On completion of the trial, findings from the study and from the fidelity reviews will be disseminated via publications and conference presentations.
Contributors LW is the chief investigator for the study and assembled the team. All authors contributed to study design. IK oversaw the development of the fidelity protocol. As trial managers for this study, BR and MC led staff training and monitored videos submitted for fidelity review. MC wrote the fidelity protocol with input from IK, BR, ST, LW, NS-M, MR, LT and TCH. MC wrote the draft manuscript; all authors (BR, LW, ST, MR, NS-M, AK, TH, EP, LT, IK) contributed to manuscript revision and approved the final manuscript for submission.
Funding This research was funded by a grant from the National Health and Medical Research Council Project from 2014 to 2018 (APP1060673).
Competing interests None declared.
Ethics approval The study protocol was approved by the Darling Downs Hospital and Health Service Human Research Ethics Committee (HREC) in Queensland, Australia under the National Mutual Acceptance scheme of multicentre human research projects conducted in publicly funded health services. Based on this approval, expedited approval for the study was granted by the University of Queensland. Ethics approval was also obtained for participating sites not approved under the National Mutual Agreement scheme at the time of application.
Provenance and peer review Not commissioned; externally peer reviewed.
Patient consent for publication Not required.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.