Evaluation of intervention fidelity of a complex psychosocial intervention Lifestyle Matters: a randomised controlled trial

Objectives Robust research of complex interventions designed to promote mental well-being in later life is required to inform service development. An essential component is ensuring that such interventions are delivered as intended. We present a detailed description of the design and implementation of a fidelity assessment within a trial of one such intervention (Lifestyle Matters). The findings help to explain the trial results and also inform the design of embedded fidelity assessments within future evaluations of complex interventions. Design We conducted a mixed-method fidelity assessment embedded as part of a multicentre pragmatic randomised controlled trial. A conceptual fidelity framework was developed from the Behaviour Change Consortium framework. From this the fidelity assessment was designed. The resulting instrument assessed the following parameters: intervention design, training, supervision; and delivery, receipt and enactment of the intervention. Intervention The Lifestyle Matters intervention was designed to assist older people to improve and sustain mental well-being through participation in meaningful activity. The aim is to enable participants to engage in both new and neglected activities through a mix of facilitated group meetings and individual sessions. Results The fidelity assessment demonstrated that the intervention was delivered as per protocol for the group component and was tailored to meet individual needs. There was substantial inter-rater agreement for training; and group member performance 0.72; and moderate agreement for facilitator performance 0.55. It was not possible to determine whether small declines seen in facilitator performance were due to facilitator drift or moderating factors such as group dynamics or participant characteristics. Conclusions The assessment methods adequately measured criteria identified as being significant indicators of fidelity. Adherence during training, delivery and supervision was good. The subjective nature of identification and rating observed behaviours was the main challenge. Future research should explore alternative methods of assessing fidelity in trials of complex interventions. Trial registration number ISRCTN67209155.

There seem to be confusion between several concepts namely fidelity assessment, techniques to improve fidelity, maintenance of fidelity and causal mechanisms. The authors do not use these terms accurately at times leading a reader to be potentially misinformed. For example authors state that fidelity assessment can increase effect sizes and improve powersurely it is maintenance of fidelity which can do this not simply assessment. There is also discussion around causal mechanisms and their role in implementation. Understanding causal mechanisms is important and is often part of a process evaluation, which fidelity too falls into, but they are not the same and it is confusing to present them as if they are. Without clear understanding by the authors of the different ideas they are presenting there is a danger that a reader would actually become less informed after reading the paper.
More background on the rationale for the intervention and content would be useful. For example it is not clear why a largely retired population would be offered an occupational intervention.

Methods
In the fidelity assessment there seems to be primarily a focus on adherence but little consideration of competence. This is a critical aspect of fidelity as discussed in Carroll 2007. A reflection of this and the limitations this gives to the current fidelity assessment should be explored.
It is not apparent how the scoring system was devised e.g. why were present scores converted to '3'. Also on what basis were the percentage scores for fidelity decided? I am not sure on the validity of having a coder also be part of the training intervention, further justification of this and discussion on the potential for bias this brings in should be added in the discussion.
How was it decided which sessions should be recorded and which groupsthis should be explicit and consider whether any potential bias was brought in to play.

Results
I do not find the tables particularly informative it would be helpful in the text to have greater discussion on which aspects of fidelity were met to a greater or lesser extent One way the qualitative interviews could have informed fidelity assessment is if there was comment about actions taken which reflected receipt and use of training, however this does not appear to be present. Discussion is more around techniques to enhance fidelity, which whilst helpful, is slightly separate from an assessment of fidelity and it would be helpful for this to be noted.

Discussion
There is ongoing debate between the importance of fidelity versus adaptation for interventions that are to be implemented. There is no discussion of these important issues and how this paper fits within these, which would make this paper of greater interest.

REVIEWER
Prof Nusrat Husain University of Manchester, UK REVIEW RETURNED 03-Dec-2020

GENERAL COMMENTS
Fidelity measure is most important to ensure robust scaling up of psychological and social interventions. This is often not reported and is a major limitation in implementation science. I have only a few comments and it would be helpful if the authors could provide some further details. I am finding it difficult to understand what was mentioned in "strengths and limitations" section -"it was not appropriate or even ethical to undertake observations of activities in the community". It will be helpful if the authors could elaborate particularly around ethics.
In background section authors made a strong rational for this study. However, some references need to be updated such as for "MRC recommendations for complex interventions".
On page number 7, line number 10; it would be useful to add reference for the manual that was used in the trial. Authors should describe facilitators load (how many participants/groups per facilitator at a given time) as it may effect quality of intervention delivery and hence fidelity. Video recording of training was done with consent of facilitators and trainers? How many facilitators in total were involved? Were all facilitators' performance was rated? Supervision is critical across all areas but is particularly important in task shifting and task sharing. If the authors could report what was the frequency of supervision sessions and format of supervision? Content checklist should be included in appendices.
In qualitative section authors need to explain the use of topic guide, who conducted interviews, why only 10% participants were selected for interviews? It would be interesting to know whether there were any differences in fidelity between group vs individual sessions.
Typos page 7 line 17 "together to on future" "findings It was estimated" "For the purposes of the fidelity assessment, data from the interviews to identify cross cutting themes and explanatory factors for fidelity outcomes" Overall in my opinion it is a well written manuscript and will be a useful addition to the literature.

Reviewer 1
There is little consideration of the recent progress that has been made in We have revised the manuscript, the area with publications such as that by Walton 2019 particularly the discussion to reflect Typos page 7 line 17 "together to on future" "findings It was estimated" All errors have been corrected.
"For the purposes of the fidelity assessment, data from the interviews to identify cross cutting themes and explanatory factors for fidelity outcomes" I think suggesting the qualitative work explores causal and mediator and moderator roles is rather overstating the role of this work. In addition, mediator and moderator are common statistical terms for describing relationships between intervention outcomes and processes. In my opinion it would be preferable to present the qualitative work as simply a complementary (but still important and useful) method to expand understanding of intervention delivery by gaining the facilitators perspective on what could have improved the intervention and things they found helpful or difficult in the delivery. I do not feel in any way the small amount of data presented represents analysis of causal mechanisms.

Editor
3. On page 5 ln 24/25 you mention the discussion of fidelity will be related to intervention implementation. It is not mentioned whether the intervention was shown to be effective and how fidelity relates to this. I think this would be of interest to the readers. Indeed on looking up publication 11 it appears there was no clinical or cost effectiveness demonstrated so it would not be appropriate to go ahead with implementation. Discussing fidelity in the context of this non-significant finding is still important as it allows you to evaluate one possible reason for this lack of effect. I think such discussion could be a helpful addition to the manuscript.
Methods 4. Pg 6says facilitators were recruited with required skills but the reference for this is then Borelli fidelity framework. Could the authors be more explicit in the exact skills they required and how they determined these were necessary.

Supervision is a strength -I wondered if there was any information on the outcome of this.
6. At what point was the fidelity framework developed, could this be made a little more explicit.
Results 7. Pg 6 Ln 17 States -'A crucial element is going into the community to try out new or neglected skills with support from each other and from the facilitators; at least four sessions should involve such activities.' However only 4% of individuals achieved this which says to me the intervention was not delivered/received as intended. I don't feel this comes across in the conclusions. Rather the manuscript says that the intervention was delivered as intended in the abstract.
8. Similarly I think it is of note that one aspect of fidelity which you mention that was not delivered as intended is the goal setting. This is an important element of behaviour change so if this was not delivered it may give an important clue as to why the intervention did not work. I think this and the above in point 7 should be reflected on more in the discussion as it has important implications for how future interventions are designed.
9. The approach to assessing training delivery by being a participant is an interesting one. It allows the researcher to get a feel for some of the more subtle elements of delivery such as group dynamics etc, however it does suffer from some disadvantages in that by being a participant one can take a less objective and panaromic perspective. I think further reflection on this in the discussion would be useful.
10. It is interesting that the inter-rater coding was at the lowest end of good. Did the authors take any actions to improve inter-rater reliability throughout the coding process.
11. I was unclear whether there was feedback after wave 1 to support the delivery of the intervention i.e. was the fidelity assessment only descriptive or was there a formative element as well?
12. The qualitative findings are very brief with no themes presented. In the methods you mention framework analysis. If there was a change in analysis plan this should be mentioned and reasons discussed.
13. In the methods you mention interviewing participants yet I do not see any presentation of the results. I think this needs to be aligned with either methods saying this is reported elsewhere or presentation of findings in this paper.
Discussion 14. One major limitation of the fidelity assessment is that it is focussed on adherence and does not reflect quality or competence. Whilst this may have been beyond the resources of the fidelity assessment and therefore not possible I do feel it is important to reflect on this limitation in the discussion.
15. As noted above I think more discussion of which bits were delivered with fidelity and which bits weren't and how this potentially relates to the outcome of the trial would be useful.

Husain, Nusrat
The University of Manchester, School of Medicine REVIEW RETURNED 09-Mar-2021 GENERAL COMMENTS I want to thank authors for addressing comments in the revised manuscript. I have few observations on revised manuscript that I want to highlight; To avoid confusion it would be better to take out lines about the causal mechanisms as this manuscript is focusing only on fidelity and there is no information on causal mechanisms in methods, results and discussion.
In "practicalities" authors have now explained that "there were ethical considerations around participant confidentiality when assessing fidelity in a community setting". Can we add a reference here describing why this is an ethical concern?
In background section there are still a number of references that are old such as on page 3 paragraph 2 in background section authors cited reference from 2004 and 2007 to explain fidelity assessment (Toomey, E., Hardeman, W., Hankonen, N., Byrne, M., McSharry, J., Matvienko-Sikar, K. and Lorencatto, F., 2020. Focusing on fidelity: narrative review and recommendations for improving intervention fidelity within trials of health behaviour change interventions. Health psychology and behavioral medicine, 8(1), pp.132-151.) On page 13 under the heading "semi-structured interviews" you have added a rationale for selection of 10% of the sample and you talked about data saturation. It would be good to add a reference here (Braun, V. and Clarke, V., 2019. To saturate or not to saturate? Questioning data saturation as a useful concept for thematic analysis and sample-size rationales. Qualitative Research in Sport, Exercise and Health, pp.1-16.). 4. Pg 6says facilitators were recruited with required skills but The Borelli paper is referenced as it the reference for this is then Borelli fidelity framework. Could discusses the importance of the authors be more explicit in the exact skills they required preserving fidelity through the right and how they determined these were necessary. "fit between the provider and the 2 population", so recruiting the right people is essential.

VERSION 2 -AUTHOR RESPONSE
We have added more detail on the recruitment process and skills required as requested.

5.
Supervision 14. One major limitation of the fidelity assessment is that it is We believe we have already covered focussed on adherence and does not reflect quality or this subject under the header competence. Whilst this may have been beyond the resources 'Training and competence' in the of the fidelity assessment and therefore not possible I do feel it discussion.
is important to reflect on this limitation in the discussion.
15. As noted above I think more discussion of which bits were We have addressed this point delivered with fidelity and which bits weren't and how this through edits made in response to potentially relates to the outcome of the trial would be useful. the above points.

Reviewer 2
To avoid confusion it would be better to take out lines about the We have addressed this point causal mechanisms as this manuscript is focusing only on fidelity and through edits made in response to there is no information on causal mechanisms in methods, results reviewer 1. and discussion.
In "practicalities" authors have now explained that "there were We have included a reference as ethical considerations around participant confidentiality when suggested.
assessing fidelity in a community setting". Can we add a reference here describing why this is an ethical concern?
In background section there are still a number of references that are We have updated the reference as old such as on page 3 paragraph 2 in background section authors suggested.
cited reference from 2004 and 2007 to explain fidelity assessment On page 13 under the heading "semi-structured interviews" you We have updated the reference as have added a rationale for selection of 10% of the sample and you suggested.
talked about data saturation. It would be good to add a reference here