Introduction The effectiveness of complex interventions for the management of musculoskeletal disorders has been estimated in many randomised clinical trials (RCTs). These trials inform which interventions are the most effective, however they do not always inform how an intervention achieved its clinical outcomes, nor how and what elements of an intervention were delivered to patients. Such information is useful for translating findings into clinical practice. A few process evaluation studies have been conducted alongside RCTs and a variety of methods have been used. To gain a better understanding of current practices of process evaluation in RCTs in musculoskeletal disorders, this systematic review is designed to answer the following research question: How are process evaluation of complex interventions tested in RCTs in musculoskeletal disorders being conducted?
Methods and analysis We will systematically search seven electronic databases (MEDLINE, SCOPUS, CINAHL, PsycINFO, EMBASE, Web of Science and Cochrane database) from the date of inception to August 2018 for studies on process evaluation of RCTs on non-surgical and non-pharmacological management of musculoskeletal disorders. We will include qualitative and quantitative studies conducted alongside RCTs, reported with the RCTs or separate studies that assessed interventions for musculoskeletal disorders. Two reviewers will screen abstracts and apply prespecified inclusion criteria to identify relevant studies, extract the data and assess the risk of bias within included studies. We will follow recommendations from the ‘Cochrane Qualitative and Implementation Methods Group Guidance Series’ when assessing methodological strengths and limitations of included studies. We will use a narrative synthesis to describe findings.
Ethics and dissemination Ethical approval is not required as this review will not collect original data. Findings from this systematic review will be presented at a scientific conference and published in a peer reviewed journal.
PROSPERO registration number CRD42018109600
- process evaluation
- musculoskeletal disorders
- systematic review
- randomised clinical trial
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- process evaluation
- musculoskeletal disorders
- systematic review
- randomised clinical trial
Strengths and limitations of this study
One strength of this study is the comprehensive search of published and unpublished literature in different databases and trial registries.
The other strength of this protocol is its scientific robustness. When designing this protocol we followed the Cochrane Qualitative and Implementation Methods Group Guidance Series.
A limitation of this review is that we may not identify other process or outcome evaluation studies that assessed aspects of process evaluation but did not explicitly report them in the title or abstract.
Musculoskeletal disorders are the second largest cause of disability.1 Common musculoskeletal disorders include low back pain (the most frequent disorder),1 shoulder pain and neck pain (the second and third most prevalent musculoskeletal disorders)2 and osteoarthritis (the most common joint disorder).1 The individual, societal and economic burden of musculoskeletal disorders is high. For example, in New Zealand, between 2005 and 2013, the direct cost of physiotherapy interventions for shoulder injuries alone was $134 million ($14 million/year).3 The total direct costs of osteoarthritis were greater than $500 million in 2005.4 Together, high direct costs and waiting lists highlight the need for effective and affordable interventions to minimise the growing burden (social and economic) of musculoskeletal pain.
To improve healthcare services, it is crucial to identify which interventions are the most effective.5–7 Randomised clinical trials (RCTs) estimate the effectiveness of different interventions5 8–11 and can be referred to as ‘outcome evaluation’ trials.12 Some RCTs may also include an economic analysis that is run in parallel.13–16 This type of trial is referred to as ‘economic evaluation’ trial.12 While being extremely important for improving healthcare, ‘outcome and economic evaluation’ trials do not inform how interventions achieved their clinical outcomes.12
Complex interventions are defined as interventions with multiple components that may interact with each other, influencing clinical outcomes.17 18 The interaction between the components of an intervention can impact on clinical outcomes.17 18 Examples of complex interventions include exercise therapy, education and behavioural change.13 19 20 Complex interventions represent a challenge for researchers when planning a trial.21 These challenges are caused by the difficulty in standardising the way an intervention is delivered,22 the possible influence of the local context on clinical outcomes,23 logistical challenges at the organisational level and complex interactions between the components of the intervention and the clinical outcome.21 22 24
When assessing the effect of a complex intervention, researchers might need to take into account the interacting elements within the experimental and control interventions, variability of outcomes, behaviour of professionals delivering as well as behaviour of patients receiving the intervention and how much flexibility is permitted for adapting an intervention being tested.18 21 There is no clear threshold for classifying an intervention as simple or complex, but only a few interventions can be considered simple.17 18 Understanding how an intervention achieves its clinical outcomes is particularly relevant when assessing effectiveness of complex interventions.18
Process evaluation studies inform how a complex intervention achieves its clinical outcomes.12 These studies provide information about how and what elements of an intervention were delivered to patients8 11 12 25 and why interventions work (or fail to do so) the way they do.26 Process evaluation studies may inform about what and how interventions are implemented (ie, implementation), how interventions generate change in clinical outcomes (ie, mechanism of impact) and how the context affects the clinical outcomes (ie, context).8 11 27 Such information is useful for translating findings into clinical practice.12 28
Process evaluation methods are still being developed12 and are relatively scarce in musculoskeletal research.29 The Medical Research Council (MRC) (UK) published its first guideline for process evaluation of clinical trials in 2015.12 The number of process evaluation studies conducted alongside RCTs is small, and the approaches used to assess implementation of interventions are varied. Previous reviews have focused on process evaluation of trials in fields other than musculoskeletal disorders, for example, complex interventions for patients with chronic diseases28 and interventions for patients with neurological disorders;30 another review assessed what is being measured in process evaluations for worksite health promotion programmes.31
We planned this review to identify approaches used for assessing process evaluations of trials focusing on complex interventions (non-surgical and non-pharmacological) for the management of musculoskeletal disorders. The systematic review was designed to answer the following overarching research question: How are process evaluations of complex interventions tested in RCTs in musculoskeletal disorders being conducted?
The specific research questions were:
Is there a theory adopted by research teams when conducting process evaluations? If so, which theory is used (eg, theory-based evaluation, realistic evaluation)?
Which study designs were used during the process evaluation (eg, qualitative, quantitative or mixed-method)?
When is process evaluation being performed (ie, phase II - feasibility and piloting; phase III - evaluation; or phase IV - implementation)?
How are results of the trial being integrated with the findings from the respective process evaluation study?
Is the process evaluation independent or does it become independent at some stage in the trial?
What are the barriers and facilitators faced by the authors while conducting process evaluations?
What are the strengths and limitations of the process evaluation methods reported by the study authors?
We used the Preferred Reporting Items for Systematic review and Meta-Analysis Protocols for planning and reporting this protocol.32 We planned the assessment of process evaluation of RCTs based on Cochrane Qualitative and Implementation Methods Group guidance papers.33–38
We will search seven databases including MEDLINE, SCOPUS, CINAHL, PsycINFO, Embase, Web of Science and Cochrane database in order to identify relevant studies evaluating process evaluation of complex interventions for the management of musculoskeletal disorders tested in RCTs. To identify any unpublished literature, we will additionally search four clinical trial registries (Cochrane Central Registry of Controlled Trials, ClinicalTrials.gov, EU clinical trials register and Australian New Zealand Clinical Trial Registry). If we identify any project that is likely to meet the inclusion criteria, we will contact the authors for the results related to the process evaluation.
We will combine ‘process evaluation’, ‘ program evaluation’, ‘fidelity’ or other search terms that mean process evaluation with ‘musculoskeletal disorders’ and its related terms. We will exclude all surgical or study protocols by using NOT ‘surgery’ and ‘protocol’ respectively or their substitute words.
Prior to conducting the full search on electronic databases, we conducted a pilot search using MEDLINE, SCOPUS, CINAHL, PsycINFO, Embase and Web of Science databases. This was done for scoping purposes and the pilot search yielded a total of 1695 studies. The terms used during the pilot search are reported in table 1. Following this, we consulted a health sciences librarian for optimising the search strategy. The search strategy will be adapted for each database.
Types of study to be included
We will include all qualitative and quantitative studies conducted alongside RCTs, reported with the RCTs and separate reports that assessed process evaluation of RCTs on complex interventions (non-surgical and non-pharmacological interventions) for musculoskeletal disorders. We defined a process evaluation study as any study aimed at understanding the functioning of an intervention by examining the implementation, mechanisms of impact and/or its contextual factors.12
To be included, studies need to indicate in the title or in the aim of the study that they are assessing components of process evaluation (eg, implementation, mechanisms of impact or context). We will include any study that: (1) explicitly indicates that it was a process evaluation study in the title or in the aim of the study, or (2) intends to evaluate process evaluation (eg, fidelity, dose delivered, dose received, reach, recruitment, context, barriers, implementation), without explicitly stating that it is a process evaluation study (eg, qualitative study alongside an RCT).
Condition or domain being studied
We will include process evaluation of RCTs investigating complex interventions for the management of musculoskeletal disorders. For the purpose of this review, we defined musculoskeletal disorders as health problems of the locomotor apparatus (including muscle, tendon, skeleton and ligaments) including for example, low back pain, neck pain, shoulder pain, elbow pain, hip pain, knee pain and soft tissue injuries. We will exclude systemic conditions (eg, gout, rheumatoid arthritis, Raynaud disease, scleroderma and dermatomyositis), osteoporosis, tumours, infections of bones and joints, fibromyalgia, diabetic neuropathy, fractures, ankylosing spondylitis and spinal cord injuries.
Studies should include participants with musculoskeletal disorders who received complex interventions (non-surgical and non-pharmacological) as part of an RCT or clinicians delivering interventions as a part of a clinical trial. To be included in the review, studies must have recruited adult humans (ie, >18 years old).
Complex interventions of musculoskeletal disorders in an RCT include, but are not limited to, exercise therapy, physical activity, self-management advice, education and psychosocial interventions. We will not include studies assessing surgical or pharmacological interventions.
We will include studies that compared complex interventions with appropriate control groups (eg, waiting list, placebo groups, other active interventions). We will exclude studies that compared surgical or pharmacological interventions to non-surgical or non-pharmacological interventions.
We will include studies that assessed effectiveness of interventions in primary care (eg, private practice, home-based interventions and community-based interventions).
These are in line with our study questions and are as follows: (1) Theory (if any) adapted to conduct the process evaluation; (2) Study designs used for process evaluation; (3) Phase of the trial when the process evaluation was performed; (4) Approach used to integrate the process evaluation with the main results of the RCT; (5) Barriers and facilitators faced by authors while conducting process evaluation of RCTs; (6) Strengths and limitations of the process evaluation methods as reported by the study authors.
Data extraction (selection and coding)
Prior to screening, we will remove duplicate articles. Then, two reviewers will independently screen all titles and abstracts following the eligibility criteria and using a standard form (see online supplementary file 1). After the first screening, the two reviewers will meet to assess the agreement on inclusion or exclusion of the studies based on title and abstract reading. Then, both the reviewers will independently screen the full text of the articles for all the articles that meet the inclusion criteria based on the title and abstract or just based on the title, the abstract being uncertain. During title, abstract and full-text screening, disagreements will be resolved by consensus. If consensus is not reached, then a third reviewer will be consulted.
Supplementary material 1
The research team will develop a form for extracting data from the included studies. The data extraction form will be designed based on the ‘MRC of UK recommendation for process evaluations’ and the ‘Cochrane Qualitative and Implementation Methods Group Guidance Series’.33–38
Data extraction forms will be piloted by two reviewers using articles that were included after full-text screening. Following recommendations from The University of York Centre for Research and Dissemination , data extraction forms will be piloted on a random sample of 10 included studies. This will ensure that resources are not wasted, and that all relevant information is being extracted from included studies. Once the research team agrees that the form is comprehensive and coherent, two reviewers will independently extract data from studies that were included after full text screening. Disagreements will be resolved by consensus. If consensus is not reached, then a third reviewer will be consulted.
We will extract data regarding: (1) Basic information about the study - publication year, authors, title, study type, aims; (2) Context and participants - study setting, population, participant characteristics, intervention delivered; (3) Methods used - design, methods used for sample recruitment, data collection and analysis, theoretical model used to interpret data and contextualise findings; (4) Process evaluation - rationale for study design adopted, dose delivered, participants’ attitudes and beliefs, approach used to assess participants’ adherence and fidelity to intervention protocol, approach used to assess clinicians’ adherence and fidelity to intervention protocol, description of clinicians, training of clinicians, implementation monitoring, theory supporting process evaluation, process evaluation findings, association between process evaluation and outcome evaluation findings.
Risk of bias assessment
We will assess the risk of bias of the RCT report if process evaluation is reported within the outcome evaluation (ie, RCT study). In that case, we will use the Cochrane Collaboration tool for assessing risk of bias (reference).
When process evaluation is reported as an independent study, we will follow recommendations from the ‘Cochrane Qualitative and Implementation Methods Group Guidance Series’ for assessing methodological strengths and limitations of the included studies. In those cases, we will assess the risk of bias of the process evaluation study alone. The following domains will be included in the assessment: (1) Clear aims and research question; (2) Congruence between the research aims/question and research design/method(s); (3) Rigour of case and/or participant identification, sampling and data collection to address the question; (4) Appropriate application of the method, richness/conceptual depth of findings, exploration of deviant cases and alternative explanations and reflexivity of the researchers; (5) We will use the Critical Appraisal Skills Programme (CASP) tool for assessing methodological strengths and limitations of included studies.39 This is the tool most used by systematic reviews focusing on qualitative evidence synthesis.38 As per recommendations from the ‘Cochrane Qualitative and Implementation Methods Group Guidance Series’, we may add other tools if we deem that a specific type of study might be at a disadvantage if we use only CASP; (6) We will classify and group interventions using the 10-dimension Complexity Assessment Tool for Systematic Reviews (iCAT-SR).40
Strategy for data synthesis
We will use a narrative synthesis to describe: (1) The theory (if any) adopted by the research teams when conducting the process evaluations; (2) The study designs used during the process evaluation alongside RCTs; (3) The phase in which process evaluation was performed; (4) The way results of the trials are being integrated with findings from the respective process evaluations; (5) The barriers and facilitators faced by the authors while conducting process evaluations; (6) Strengths and limitations of the process evaluation methods as reported by the study authors.
We will use narrative summaries of the individual studies and shared themes to synthesise the findings.
Analysis of subgroups or subsets
Depending on the number of articles included, we will conduct a subgroup data analysis based on: context (eg, indigenous, non-indigenous participants; under-developed and developed countries, healthcare systems) or the category of interventions (as categorised by iCAT-SR) for describing barriers and facilitators, fidelity and adherence to implementation of the planned intervention.33 40
Process evaluation studies can help improve translation of research into clinical practice. The information gathered by process evaluation studies is valuable for healthcare professionals, policy makers and researchers. Such evidence can inform whether findings from a small trial should be scaled up or whether findings from a trial need to be modified and adapted to another context.
This review will contribute to the field by identifying methods used for assessing process evaluation of clinical trials assessing the effectiveness of interventions for musculoskeletal disorders. There are no definitive methods or guidelines for conducting process evaluation studies.12 35 This is due in part to the fact that the term ‘process evaluation’ includes different domains: implementation of interventions, the mechanisms of action of interventions and the impact of context factors (ie, how context influences clinical outcomes or is influenced by an intervention). To address each of these three domains, different research methods are required. Findings from this review will identify current practices adopted by musculoskeletal researchers when conducting, analysing and reporting process evaluations studies. Our findings will identify gaps in the literature and inform future research conducted in the area of musculoskeletal disorders and rehabilitation.
This protocol has limitations. We will only include studies that explicitly state process evaluation of an intervention was assessed or that include outcome measures that allow researchers to assess the process evaluation of an intervention (eg, fidelity, or adherence to an intervention). It will not be feasible to screen the full text of all published trials on musculoskeletal disorders. We may not identify other process or outcome evaluation studies that assessed process evaluation but did not explicitly report it in the title or abstract. The advantage of our approach is that it identifies current practices using studies in the broad area of musculoskeletal disorders.
To our knowledge, this will be the first systematic review to assess how process evaluations are currently being conducted in RCTs of non-surgical and non-pharmacological interventions in the management of musculoskeletal disorders. This review will describe current practices on process evaluation of clinical trials and inform future research that will be conducted in this area. Recently, there has been increased encouragement to conduct process evaluation studies to better inform implementation of findings from clinical trials in clinical practice and policy-making. It is reasonable to expect that this review will yield a diversity of methods used by different research groups. Hence, the importance of this review is in identifying best practices for future process evaluation studies tested in RCTs in musculoskeletal disorders.
Contributors DCR is the leading researcher and was responsible for conceiving this study, designing the protocol, preparing the search strategy and co-authoring the first draft of this manuscript. SS contributed to the design of the review protocol, preparing the search strategy and co-authoring the first draft of this manuscript. JHA and SEL contributed to the design of the review protocol and to the manuscript. DCR (principal investigator), JHA (mentor) and SEL (mentor) secured the Sir Charles Hercus Health Research Fellowship of the Health Research Council of New Zealand [Grant number: 18/111]. All authors have contributed to the drafting of this protocol and accepted the final version of the manuscript for submission.
Funding The research was conducted during tenure of The Sir Charles Hercus Health Research Fellowship of the Health Research Council of New Zealand [Grant number: 18/111]. The Health Research Council – New Zealand had no role in the design of the review and will not have any role in its execution, data analysis and interpretation or on the submission of the review for publication.
Competing interests None declared.
Ethics approval The systematic review will be submitted for publication to a peer reviewed journal and the findings will be presented at a relevant conference and research seminars. We will follow the recommendations from the International Committee of Medical Journal Editors for authorship eligibility. Any important protocol amendments will be registered at PROSPERO and described in the systematic review report.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement This is a protocol and all data is available on the protocol reporting.
Patient consent for publication Not required.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.