Article Text
PDF
Abstract
Objectives Physical rehabilitation is a complex process, and trials of rehabilitation interventions are increasing in number but often report null results. This study aimed to establish treatment success rates in physical rehabilitation trials funded by the National Institute of Health Research Health Technology Assessment (NIHR HTA) programme and examine any relationship between treatment success and the quality of intervention development work undertaken.
Design This is a mixed methods study.
Setting This study was conducted in the UK.
Methods The NIHR HTA portfolio was searched for all completed definitive randomised controlled trials of physical rehabilitation interventions from inception to July 2016. Treatment success was categorised according to criteria developed by Djulbegovic and colleagues. Detailed textual data regarding any intervention development work were extracted from trial reports and supporting publications and informed the development of quality ratings. Mixed methods integrative analysis was undertaken to explore the relationship between quantitative and qualitative data using joint displays.
Results Fifteen trials were included in the review. Five reported a definitive finding, four of which were in favour of the ‘new’ intervention. Eight trials reported a true negative (no difference) outcome. Integrative analysis indicated those with lower quality intervention development work were less likely to report treatment success.
Conclusions Despite much effort and funding, most physical rehabilitation trials report equivocal findings. Greater focus on high quality intervention development may reduce the likelihood of a null result in the definitive trial, alongside high quality trial methods and conduct.
- Rehabilitation
- randomised controlled trials
- intervention development
- mixed methods
This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
To our knowledge, this study is the first to use mixed methods integrative analyses to explore the relationship between quality of intervention development work and treatment success.
Using the National Institute of Health Research Health Technology Assessment Journal monographs, published protocols and other supporting publications for each study together provided a detailed and rich source of data beyond what would be found in a single traditional journal publication.
The study reviewed randomised controlled trials of physical rehabilitation from a single UK funder as an exemplar and therefore findings may not be representative of other complex interventions or other funding bodies.
Background
Rehabilitation is ‘a set of interventions designed to optimise function and reduce disability in individuals with health conditions in interaction with their environment’1 and is an essential aspect of healthcare provision. By its very nature, rehabilitation in clinical practice is an individually focused, complex activity, involving interventions that are multifaceted and often implicit in nature,2 and as such, historically, this has been viewed as a barrier to undertaking research.3 This said, there is a growing body of randomised controlled trials (RCTs) of rehabilitation, suggesting that these challenges can be overcome.4 This may, in part, be supported by the publication of the Medical Research Council (MRC) Framework for developing and evaluating complex interventions.5 6
The MRC framework was developed to optimise the likelihood that new interventions are not rejected as being ineffective when inadequate effort has been made in the development of the intervention.7 Likewise, Chalmers and Glasziou8 highlighted the importance of avoiding research waste and recommended that sufficient effort is made to ensure the relevant research questions are identified and addressed using high quality research methods. However, there appears to have been no formal evaluation of the impact of using the development component of the framework on trial outcomes and whether we are observing evidence of effective interventions being developed.
Previous UK9 and US10 reviews synthesised successful and non-successful treatment outcomes from trials of new interventions in order to assess the equipoise principle and to understand what return has been achieved on the investment made by those taking part in the trials, researchers and funders. Dent and Raftery9 reported 24% (20/85) primary outcome comparisons as having a positive result, of which 19% (16/85) were in favour of the new intervention, with 22% (19/85) comparisons reporting a true negative outcome. However, these authors did not focus on rehabilitation interventions, nor did they seek to understand factors that may impact on treatment success, such as the quality or intensity of intervention development pretrial procedures. Informal discussions with colleagues in the UK and internationally noted that an increasing number of publicly funded, large RCTs evaluating physical rehabilitation interventions had reported null findings. Similar concerns have been reported in studies of public health interventions.11 12 Our study, therefore, sought to assess this observation and also explore whether intervention development activities contributed to treatment success using the National Institute of Health Research Health Technology Assessment programme (NIHR HTA) as an exemplar.
We aimed to use data from the NIHR HTA to
Establish the treatment outcomes of funded RCTs of physical rehabilitation.
Establish how many new interventions were found to be effective.
Examine what work had been done in terms of developing the new intervention.
Examine the relationship between (a) and (c).
We adopted a mixed methods approach to address the study aims. Although evidence of using integrative mixed methods approaches in synthesising evidence on complex interventions is limited, mixing together qualitative and quantitative data can generate understanding that has the potential to be greater than the sum of the individual parts.13
Methods
Design
We undertook a review of NIHR HTA funded RCTs of physical rehabilitation interventions using narrative synthesis of outcomes and mixed methods analysis of the relationship between intervention development and categorical treatment outcomes using joint displays.
Patient and public involvement
Patients and the public were not involved in this study.
Data sources and inclusion criteria
We included superiority RCTs of physical rehabilitation funded by the NIHR HTA programme. The interventions could be delivered by a single profession or be multiprofessional. The NIHR HTA programme is the leading public funding source for RCTs in the UK and trials of rehabilitation are increasingly part of the portfolio. We only included completed RCTs whose main trial findings were reported in an HTA monograph or peer-reviewed publication in order to establish treatment success. We excluded pilot and feasibility RCTs as they do not aim to assess the efficacy or effectiveness of interventions14; studies where the interventions were primarily psychological or cognitive as the focus of the study was physical rehabilitation; where the primary outcome findings were not reported with a 95% CI as these data were required to assess treatment success.
Search and screening
We searched the HTA Project Portfolio (since superseded by the NIHR Journals Library https://www.journalslibrary.nihr.ac.uk/#/) from inception to July 2016 using the following keywords: physiotherap*OR occupational therap* OR speech and language therap* OR rehabilitation. We removed duplicate, and then titles and scientific abstracts were reviewed for potential inclusion by one person and checked by a second. Subsequently full text reports were screened for inclusion by one person and checked by a second. Any disagreements were discussed and agreed with a third person.
Data extraction
All data were extracted by one person and checked by a second. Discrepancies were discussed and resolved with a third person.
Quantitative trial data
Data extracted from each trial publication included trial design, target population, health categories (using the Health Research Classification System), primary outcome(s) and time point, minimal important clinical difference or percentage change that the trial aimed to detect, planned and achieved sample size, and primary outcome results with 95% CI. We also recorded the professional background of the chief investigator and amount of funding awarded.
Qualitative intervention development data
Using the revised version of Criteria for Reporting the Development and Evaluation of Complex Interventions (CReDECI 2)15 and the Template for Intervention Description and Replication checklist (TIDieR)16 as frameworks, we extracted all available documentary (qualitative) data from the body of the text regarding intervention development, including descriptions of underlying theory, intervention components and reasons for selection, intended interactions between components, contextual considerations, piloting of intervention and impact of definitive intervention to be evaluated, control components, planned intervention delivery and materials. Where additional supporting publications were cited, such as a protocol or intervention development studies, we used these as additional sources of documentary data.
Data analysis
We used summary statistics to describe the characteristics of the included studies. We categorised primary outcome findings into one of six treatment outcome categories as described by Djulbegovic and colleagues,10 these being (1) statistically significant in favour of the new treatment, (2) statistically significant in favour of the control treatment, (3) true negative, (4) truly inconclusive, (5) inconclusive in favour of new treatment or (6) inconclusive in favour of the control treatment. This was achieved by comparing the 95% CI for the difference in primary outcome to the difference specified in the sample size calculation.9 If the 95% CI excluded a meaningful difference in either direction, implying the treatments have similar effects, the results were categorised as true negative. If the 95% CI included a meaningful difference in either direction (ie, trial failed to answer the primary question), the results were categorised as being truly inconclusive.
Where a single primary outcome and primary time point were not explicitly identified, we used the following hierarchy to determine which primary outcome would be used in the analysis:
Explicitly defined primary outcome.
Outcome used in power calculation.
Main outcome stated in trial objectives.
First outcome reported in sample size calculation.
If a primary time point was not reported, we used the first follow-up time point as this is when we would expect the intervention to have had the greatest effect.
Our preliminary analysis of the qualitative documentary data involved the reading and re-reading of source documents and the extracted descriptions to consolidate our understanding of the development work undertaken in each study. Using a reflective and iterative process, we undertook thematic analysis to distill, structure and make sense of intervention development activity by coding and organising data into themes and subthemes. Each theme and subtheme provided a coherent description of the development work undertaken for each study, which were then synthesised into short descriptors to allow us to produce summary tables. The summary tables comprised a row for each study with columns for each theme and, where relevant, each subtheme. A second researcher checked, discussed and refined descriptors to ensure accuracy. From these descriptions, we then developed descriptive ratings on the quality of the intervention development. Depending on the nature of the data, ratings were categorised and the iterative process involved two researchers refining and checking ratings to ensure they reflected the summary data from each study. In order to provide a visual representation of the quality of intervention development work, these ratings were then converted to a quality coding to indicate high quality, some or unclear quality or limited quality. For example, under co-design the highest quality rating was given when the intervention was co-designed with both clinical and service user input, a middle rating when either clinicians or service users were involved, and the lowest quality rating when neither clinicians nor service users were involved.
To examine the relationship between intervention development and treatment success, we applied mixed methods analytical techniques in novel ways. For each study, we combined ratings derived from the qualitative data on intervention development with the quantitative data on treatment outcomes in a joint display.
Results
We included 15 RCTs (figure 1),17–31 of which 13 used a two-arm, parallel RCT design, one was a two-arm cluster RCT and one was a four-arm factorial design (of which only two arms related to physical rehabilitation). Table 1 provides a summary of the population, intervention, control and outcomes for each study. The combined sample size required to demonstrate a true difference in primary outcomes (excluding any inflation to account for loss to follow-up) was 7548 participants. The total number of participants who provided primary outcome data was higher than this (n=7834), likely due to lower loss to follow-up that estimated, although three studies19 29 30 were considerably below their target sample size at the primary time point. Five primary outcomes were symptom-based or clinical outcomes, seven were functional measures, two were combined measures and one assessed quality of life. Primary time points varied from immediately postintervention to 1 year (median 6 months). The health categories were stroke (n=4), neurological conditions (n=2), inflammatory/immune system disorders (n=2), respiratory (n=1), musculoskeletal (n=1), cardiovascular (n=1), mental health (n=1), accident/injuries (n=1), renal/urogenital (n=1) and other (n=1). Seven interventions were delivered by physiotherapists, one by occupational therapists, one by speech and language therapists, one by nurses, two could be delivered by either a physiotherapist or a nurse, two could be delivered by a physiotherapist or an occupational therapist and one was delivered by both a physiotherapist and an occupational therapist. The chief investigators leading the studies were physicians (n=7), physiotherapists (n=5), occupational therapists (n=1), psychologists (n=1) and methodologists (n=1). The total amount of research funding awarded was £12 515 823.
Study selection. HTA, Health Technology Assessment; RCT, randomised controlled trial.
Summary of included studies
One-third of studies (5/15) reported a definitive finding in favour of one of the treatment arms—four studies in favour of the new treatment, one in favour of the control. Of those with negative results, eight studies reported a true negative (no difference) outcome, one was inconclusive in favour of the new treatment and one inconclusive in favour of the control treatment (figures 2 and 3).
Classification of primary outcome.
Treatment success of included trials based on 95% CI and minimum clinically important difference from sample size calculation (d).
Qualitative data informed 2 themes and 10 subthemes which enabled us to develop data-driven quality ratings:
Preparatory work (need for the study, underpinning theory for the intervention, co-design, context considerations and intervention piloting).
Intervention and control (intervention content and dose, individual tailoring, adherence strategies, standardised training, control content and dose).
Table 2 provides examples of summary data underpinning each rating, with table 3 describing the quality rating for each study in chronological order. Table 4 presents the integrative qualitative and quantitative analysis using a joint display. No single study was deemed to be high quality in each subtheme. This said, the two best rated studies reported only expert clinical input into co-designing the intervention with a lack of clear patient and public involvement; however, they reported a definitive trial outcome in favour of the new intervention. There does not appear to be a single aspect of intervention development driving study outcomes. This said, those with lower quality development work appear more likely to show no difference in outcomes compared with those with higher quality development work. Some areas of intervention development appear to be improving with time, these being articulating a clear need and theoretical underpinning, co-design, piloting and descriptions of intervention and control components.
Description of themes, subthemes and quality ratings with examples
Quality of intervention development work ordered by year of publication
Joint display of treatment success ordered by quality of intervention development work
Discussion
Physical rehabilitation research targets a broad population, although we found that studies for people with stroke to be the most common (n=4). We established that only one-third (5/15) of the RCTs of physical rehabilitation funded by the NIHR HTA programme successfully demonstrated a statistically significant effect for one of the randomised groups in each trial. Four (27%) trials found an effect in favour of the ‘new’ intervention. Although we would not expect all studies to demonstrate effectiveness in favour of the ‘new’ intervention, the equipoise principle implies that there would be no difference between the proportion of studies favouring intervention or control.9 However, this does not account for a null outcome. We were able to use contemporary research methods to develop an assessment of the quality of development work and assessed the included trials to be of varied quality in terms of intervention development work. In general, we found that comprehensive intervention development may have a positive relationship with treatment success. Two studies23 31 with high quality intervention development reported treatment success, although two older25 28 and possibly less well reported trials also reported effective interventions. Developments in complex intervention evaluation,5 reporting standards16 32 and involving patients and the public in research33 have occurred since the inception of the HTA programme, and as such, some development work may have been undertaken but not reported in the older studies. A recent overview of approaches to developing interventions noted the absence of patient and public involvement.34 In addition, there was limited evidence of piloting the intervention prior to proceeding to the full trial with only four studies reporting this having been done. Most (>80%) drug intervention development studies fail to reach the ‘phase III’ trial stage.35 Public health interventions have tended to go straight to an RCT without piloting, which may contribute to challenges in demonstrating effectiveness.11 There are, of course, other factors that influence trial findings, including trial methods and conduct; however, our question was specifically determined to explore what, if any, relationship existed between intervention development and outcomes and not in the effectiveness of particular interventions.
A strength of our study is the use of integrative mixed methods analysis which has enabled us to explore the relationship between development work and outcome. This rarely used approach in evidence synthesis36 has given us a unique insight that would not have been possible using a quantitative or qualitative analysis alone. A limitation of our work could be the focus on a single UK funding stream which does not necessarily reflect the body of research funded from other sources, and therefore, the quality of intervention development work is not necessarily generalisable. However, the NIHR HTA programme is the single largest funder of RCTs of applied health research in the UK. They publish detailed monographs of their funded studies, along with protocols and other supporting publications that provide a detailed and rich source of data beyond what would normally be available in journal-based peer-reviewed publications alone. We were able to retain the essence and nuances of the qualitative data while developing categorical ratings of quality to help us better explore the relationship between development work and treatment success.
Our findings are similar to those of Dent and Raftery9 in relation to those trials showing a benefit who reported 19% (16/85) of studies found in favour of the new intervention. It has been suggested that a 50% success rate is a good investment for healthcare research37; however, our findings indicate that the studies we reviewed fell well below this. In contrast, we observed a considerably larger proportion of true negative studies (8/15; 53%) compared with 19/85 (22%) reported by Dent and Raftery.9 The difference is even greater when compared with a review of cancer trials in the USA where only 2% of trials found a true negative outcome.10 The reasons for the differences are unclear but could include the pragmatic nature of HTA-funded trials and the relative smaller effect sizes often associated with trials of rehabilitation.38
It has been recently suggested that RCTs should only be undertaken if they are justified both scientifically and ethically by having a clear hypothesis and established uncertainty39 and our findings support that by way of good quality intervention development work. Our findings also align with the elements suggested to be key for developing interventions and reducing research waste by increasing the likelihood of success40 which will form a comprehensive supplement to the development phase of the updated MRC guidance on developing and evaluating interventions due for publication in 2019. The NIHR HTA is publicly funded and by increasing effort and focus on developing rehabilitation and other interventions in the future, researchers and funding bodies could increase the possibility of a definitive trial reporting beneficial findings after much investment of time and public money.
Conclusions
Despite much research effort and funding, only 4 out of 15 evaluations of ‘new’ rehabilitation interventions funded by the NIHR HTA programme were found to be unequivocally effective. Most studies reported no difference in outcome between study arms. We have used mixed methods research to explore the relationship between intervention development work and treatment success and developed a method of assessing the quality of this work, which suggests comprehensive intervention development work may have a positive relationship with treatment success.
Recommendations
As this was an exploratory study, further work should be undertaken to establish the validity of quality assessment of intervention development work. This said, researchers and funding agencies should not undervalue the potential benefit of high quality intervention development work prior to definitive RCTs to reduce the likelihood of a null outcome and improve current rates of treatment success.
Acknowledgments
This work was supported by the National Institute of Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care South West Peninsula (PenCLAHRC). The views expressed are those of the authors and not necessarily the NHS, the NIHR or the Department of Health and Social Care.
References
Footnotes
Contributors VAG: conception and design, data collection, analysis and interpretation, drafting and approving the manuscript; JJH: design, data collection, analysis and interpretation, drafting and approving the manuscript; JAF: data collection, analysis, revising and approving the manuscript; KF: data collection, revising and approving the manuscript; CP: data collection, revising and approving the manuscript; DAR: conception, revising and approving the manuscript.
Funding This study was not funded.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement No data are available.