Article Text
PDF
Abstract
Objective To assess, in a sample of systematic reviews of non-pharmacological interventions, the completeness of intervention reporting, identify the most frequently missing elements, and assess review authors’ use of and beliefs about providing intervention information.
Design Analysis of a random sample of systematic reviews of non-pharmacological stroke interventions; online survey of review authors.
Data sources and study selection The Cochrane Library and PubMed were searched for potentially eligible systematic reviews and a random sample of these assessed for eligibility until 60 (30 Cochrane, 30 non-Cochrane) eligible reviews were identified.
Data collection In each review, the completeness of the intervention description in each eligible trial (n=568) was assessed by 2 independent raters using the Template for Intervention Description and Replication (TIDieR) checklist. All review authors (n=46) were invited to complete a survey.
Results Most reviews were missing intervention information for the majority of items. The most incompletely described items were: modifications, fidelity, materials, procedure and tailoring (missing from all interventions in 97%, 90%, 88%, 83% and 83% of reviews, respectively). Items that scored better, but were still incomplete for the majority of reviews, were: ‘when and how much’ (in 31% of reviews, adequate for all trials; in 57% of reviews, adequate for some trials); intervention mode (in 22% of reviews, adequate for all trials; in 38%, adequate for some trials); and location (in 19% of reviews, adequate for all trials). Of the 33 (71%) authors who responded, 58% reported having further intervention information but not including it, and 70% tried to obtain information.
Conclusions Most focus on intervention reporting has been directed at trials. Poor intervention reporting in stroke systematic reviews is prevalent, compounded by poor trial reporting. Without adequate intervention descriptions, the conduct, usability and interpretation of reviews are restricted and therefore, require action by trialists, systematic reviewers, peer reviewers and editors.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
Analysis of intervention reporting in 568 trials from a random sample of 60 systematic reviews (both Cochrane and non-Cochrane reviews).
Intervention descriptions in each eligible trial, in each review, were assessed by two independent raters using the Template for Intervention Description and Replication (TIDieR) checklist.
A parallel survey of the corresponding author (n=46) of each included review was also conducted.
Response rate to the author survey was good (71%), although there may have been recall bias and responses from authors who were interested in the topic.
Our sample contained only reviews of non-pharmacological and non-surgical stroke interventions, limiting generalisability of the results.
Introduction
Inadequate descriptions of interventions in randomised trials, with crucial details missing from many, have been identified as a problem for both implementation of research and its synthesis. For non-pharmacological interventions, the problem is even greater, with more than half of trials lacking essential information about the interventions.1 A serious consequence of intervention details missing from trial reports is that it hampers clinicians from using the interventions in practice. Yet it is systematic reviews, rather than trials, that many clinicians turn to when looking for evidence. For effective interventions to make their way into practice, clinicians need to know the details of the intervention and be able to answer the question of ‘How do I actually provide the intervention that this review is recommending?’
Exploration of the completeness of reporting of interventions in systematic reviews has received little attention. Among the recommendations of studies into the applicability and usability of systematic reviews, it has been noted that more detailed information about interventions should be included.2 ,3 Lack of usefulness has been identified as a barrier to the uptake of evidence from systematic reviews.4 A preliminary analysis of 25 systematic reviews, sourced from a pool of reviews judged to be of high quality, found that only 12% of reviews had sufficient intervention information to be replicable (as assessed by a single question).5 Detailed analysis of the reporting of interventions in a sample that includes both Cochrane and non-Cochrane systematic reviews has not occurred.
While this issue is problematic for nearly all non-pharmacological interventions, some of the earliest work to identify effective components of complex non-pharmacological interventions and facilitate implementation of systematic review results was done for stroke.6 Rehabilitation is a major part of stroke care,7 yet evidence-practice gaps abound. Stroke rehabilitation is frequently acknowledged as a ‘black box’,8 with limited knowledge about what are the effective components of interventions and their features (eg, optimal intensity, duration and timing).
Uptake of evidence into practice is hampered by these knowledge gaps and a need for rehabilitation to be standardised based on the best evidence has been identified as one of the priorities of the stroke world agenda.9 However, progress towards evidence-based rehabilitation will be hindered if trials and reviews do not clearly describe their interventions. Using a sample of systematic reviews of non-pharmacological stroke interventions, this study aimed to: (1) assess the completeness of reporting of interventions in systematic reviews; (2) identify the most frequently missing elements in descriptions of interventions; and (3) survey review authors about their use of and beliefs about intervention information in systematic reviews.
Methods
Search strategy and selection of reports of trials
To be eligible for inclusion, reviews had to be a systematic review of non-pharmacological and non-surgical interventions which were provided to a stroke population. Non-pharmacological interventions were defined as interventions, including rehabilitation, psychotherapy, behavioural interventions, non-surgical technical procedures and devices, and complementary medicine.1 Reviews which included trials of both non-pharmacological and pharmacological interventions were eligible for inclusion as a review, although only the non-pharmacological interventions were rated. Reviews were not eligible if they were reviews of only pharmacological or surgical interventions, or of diagnostic or screening tests.
We aimed to randomly select 30 Cochrane and 30 non-Cochrane systematic reviews of non-pharmacological and non-surgical stroke interventions. Though we did not do a formal sample size calculation, we were aware that 60 systematic reviews should have at least 500 studies (Cochrane reviews have a median of 8 trials; non-Cochrane, a median of 23).10 Hence, estimates of proportions would have 95% CIs better than ±7% which we felt sufficient for the descriptive purposes of this study. We used a stratified sampling strategy, so that a matched number of Cochrane and non-Cochrane reviews were obtained. Our rationale for this is that Cochrane reviews are high profile and high impact reviews, but account for approximately 10% of all published reviews,11 and in a random sample of reviews, these would be under-represented. A medical librarian experienced in conducting systematic reviews helped develop and run the searches. We first searched the current (Issue 2, 2013) Cochrane Library for all completed reviews published by the Cochrane Stroke Group, which identified 156 Cochrane reviews on stroke. Two people (SE, TCH) independently screened titles and abstracts to identify reviews of non-pharmacological and non-surgical interventions (n=58). We then developed a search strategy that included the MeSH terms used to index the selected Cochrane reviews and searched in PubMed with the strategy: MeSH descriptor[Stroke] explode all trees, with qualifiers of[Complications—CO, Psychology—PX, Rehabilitation—RH]; medline[tiab] OR (systematic[tiab] AND review[tiab]) OR meta-analysis[pt]; and excluding publications which had the Cochrane Database of Systematic Reviews as a source. The search was limited to the most recent 5-year period (August 2008–2013) and produced 244 results.
The article identification numbers from all of the articles in both the Cochrane and non-Cochrane search results were separately exported to Excel. A computer-based random number generator was then used to select 30 reviews from each set of search results. The full text of the selected reviews was examined by two raters (SE, ET) to identify reviews that met the inclusion criteria as described above. This process continued until 30 reviews from each search (total of 60 reviews) were identified. Disagreements about unclear eligibility were resolved by discussion.
Rating of intervention reporting in each systematic review
For each included systematic review, the eligibility of each intervention in each trial was assessed before the intervention reporting was assessed. Interventions were eligible if they were: (1) a non-pharmacological intervention or a complex intervention in which the major focus was a non-pharmacological component; and (2) evaluated in a randomised trial.
The completeness of reporting of each eligible intervention in each systematic review was assessed by two authors (SE, ET) independently using the Template for Intervention Description and Replication (TIDieR) guide and checklist, which was developed to provide authors and peer reviewers with guidance about the essential elements of interventions that should be reported.12 The TIDieR checklist contains 12 items (see table 1). For this study, item 1 (intervention name or phrase) was not scored as this is unlikely to be reported for each intervention in a systematic review. Item 12 (actual/extent of intervention fidelity) was scored as not applicable if item 11 (planning/assessment of intervention fidelity) was scored as inadequate. When trials contained evaluations of more than one eligible intervention (such as by using three groups or a factorial design), the reporting of each intervention was assessed separately. Items missing from the intervention description, or not described in sufficient detail for replication, were considered to be incomplete. For interventions which were assessed as being incompletely reported, we recorded comments about what was missing. If review authors reported that intervention details were described further in other sources (such as websites, online appendices or reference materials), we located and examined these additional sources before completing the rating of the checklist items. The raters completed the assessment of reviews in blocks of 5 and after each block assessment, discussed their ratings. Disagreements in ratings were resolved by discussion between the two assessors, with a third author (TCH) arbitrating where necessary.
Brief description of the Template for Intervention Description and Replication (TIDieR) items that were used to assess intervention reporting
Survey of corresponding review authors
For each systematic review, the corresponding author was emailed and invited to complete an online survey via SurveyMonkey. The survey contained eight questions (a mixture of open-ended, and fixed-choice questions with the option of providing additional comments) and asked authors: whether they had additional intervention information that was not reported in the review and the reason for this; whether they attempted to obtain additional intervention information and if so, methods used to obtain such information; details of additional information obtained; and the role of intervention descriptions in systematic reviews. If no response was received, a reminder email was sent after 2 weeks, and after a further 4 weeks.
Data analysis
Details of ratings and authors’ survey responses were entered into Excel and used to analyse data descriptively. For each systematic review, we synthesised the ratings across the multiple interventions for each TIDieR item as: adequate for all (indicating that the item had been clearly reported for all of the eligible interventions in the review); adequate for some (indicating that the item had been described clearly for some, but not all, of the eligible interventions in the review); or adequate for none (indicating that the item had been not described clearly or not at all for any of the interventions in the review). The broad interventions covered by the eligible reviews (as shown in table 2) were categorised by the two raters (ET, SE) according to the National Stroke Foundation's Stroke Guidelines categories.13 When comparing Cochrane and non-Cochrane reviews, we collapsed the ‘adequate for all’ and ‘adequate for some’ into one category. Authors’ responses to open-ended survey responses were independently coded into thematic clusters by two authors (TCH, SE), with any discrepancies discussed until consensus was reached.
Categories of interventions of the included systematic reviews, the number of eligible studies and interventions, and excluded studies
Results
Our sample of 60 systematic reviews included a total of 568 eligible trials and 589 eligible interventions (mean of 10.1 eligible interventions per Cochrane review, range 0–51; mean of 9.5 eligible interventions per non-Cochrane review, range 6–28). Table 2 shows the broad interventions covered by the reviews, along with the number of eligible interventions. Two Cochrane reviews identified no studies eligible for inclusion and were not able to be analysed further for this study.
Completeness of intervention descriptions in systematic reviews
Most reviews were missing information for the majority of items (figure 1). The most incompletely described items were: modifications, intervention fidelity, materials, procedure and tailoring (missing from all interventions in 97%, 90%, 88%, 83% and 83% of reviews, respectively). Items that scored higher, but still incomplete for the majority of reviews, were: details of ‘when and how much’ of the intervention (in 31% of reviews, it was adequate for all interventions; in 57% of reviews, it was adequate for some interventions); how (ie, the mode) the intervention was provided (in 22% of reviews, it was adequately reported for all interventions; in 38% of reviews, it was adequately reported for some interventions); and where (setting) the intervention was provided (in 19% of reviews, it was adequate for all interventions; and adequate for some interventions in 19% of reviews). Only two reviews, both non-Cochrane reviews, directed readers to other sources for additional intervention information. In both of these reviews, supplemental tables contained additional data, and as part of this, also contained information about item 8 (‘when and how much’ of the intervention).
Percentage of systematic reviews, per checklist item, in which intervention descriptions were ‘adequate for all interventions’, ‘adequate for some’ and ‘adequate for none’ of the interventions included in each review.
The completeness of intervention reporting of Cochrane reviews compared with non-Cochrane reviews is shown in figure 2, where, for each TIDieR item, the mean percentage of systematic reviews in which there were adequate description of all or some of the interventions in the review is presented. Reporting of details of interventions was clearly better (in ≥20% of reviews) in Cochrane reviews for 5 of the 11 items: intervention rationale; procedure; intervention provider; how the intervention was delivered; and where the intervention was delivered. For item 11 (assessment of intervention fidelity or adherence), there was a slightly higher percentage of non-Cochrane reviews (13%), compared with Cochrane reviews (7%), in which some or all of the interventions in the reviews described this item, although this item was only reported in a very small proportion of reviews. For the remaining five items (materials; when and how much; tailoring; modifications; actual/extent of fidelity), there was little difference between the completeness of reporting in Cochrane reviews compared with non-Cochrane reviews.
Mean percentage of systematic reviews, per Template for Intervention Description and Replication (TIDieR) checklist item, in which interventions were described ‘adequately for all’ or ‘adequately for some’ of the interventions in the review, for Cochrane and non-Cochrane reviews.
Survey of review corresponding authors
We contacted the 46 corresponding authors of the 58 reviews with eligible interventions (several authors were corresponding authors for two or more reviews). Of the 33 (71%) authors who responded, 19 (58%) reported that they had further information about the intervention that was not included in the review, while 23 (70%) indicated that they had tried to obtain further information about the intervention as part of the review, using various methods, with variable success. Reasons for not attempting to obtain further intervention information included: not perceived as necessary as part of a review, not needed because interventions were already fully described, or would not subsequently have been able to incorporate the information into the review.
A summary of responses to questions where authors could provide comments and elaborate on these issues is shown in table 3. Some authors acknowledged the importance of completely describing interventions in reviews, with multiple reasons given, such as helping with decisions about study inclusion, appropriateness of data synthesis and subgroup analysis, and facilitating the use of interventions in practice. Some authors commented that reviews typically do not provide sufficient intervention information, while others commented that such information could be useful as part of the review process, but did not need to be included in the review itself. Suggested contributors to the problem were lack of details provided in the original trials and word limits (for both trials and reviews).
Survey questions to and responses from corresponding authors (n=33)
Discussion
We found a high rate of incomplete reporting of intervention items in systematic reviews, with details describing most intervention elements missing from the majority of the reviews. This has important consequences for the conduct, interpretation and use of the reviews. As trials are synthesised into reviews, the problem of intervention details missing from trial publications is compounded. Lack of intervention detail affects more than the usability and interpretation of a completed review. It can also hamper steps in the review process, such as decisions about eligibility, meta-analysis and subgroup analysis. Some review authors were conscious of the problem and attempted to obtain additional information, with variable success.
Strengths of this study include: the use of a comprehensive checklist and two independent raters to assess intervention reporting; the number and variety of reviews included studies and interventions; inclusion of both Cochrane and non-Cochrane reviews; and a good response rate to our author survey. Some of the limitations of the study are that our sample of reviews only contains non-pharmacological and non-surgical interventions, and only stroke interventions and hence, the Cochrane reviews are from only one Cochrane review group. The extent to which our random sample reflects systematic reviews in stroke and the generalisability of these results to other conditions and interventions is unknown. Although the response rate to our author survey was good (71%), authors with an interest in the issue may have been more likely to respond, whereas authors of older reviews may not have responded or been able to recall information about the questions asked, and a small number of the authors had authored more than one of the included reviews. We did not formally measure a single level of agreement between the raters primarily because an iterative process was used with the raters comparing ratings after every five reviews. There were no items which raters consistently disagreed on or were notably more problematic than others.
Analysis of the reporting of the intervention elements in systematic reviews has rarely been studied. Assessment of the intervention reporting in 25 Cochrane reviews of acupuncture found that many reviews lacked intervention details, even though the details were often reported in the trials included in the reviews,14 reinforcing that lack of details in trials is only one contributor to the problem. In this study, intervention elements that were mostly absent from nearly all reviews include details of intervention procedures, materials, tailoring, modifications and fidelity. These are the elements that may be perceived as the most difficult to describe and require the most words. Yet, for intervention replication, they are among the most crucial. Items 3 (materials) and 4 (procedures) have been described as analogous to the ingredients and methods of a recipe12—without these details, the intervention cannot be provided with fidelity. These elements have also been reported as the elements most frequently missing from publications of randomised trials of non-pharmacological interventions;1 thus, it is not surprising that these are also among the most frequently missing elements in reviews. A particular challenge with these elements is the amount of detail required to report them adequately. Elements that are easier and more succinct to report, such as the ‘when and how much’ of the intervention and its mode of delivery, were better reported. In Cochrane reviews, these and other intervention details were typically found in the Characteristics of Included Studies tables, rather than in the main text of the review.
For almost half of the TIDieR items, Cochrane reviews more completely reported intervention details than non-Cochrane reviews and for all but one of the remaining items, there was no difference between the two samples of reviews. Better quality reporting of Cochrane reviews than non-Cochrane reviews has been previously identified in a cross-sectional sample of reviews10 and a sample of stroke rehabilitation systematic reviews,15 although these studies did not examine intervention reporting. Among the reasons suggested for better reporting in Cochrane reviews are the absence of word limits, and detailed guidance and policies on review conduct and reporting.10 A number of the authors surveyed mentioned word limits as a barrier to reporting intervention details, although presumably these were authors of non-Cochrane reviews. For reviews where word limits may prevent detailed reporting, authors are encouraged to provide intervention details in online supplementary materials, which is now an option in the majority of journals.16
However, having no word limit on reviews is clearly not sufficient to ensure complete reporting of intervention details. As lack of awareness among authors of trials about what constitutes an adequate description of an intervention contributes to poor reporting of trials,17 it may also be problematic for review authors as guidance for review authors about how to report interventions is lacking. The most relevant section (7.3.4) of the current version (V.5.1.0) of the Cochrane Handbook (handbook.cochrane.org) does mention that reviews should provide information that is ‘sufficient for replication’, and gives a few examples of what this might include (eg, route of delivery, dose, format, timing). However, this guidance does not appear in the most recent version (V.2.3) of Cochrane's ‘Methodological standards for the conduct of Cochrane Intervention Reviews’, the most relevant mandatory standard (C44) simply advises that “basic characteristics of each study will need to be presented…, including details of participants, interventions and comparators…”.18 Other standards that are relevant but unlikely to be sufficient guidance to facilitate adequate intervention reporting include: a non-mandatory standard (C49) that encourages review authors to contact study authors for details of unpublished information, including ‘details of interventions’; and a mandatory standard (C7) that requires any restrictions (ie, those that are ineligible) on interventions and comparators to be predefined and explained. While these standards continue to undergo refinement, the 2012 version (V.1.1) of these standards provided more details about how interventions should be reported and encouraged authors to report ‘in a way that allows the intervention to be replicated’,19 whereas this has been removed in the 2013 version.18
For non-Cochrane reviews, authors may use the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) statement as reporting guidance;20 yet it also does not provide guidance about how to report interventions and only requests ‘study characteristics’. Review authors may find the newly developed statement that guides intervention reporting for any evaluative study design (TIDieR) useful when considering what intervention elements they need details about while conducting, and particularly when writing, the review. Another tool which may help during the review process is the Oxford Implementation Index21 which aims to assist reviewers to extract and compare implementation data (includes intervention design) across trials.
Poor reporting of intervention details in systematic reviews is part of the broader problem of how to implement the results of systematic reviews. A recent paper22 described three possible approaches to selecting and implementing a specific intervention from a systematic review. Two of these approaches are dependent on sufficient intervention details being able to be obtained and the other, the simplest approach (single-trial-based choice), requires at least some interventions in the review to be replicable and for agreement about what constitutes the ‘best’ intervention. One way that review authors can help review users to consider which interventions might be most appropriate for their circumstance involves providing an ‘intervention options table’.22 This table could summarise the pros and cons of the usable and feasible interventions in the review (and the criteria used to decide this), including elements of each version of the intervention studied and differences between trials’ interventions.
In many reviews, authors commented (typically in the Discussion) that lack of intervention detail made synthesising, interpreting and using the results difficult or that trialists should clearly define each intervention tested. Reflecting this, some authors surveyed reported contacting trial authors for additional intervention information—some with success. Those that were able to obtain additional details described using the information in various parts of the review process, such as deciding on study eligibility and the appropriateness and interpretation of subgroup and meta-analyses. While some review authors viewed intervention details as important in both the conduct and reporting of the review, others did not include detailed intervention information in the review report or think it was necessary. Although many systematic review users are aware of the need to critically appraise them before deciding whether to use them, appraisal tools typically do not consider the adequacy of intervention reporting and how that may have influenced the review's steps and results. Future research that explores whether the completeness of intervention reporting (of both the experimental and comparison interventions) is related to variables, such as the quality of the review, would be valuable.
Most of the focus of poor intervention reporting as a barrier to evidence interpretation and uptake has been directed at the primary trials. This study highlights an even greater problem in systematic reviews, where the poor reporting of interventions in trials is compounded. This reinforces the importance of trialists providing sufficient intervention details. This issue is not unique to intervention details; a recent analysis of selective reporting bias of harms outcomes identified that harms data are incompletely reported in many primary studies and also in subsequent systematic reviews.23 Both trialists and systematic review authors have an important role to play in ensuring that interventions are adequately reported in systematic reviews. Poor reporting in the trials, however, is not the sole contributor to the inadequacies of intervention reporting in systematic reviews. As neither the current PRISMA statement or the Cochrane methodological standards provide guidance about intervention reporting, authors of reviews should consider also using guidance about intervention reporting and/or a structured method of reporting intervention details, and peer reviewers and editors should be cognisant of the adequacy with which interventions are reported in reviews. Having systematic reviews which contain adequate intervention information will assist at multiple stages of the review process as well as the interpretation of review results. Ultimately, this will help clinicians to implement effective interventions into practice.
Acknowledgments
The authors thank the authors of the reviews who responded to the survey and Sarah Thorning (Medical Librarian, Centre for Research in Evidence-Based Practice, Bond University) for her assistance with the searches.
References
Footnotes
Twitter Follow Tammy Hoffmann at @Tammy_Hoffmann
Contributors TCH was primarily responsible for study conception, design, and data analysis and interpretation. All authors had full access to all of the data in the study, and take responsibility for the integrity of the data and the accuracy of the data analysis. SE and ET rated the intervention descriptions. TCH led the writing of the first draft of the manuscript, and all authors contributed to drafting and revising the manuscript.
Funding TCH is supported by a National Health and Medical Research Council of Australia (NHMRC)/Primary Health Care Research Evaluation and Development Career Development Fellowship (number: 1033038) with funding provided by the Australian Department of Health and Ageing. PG is supported by a NHMRC Australia Fellowship (number: 527500).
Competing interests TCH and PG are part of the author team that developed the TIDieR statement. PL is a co-author on some of the included reviews and is Coordinating Editor of the Cochrane Stroke group, and TCH is a member of its Editorial Board.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.