Introduction The validity of systematic reviews and meta-analysis depends on methodological quality and unbiased dissemination of trials. Our objective is to evaluate the association of estimates of treatment effects with different bias-related study characteristics in meta-analyses of interventions used for treating pain in osteoarthritis (OA). From the findings, we hope to consolidate guidance on interpreting OA trials in systematic reviews based on empirical evidence from Cochrane reviews.
Methods and analysis Only systematic reviews that compare experimental interventions with sham, placebo or no intervention control will be considered eligible. Bias will be assessed with the risk of bias tool, used according to the Cochrane Collaboration’s recommendations. Furthermore, center status, trial size and funding will be assessed. The primary outcome (pain) will be abstracted from the first appearing forest plot for overall pain in the Cochrane review. Treatment effect sizes will be expressed as standardised mean differences (SMDs), where the difference in mean values available from the forest plots is divided by the pooled SD. To empirically assess the risk of bias in treatment benefits, we will perform stratified analyses of the trials from the included meta-analyses and assess the interaction between trial characteristics and treatment effect. A relevant study-level covariate is defined as one that decreases the between-study variance (τ2, estimated as Tau-squared) as a consequence of inclusion in the mixed effects statistical model.
Ethics and dissemination Meta-analyses and randomised controlled trials provide the most reliable basis for treatment of patients with OA, but the actual impact of bias is unclear. This study will systematically examine the methodological quality in OA Cochrane reviews and explore the effect estimates behind possible bias. Since our study does not collect primary data, no formal ethical assessment and informed consent are required.
Trial registration number PROSPERO (CRD42013006924).
- risk of bias
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
The Cochrane Musculoskeletal Group (CMSG) is one of the 53 existing review groups in the Cochrane Collaboration, which aims to prepare, maintain and disseminate high-quality systematic reviews for musculoskeletal diseases including osteoarthritis; and help healthcare providers, patients and carers to make well-informed decisions on prevention, treatment and management of musculoskeletal conditions.1 ,2 (http://musculoskeletal.cochrane.org/more-about-us 29 August 2013). The Cochrane Collaboration makes a considerable contribution to reduce bias in scientific interpretation of research.3–5 Systematic reviews founded on randomised trials provide the most reliable evidence about the effects of healthcare interventions.6 Unfortunately, inadequate methodology may distort the outcomes from systematic reviews and metaanalyses7 and produce misleading results.8
Description of the problem or issue
Bias in trials can lead to underestimation or overestimation of the true intervention effect.9 Regardless of the tools used to assess risk of bias, the methods for assessing and summarising potential bias and incorporating bias assessments into meta-analyses vary greatly.3 ,4 Bias associated with particular characteristics of studies may be examined using meta-epidemiology, which analyses a cluster of meta-analyses where the influence of the trial characteristics—such as judgements on risk of bias on treatment effects estimates—is explored.8 By using meta-epidemiological studies, it is possible to examine the association of specific trial characteristics in a collection of meta-analyses and their included trials.10 ,11
Most meta-epidemiological studies focus on allocation concealment and blinding and their influence on treatment effects of dichotomous outcomes,10 ,12 but this study will focus on other characteristics as well: attrition bias, the issue of center status and funding. Further, this study will focus on pain—a continuous patient reported outcome (PRO). Non-drug intervention trials researching treatment in osteoarthritis are difficult to blind; the patients might know whether they are receiving the intervention or control, which could affect their (self-reported) pain scores.
Description of the methods being investigated
Much research has been conducted to understand bias in trials and how it may influence the results of systematic reviews.5 In 2001, the Cochrane Bias Methods Group was established to investigate how bias influences primary studies. Through its acknowledged work, the group developed the Cochrane risk of bias tool in 2008, based on the methodological contributions of meta-epidemiological studies. This tool has since been updated and is a mandatory component of Cochrane reviews.4 There remain additional possible contributors to bias which are not currently standard components for the Cochrane risk of Bias tool.
At the Cochrane Methods Symposium in Quebec City, Canada, October 2013, L Bero and J Sterne—among others—contributed in a debate on whether or not to include funding as a standard domain in the Cochrane risk of bias tool, or if the bias related to source of funding could be captured as part of various ‘reporting biases,’ as eligible studies tend to remain unpublished for reasons related to their ‘negative’ results.13 ,14
Studies have suggested that the number of study sites might have an influence on effect size.15 Single-centre trials tend to include more homogeneous populations within the (highly selected) study eligibility criteria, and they are more likely than multicentre trials to perform well in experienced teams of providers with expertise in the specific technical or other aspects of an intervention.15 However, single-centre trials risk overestimating effect sizes.15
Trial size also is known for being inversely associated with bigger effect estimates.16 This phenomenon could be a reflection of smaller trials’ being prone to reporting bias: studies with statistically significant results tend to be published more frequently than those with non-significant results.17 However, the larger the trial, the greater the probability that results are published.18 Furthermore, smaller trials might tend to be more susceptible to overall bias assessed by the Cochrane risk-of-bias tool.18
Why this review is needed
Bias domains from the Cochrane risk of bias tool (selection bias, performance bias, detection bias, attrition bias and reporting bias), and further centre status, trials size and source of funding risk positively influencing an experimental treatment if they are found inadequate or unclear.11 ,12 ,16 ,18–20 A meta-epidemiological study assessing binary outcomes showed that ORs from trials with unclear or inadequate concealment were on average 30% lower (ie, more beneficial to the intervention) than ORs from trials with adequate methodology (combined ratio of ORs=0.70, 95% CI 0.62 to 0.80).7 Results for binary outcomes may not be extrapolated to trials assessing continuous outcomes because such trials usually differ in medical condition, risk of bias, sample size and statistical analysis.21
In spite of this knowledge, systematic reviews and meta-analyses still provide summary data to policymakers, healthcare professionals, patients and their relatives to make decisions about the patients’ treatment and well-being. It is interesting to explore whether these systematic reviews and meta-analyses provide the true effect estimates, and if all these groups are making decisions on the basis of spurious (biased) effect estimates. The validity of systematic reviews and meta-analysis is depending on methodological quality and unbiased dissemination of trials.22 The association between methodological components and the overestimated treatment effects will be determined in this meta-epidemiological study, which will enable us to provide guidance on how to interpret OA trials in systematic reviews based on empirical evidence from Cochrane reviews.
Protocol and registration
Our protocol is registered on PROSPERO (CRD42013006924); our protocol manuscript conforms to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for reporting systematic reviews and meta-analyses.25
Only systematic reviews of randomised or controlled trials available will be used in this analysis. Only systematic reviews that compare experimental interventions with sham, placebo or no intervention control in patients with OA and which use a patient-reported pain measure as an outcome will be considered eligible. Moreover, only trials where patients in both the intervention and the control group receive the same treatment except from an add-on in the intervention group will be eligible for inclusion. Reviews will be excluded if no relevance is found when reading the title and abstract. Further exclusion will be made if the review did not report a meta-analysis on pain—preferably a meta-analysis showing overall pain, including at least two trials. Two reviewers will independently evaluate the reports for eligibility, and any disagreements will be resolved by discussion or by involvement of a third reviewer. Data will be eligible from the trials, which are included in eligible reviews and meta-analyses.
Search for meta-analyses and inclusion of trials
We will search the Cochrane Database of Systematic Reviews via the Cochrane Library for osteoarthritis reviews using a combination of MeSH terms and natural language vocabulary (see table 1). The latest update of the Cochrane review will be used. Eligible trials will be identified from the reference lists of the published Cochrane reviews.
Risk of bias in individual studies
Cochrane risk of bias tool
The risk of bias within each full-text trial will be assessed using the domains of the risk of bias tool as recommended by the Cochrane Collaboration,26 which comprise methods for sequence generation and maintaining allocation concealment, blinding and management of incomplete outcome data. Each domain will be rated as adequate, inadequate or unclear risk of bias (see table 2).
Data collection process and data items
We will use a systematic, standardised data extraction approach to gather information from all eligible OA studies. The primary outcome (pain) will be abstracted from overall pain reported in the Cochrane review’s first pain measure forest plot. Outcome will be collected at the time point closest to 12 weeks’ follow-up. Two reviewers will independently extract data from the trials.
We anticipate that pain will be measured in various ways, and thus different instruments will be applied to monitor the changes during any given trial period.28 Commonly pain is measured by either a visual analogue scale (VAS) or a numeric rating scale (NRS) and less commonly by more complex item-based instruments (eg, WOMAC29 KOOS30 and ICOAP31).
‘Systematic review-level’ data extraction
At the level of the systematic review, we will extract data on review ID, author, year of publication and accumulated trial size combined in the meta-analysis (ie, NTotal, NI and NC). Type of intervention will be categorised according to the suggested ‘sequential, pyramid approach to management of OA’ by Dieppe and Lohmander32 and the Osteoarthritis Research Society International (OARSI) recommendations33 (see table 3, OA intervention categories).
‘Trial-level’ data extraction
At the trial level, we will assign studies a trial ID and extract information on author name, year of publication, type of pain measure, type of intervention and type of OA condition. From the forest plot available in the included review, we will extract the following study-level covariates: mean values (mI and mC), SDs (SDI and SDC) and size of the trials.
The size of the study will be handled in two ways: (1) trial size according to the meta-analysis (ie, forest plot, nI and nC), and (2) trial size according to the original (ITT) population according to the trial report (NI and NC). Further, we will assess the following characteristics from the trials: attrition rate during the trial (aTotal, aI and aC), and trial duration (weeks). Further, we will extract information on the comparator groups applied in each trial, whether placebo/sham or waiting-list/nothing.
Treatment effect sizes will be expressed as SMD by dividing the difference in mean values available from the forest plots by the SD. Negative effect sizes will indicate a beneficial effect of the experimental intervention (ie, pain reduction). We will register whether the data are expressed as values from follow-up, change from baseline or a mix, as studies have shown that there is no relevant difference between follow-up and change data SMDs in meta-analysis.35 To explore whether the choice of pain instrument (ie, VAS, NRS or item-based scores) interacts with the apparent treatment effect, we will also combine SMDs derived from each of these instrument categories, in a stratified meta-analysis. This will reveal whether there are any relevant differences in effect sizes depending on the pain-instrument chosen. To address the effects of trial size in addition to the categorisation of small and large trials we will do funnel plots based on sample sizes. If small study effects are present, funnel plots will be asymmetrical.
To empirically assess the risk of bias in the treatment effects in reviews, we will perform stratified analyses across trials according to all the different extracted trial characteristics and derive p values for interaction between trial characteristics and treatment effect. Within each systematic review, the ES for trial effects according to the different risk of bias assessments will be estimated by using a random effect meta-analysis model. The differences between the pooled estimates will be derived and combined using random effect meta-analysis fully allowing for heterogeneity, to measure the variability in bias estimates, expressed in τ2. A relevant study-level covariate is defined as one that decreases the between-study variance (τ2, estimated as Tau-squared) as a consequence of inclusion in the mixed-effects statistical model. All the statistical meta-regression analyses will be based on REstricted Maximum Likelihood (REML; ie, random-effects) models36; models will be developed and analysed using SAS software (V.9.2, by SAS Institute Inc, Cary, North Carolina, USA).
Ethics and dissemination
We believe that the findings of this meta-epidemiological study will have important implications for future research strategies and implementation of study conclusions. Meta-analyses and randomised controlled trials provide the most reliable basis for treating patients with OA. However, even though the influence of bias items is well known and described, authors tend to forget that biased trials included in systematic reviews will cause meta-analyses to be biased.37
Therefore, it is essential that those who conduct systematic reviews be familiar with the potential biases within primary studies and how such biases could influence review results and the ensuing conclusions. Still, many Cochrane review authors are reluctant or fail to incorporate the risk of bias assessment in their analysis and conclusions,38 compromising the trustworthiness of results from meta-analysis derived from these reviews. Although the Cochrane Collaboration has changed the way it assesses bias in included trials5 owing to Ken Schulz's work on bias assessment,23 the Cochrane risk of bias tool in 2008,4 AMSTAR (Assessing the methodological quality of systematic reviews checklist),39 and the grading of recommendations assessment, development and evaluation (GRADE) approach,40 meta-analyses might still overestimate the effect estimates from biased studies.
As previously described by Nüesch et al16 the presence and extent of small study effects in OA research might distort the overall evidence from meta-analyses. Thus the influence of small trials on estimated treatment effects might be considered a novel risk of bias domain that needs to be addressed in the future. However, one of the reasons why small studies may give different answers from larger studies is that the participants in small studies may be more homogeneous than those in larger studies. Thus changes with treatment may be more similar, and if they are a more severely affected group the potential for gain may be larger. This work will facilitate consensus on the need for mandatory sensitivity analyses before making conclusions of a meta-analysis if the overall result is not consistent with those of the largest trials.18
After two decades of assessing risk of bias, it is clearly time we examine whether the tool for assessing risk of bias is complete38 What items should be added or removed? Which items have the greatest impact on estimates of effect? This study will examine the evidence in CMSG reviews using patient reported outcomes from OA trials as an example and explore the true implications on effect estimates behind the possible shades of bias. As this study collects no primary data, no additional formal ethical assessment and informed consent are required.
The authors thank the Parker Institute, Frederiksberg Hospital, and The Oak Foundation for creating the possibility and right settings for this research. Additionally, the authors thank the Cochrane Musculoskeletal Group Editorial based at The University of Ottawa, Centre for Global Health.
Contributors JBH, RC, CBJ, PT and HL conceived and designed the study. JBH, RC and CBJ contributed to the development of the protocol. All of the authors (JBH, CBJ, IB, PT, EG, JPP, TR, GAW, AM, LM, HL and RC) assisted in the final protocol and agreed to its final approval before submission.
Funding This research was funded by The Oak Foundation. The Cochrane Musculoskeletal Group is grateful for the financial support of the Canadian Institutes of Health Research (CIHR), the University of Ottawa, the National Health and Medical Research Council in Australia (NHMRC), and the Cabrini Institute, Melbourne, Australia.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.