You are here

  • Home
  • Archive
  • Volume 5, Issue 9
  • What is the influence of randomisation sequence generation and allocation concealment on treatment effects of physical therapy trials? A meta-epidemiological study

Article Text

Article menu
XML
Epidemiology
Research
What is the influence of randomisation sequence generation and allocation concealment on treatment effects of physical therapy trials? A meta-epidemiological study
  1. Susan Armijo-Olivo1,2,3,
  2. Humam Saltaji4,
  3. Bruno R da Costa5,
  4. Jorge Fuentes6,7,
  5. Christine Ha3,
  6. Greta G Cummings8
  1. 1CLEAR Outcomes (Connecting Leadership, Education and Research) Research Program, University of Alberta, Edmonton Clinic Health Academy (ECHA), Edmonton, Alberta, Canada
  2. 2Faculty of Rehabilitation Medicine, Department of Physical Therapy, University of Alberta, Edmonton, Alberta, Canada
  3. 3Faculty of Rehabilitation Medicine, Rehabilitation Research Center, University of Alberta, Edmonton, Alberta, Canada
  4. 4Orthodontic Graduate Clinic, School of Dentistry, University of Alberta, Edmonton, Alberta, Canada
  5. 5Universitat Bern, Institute of Primary Health Care, Bern, Switzerland
  6. 6Faculty of Rehabilitation Medicine, University of Alberta, Edmonton, Alberta, Canada
  7. 7Department of Physical Therapy, Catholic University of Maule, Talca, Chile
  8. 8Faculty of Nursing, CLEAR Outcomes (Connecting Leadership Education & Research) Research Program, University of Alberta, Edmonton Clinic Health Academy|University of Alberta, Edmonton, Alberta, Canada
  1. Correspondence to Dr Susan Armijo-Olivo; sla4@ualberta.ca/susanarmijo@gmail.com

Abstract

Objective To determine if adequacy of randomisation and allocation concealment is associated with changes in effect sizes (ES) when comparing physical therapy (PT) trials with and without these methodological characteristics.

Design Meta-epidemiological study.

Participants A random sample of randomised controlled trials (RCTs) included in meta-analyses in the PT discipline were identified.

Intervention Data extraction including assessments of random sequence generation and allocation concealment was conducted independently by two reviewers. To determine the association between sequence generation, and allocation concealment and ES, a two-level analysis was conducted using a meta-meta-analytic approach.

Primary and secondary outcome measures association between random sequence generation and allocation concealment and ES in PT trials.

Results 393 trials included in 43 meta-analyses, analysing 44 622 patients contributed to this study. Adequate random sequence generation and appropriate allocation concealment were accomplished in only 39.7% and 11.5% of PT trials, respectively. Although trials with inappropriate allocation concealment tended to have an overestimate treatment effect when compared with trials with adequate concealment of allocation, the difference was non-statistically significant (ES=0.12; 95% CI −0.06 to 0.30). When pooling our results with those of Nuesch et al, we obtained a pooled statistically significant value (ES=0.14; 95% CI 0.02 to 0.26). There was no difference in ES in trials with appropriate or inappropriate random sequence generation (ES=0.02; 95% CI −0.12 to 0.15).

Conclusions Our results suggest that when evaluating risk of bias of primary RCTs in PT area, systematic reviewers and clinicians implementing research into practice should pay attention to these biases since they could exaggerate treatment effects. Systematic reviewers should perform sensitivity analysis including trials with low risk of bias in these domains as primary analysis and/or in combination with less restrictive analyses. Authors and editors should make sure that allocation concealment and random sequence generation are properly reported in trial reports.

  • allocation concealment
  • sequence generation
  • physical therapy
  • meta-epidemiological
  • risk of bias

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/

https://doi.org/10.1136/bmjopen-2015-008562

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • To the best of our knowledge, this is the first meta-epidemiological study using continuous outcomes in allied health disciplines such as physical therapy (PT).

    • This large meta-epidemiological study, which was based on 43 Cochrane reviews, 393 trials and 44 622 patients makes an important contribution to the field.

    • This study provides with novel evidence regarding the association between adequacy of randomisation and allocation concealment and treatment estimates in PT.

    • We restricted our analysis to Cochrane systematic reviews in PT and results might not be applicable to all Cochrane reviews or other reviews conducted in other areas of research.

    • For determining the association between random sequence generation and allocation concealment, we limited our analysis to trials describing a true control group, or placebo intervention, reducing our statistical power.

    Introduction

    Randomisation and allocation concealment have been extensively investigated in the medical literature as key methodological characteristics of randomised controlled trials (RCTs) in medical research.1–3 Allocation concealment is the most commonly evaluated quality characteristic in reviews of RCTs due to its crucial role as a key marker of internal validity of RCT.4 Moreover, it is firmly established the conventional wisdom that adequate randomisation sequence generation is an essential part of a valid clinical trial.5

    There has been an extensive body of empirical research looking at the influence of random sequence generation and allocation concealment on treatment effect estimates of health RCTs. Several meta-epidemiological studies investigating the association between trial characteristics and treatment effects have found an association between inadequate randomisation and/or allocation concealment and an overestimation of treatment effects.1–3 ,6–9 For example, inadequate random sequence generation can overestimate treatment effects by 12% (Ratio of ORs (ROR) 0.88, 95% CI 0.79 to 0.99).10 Inadequate allocation concealment can overestimate treatment effects on average by 18%.3 ,8 ,11 In addition, Pildal et al12 found that 2/3 of the conclusions from meta-analyses were no longer supported if only trials with adequate allocation concealment were included and that 69% of meta-analyses lost statistical significance when trials with unclear or inadequate allocation concealment were excluded.12 Over-estimates or underestimates of treatment effects can lead to biased or inaccurate results and conclusions in systematic reviews and meta-analyses.3 ,12–15 These factors can ultimately have repercussions on decision-making and quality of patient care since different assessments could lead to different decisions for clinical practice.

    Most of the empirical evidence regarding the relationship between trial components, and specifically random sequence generation and allocation concealment, and treatment effect estimates comes from RCTs in medicine and is based mainly on dichotomous outcomes.3 ,8 ,11 No such studies using continuous outcomes have been conducted in other health areas such as the allied health professions, including physical therapy (PT). Furthermore, although it has been reported that the quality of reporting of PT trials has improved over time,16 a finer analysis of types of random sequence generation and allocation concealment in PT trials is lacking.

    Therefore, it is necessary to determine the extent to which sequence generation and allocation concealment affect treatment effect estimates in PT trials to provide accurate results to the PT clinical community. This information is urgently needed to develop guidelines for designing, conducting and implementing PT trials as well as providing clear benchmarks to assess the quality and/or risk of bias of PT trials in systematic reviews and meta-analyses and ultimately the strength of evidence for decision-making in PT. Therefore, our research questions were: (1) are random sequence generation and allocation concealment adequately used and reported in RCTs of PT; (2) do random sequence generation and allocation concealment have an effect on estimates of treatment in PT trials; and (3) do effects sizes on PT trials differ depending on some characteristics of the meta-analyses analysed such as magnitude of the effect size, meta-analysis heterogeneity, type of outcome (subjective or objective) and whether the meta-analysis involved the musculoskeletal area.

    Method

    Design

    Meta-epidemiological approach.

    Study selection

    A random sample of RCTs included in meta-analyses in the PT discipline were identified by searching the Cochrane Database of Systematic Reviews from January 2005 to 25 May 2011 on PT interventions. The search strategy can be found elsewhere and can be provided on request.17 ,18 Meta-analyses were included if they met the following eligibility criteria: (1) the meta-analysis included at least three RCTs comparing at least two interventions, with at least one of the interventions being part of PT scope of practice according to the World Confederation for Physical Therapy (WCPT);19 and (2) the main outcome or the outcome of the meta-analysis with the largest number of trials conducted in the review was continuous.

    Assessment of random sequence generation and allocation concealment domains

    Assessments of random sequence generation and allocation concealment domains were performed by two independent reviewers following the Cochrane collaboration guidelines.20

    Random sequence generation assessment

    In addition to the general evaluation of adequacy of random sequence generation, different methods of random sequence generation were determined. We grouped these methods into four categories as follows: Category 1 included trials where random sequence generation was unclear or not reported; category 2 included trials that had adequate randomisation (eg, use of a computer software, random number table and minimisation); category 3 included trials using acceptable methods of randomisation, but less efficient than the previous category (eg, drawing lots, envelopes, shuffling cards, throwing a dice); category 4 involved trials using inappropriate methods of sequence generation (eg, date of birth, day of admission, hospital record number). Categorisation into one of these four categories was conducted in duplicate. To facilitate comparison with previous meta-analyses, these four categories were combined to create two main groups: an “adequate sequence generation” group (combining categories 2 and 3) and an “inadequate or unclear sequence generation” group (combining categories 1 and 4).

    Allocation concealment assessment

    We divided the methods of allocation concealment used in the trials into the following categories: Category 1 comprised trials that used any type of central randomisation (eg, a remote telephone service or a central office); category 2 comprised trials that used sequentially numbered, opaque and sealed envelopes; category 3 comprised trials that used sealed envelopes without reporting any further details; and category 4 comprised trials where allocation was clearly not hidden (eg, being based on an open list, odd or even days of the week, participant's birth date or the team on duty at enrolment); and category 5 comprised trials were concealment of allocation was not reported or unclear. Categorisation into one of these five categories was conducted by two independent assessors. To facilitate comparison with previous meta-analyses, these five categories were combined to create two main groups: an “adequate concealment” group (combining categories 1 and 2) and an “inadequate or unclear concealment” group (combining categories 3, 4 and 5).

    Data extraction of treatment estimates and trial characteristics

    Two independent reviewers extracted specific data (eg, type of interventions, type of outcomes (ie, objective, subjective), PT area) for all trials included in the meta-analyses as well as data on means, SDs and sample sizes. The primary outcome chosen for the analysis was the main outcome of interest reported in the review or determined from the meta-analysis that contained the largest number of trials in the review. Details on the reviewers’ panel and training process can be found elsewhere.17 ,18

    Data analysis

    Data on sequence generation and allocation concealment were analysed descriptively based on the categories described previously. In order to determine whether random sequence generation and allocation concealment domains affect treatment effect estimates, a two-level analysis was conducted using a meta-meta-analytic approach with a random-effects model to allow for within and between meta-analyses heterogeneity as suggested by Sterne.21

    The first level analysis (within meta-analysis) was as follows: we derived effect sizes (ES) for each trial by dividing the between-group difference in mean values by the pooled SD.22 A negative ES indicates a beneficial effect of the experimental intervention. If some required data were unavailable, we used approximations as previously described.23 The data from each trial were obtained from the meta-analyses included in our study. We followed the classification used in the Cochrane reviews to classify the treatment arms as the experimental treatment of interest or as the control group. In the case of studies appearing in more than one review, the study was only considered once in the meta-analysis with the fewer number of overall studies. We then calculated two pooled ES for each meta-analysis: one corresponding to the pooled effect size from studies having the characteristic of interest (eg, allocation concealment) and the other for studies that did not (eg, no or unclear allocation concealment). We used standard random-effects meta-analyses to combine ES across trials and calculated the DerSimonian and Laird estimate of the variance to determine heterogeneity between trials.1 ,24 Then, for each meta-analysis, we derived the difference between pooled ES estimates from trials with and without the characteristic of interest (eg, allocation concealment). A negative difference in ES indicates that trials without the characteristic of interest show a more beneficial effect for the experimental group.

    The second level analysis (between meta-analyses) involved pooling the results of the previous analysis to describe the effect of each trial component across all meta-analyses. The ES were also combined at this stage using the DerSimonian and Laird random-effects models25 to allow for between meta-analysis heterogeneity.

    Formal tests of interaction between adequate sequence generation and concealment of allocation and estimated treatment benefits were performed separately for each meta-analysis based on Z scores using the estimated difference in ES between trials with and without adequate sequence and concealment of allocation and the corresponding SE.

    We additionally stratified analyses accompanied by interaction tests according to the prespecified characteristics as reported by Nuesch et al1 as follows: treatment benefit in overall meta-analysis: small (ES greater than −0.5) versus large (ES≤ to −0.5); between-trial heterogeneity in overall meta-analysis (low (τ2<0.06) vs high (τ2≥0.06)), nature of the outcome (subjective or objective) and if the intervention was classified as musculoskeletal or other PT area.

    In order to evaluate the effect of random sequence generation and allocation concealment on treatment effect estimates, we limited the analyses to studies describing a true control group, or placebo intervention as well as studies in which the direction of the expected treatment effect was evident (ie, standard care vs standard care plus active intervention; and active intervention 1 plus active intervention 2 vs active intervention 1).

    Finally, results of our analyses were pooled with results from the meta-epidemiological study performed by Nuesch et al,1 which also investigated the effect of concealment of allocation on treatment effects on osteoarthritis measured on a continuous scale. Stata statistical software V.12 was used to perform these analyses.

    Results

    Selection and characteristics of meta-analyses and RCTs

    The search identified 3901 Cochrane reviews, with 271 reviews potentially relevant to PT. Of these, 68 reviews included a meta-analysis of at least three studies of PT interventions and used a continuous outcome. We randomly selected 44 meta-analyses but excluded one 26 because it used follow-up data from the same group rather than a control group for comparison (figure 1). Forty-three meta-analyses including 393 trials and analysing 44 622 patients contributed to this study. Table 1 summarises the characteristics of the 43 Cochrane reviews. Briefly, the reviews were published between 2008 and 2011 and included meta-analyses of the effectiveness of PT interventions for musculoskeletal (22 reviews),27–35 cardiorespiratory (9 reviews),36–44 neurological (6 reviews)45–51 and other areas of physical therapy (6 reviews).51–56 A median number of six trials were included in the meta-analyses (IQR 5–8). Most trials were parallel group trials (367; 93.4%), single-centre studies (298; 76%) and had active control interventions (362; 92%). The most common intervention was exercise (n=282, 71.8%). Online Supplementary table S1 (appendix S1) lists the characteristics of each of the 43 meta-analyses.

    View this table:
    Table 1

    Characteristics of the selected meta-analysis within physical therapy areas

    Figure 1
    Figure 1

    Diagram for identification of studies.

    Sequence generation descriptive

    From the 393 trials included in the 43 meta-analyses, 156 trials (39.7%) had appropriate sequence generation and 237 (60.3%) had inappropriate sequence generation (229 were classified as ‘unclear’ and 8 as ‘high’ risk of bias) according to the Cochrane Collaboration guidelines.

    When analysing the sequence generation categories, we found that 229 trials (58.27%) did not clearly report the mechanism of how the sequence was generated. One hundred and thirty-two trials (33.6%) generated the sequence through computer software, a table of random numbers or by minimisation method. For a more specific sequence generation methods description, see table 2.

    View this table:
    Table 2

    Methods of sequence generation in physical therapy trials

    In addition, simple randomisation was reported in 34 trials, block randomisation in 54 trials, stratification in 62 trials and unclear or not reported in 218 trials. The rest of the trials reported other methods of random sequence generation.

    Allocation concealment descriptive

    From the 43 meta-analysis and 393 trials, 45 trials (11.5%) had appropriate allocation concealment and 348 (88.6%) had inappropriate concealment of allocation according to the Cochrane Collaboration guidelines. Results of analysis of the allocation concealment within each category were as follows: 282 trials (71.8%) had unclear method of allocation concealment, 21 (5.34%) trials used central allocation, 24 (6.11%) trials used envelopes with all 3 safe wards (opaque, sealed and sequentially numbered), 15 (3.82%) trials used envelopes without safeguards and 51 (16.8%) trials used any other non-concealed methods. Further details on allocation concealment methods are displayed in table 3.

    View this table:
    Table 3

    Methods of allocation concealment in physical therapy trials

    Sequence generation and allocation concealment descriptive

    Only 8.9% of the trials (n=35) had both appropriate sequence generation and appropriate allocation concealment. A great percentage (51%; n=199) of the trials did not have either appropriate sequence generation or appropriate concealment of allocation.

    Sequence generation and treatment effects in PT trials

    For the purpose of analysing the effect of sequence generation on treatment effects, 22 meta-analyses including 257 trials and analysing 30 287 patients contributed to this analysis. Figure 2 shows the forest plot of the differences in ES between trials with adequate and inadequate random sequence generation. There was no statistically significant difference between the ES of trials with adequate or inadequate random sequence generation (ES=0.02; 95% CI −0.12 to 0.15). The results of the stratified analyses are displayed in figure 3. None of the meta-analyses characteristics had a statistically significant interaction.

    Figure 2
    Figure 2

    Forest plot of the differences in effect sizes between trials with and without adequate sequence generation.

    Figure 3
    Figure 3

    Forest plot of the differences in effect sizes between trials with and without sequence generation stratified by meta-analyses characteristics (effect size magnitude, heterogeneity, type outcome and physical therapy area).

    When analysing trials belonging only to category 2 which included trials that had adequate sequence generation (eg, use of a computer software, random number table and minimisation) versus unclear or not reported sequence generation, no statistically significant difference between the ES of these trials was found (ES=0.10; 95% CI −0.04 to 0.23). However, trials with unclear or not reported sequence generation tended to overestimate the treatment effect when compared with trials with adequate sequence generation from category 2. It was not possible to conduct the subgroup analysis between trials of category 2 versus trials from category 4 (using inappropriate methods to perform sequence generation such as date of birth, day of admission, hospital record number) since there were not enough trials for such analysis.

    Allocation concealment and treatment effects in PT trials

    For the purpose of analysing the effect of allocation concealment on treatment effects, 17 meta-analyses including 198 trials and analysing 27 011 patients contributed to this analysis. Figure 4 shows the forest plot of the differences in ES between trial with adequate and inadequate allocation concealment. Although trials with inappropriate allocation concealment tended to have an overestimate treatment effect when compared with trials with adequate concealment of allocation, the difference was non-statistically significant (ES=0.12; 95% CI −0.06 to 0.30). The results of the stratified analyses are displayed in figure 5. None of the meta-analyses characteristics had a statistically significant interaction. When focusing on trials with appropriate allocation concealment (category 1 and 2) versus trials with clearly inappropriate methods of concealment (category 4), the difference was not statistically significant (ES=0.26; 95% CI −0.04 to 0.55). However, trials with clearly inappropriate methods of concealment (category 4) tended to overestimate the treatment effects. The same was the case when comparing the trials with appropriate allocation concealment (category 1 and 2) versus trials with unclear or unreported concealment of allocation (ES=0.20; 95% CI 0.05 to 0.34). This time, the difference between these trials was statistically significant.

    Figure 4
    Figure 4

    Forest plot of the differences in effect sizes between trials with and without adequate concealment of allocation.

    Figure 5
    Figure 5

    Forest plot of the differences in effect sizes between trials with and without adequate concealment of allocation stratified by meta-analyses characteristics (effect size magnitude, heterogeneity, type outcome and physical therapy area).

    Pooling results with a previous meta-epidemiological study

    When pooling our results with those of Nuesch et al,1 who performed a meta-epidemiological study with continuous outcomes as well, we obtained a pooled statistically significant value (ES=0.14; 95% CI 0.02 to 0.26), meaning that trials with inappropriate concealment of allocation had more beneficial effect than trials with appropriate concealment of allocation (figure 6).

    Figure 6
    Figure 6

    Pooled data of the effect of concealment of allocation on treatment effect estimates using continuous outcomes.

    Discussion

    Our findings showed that adequate sequence generation and appropriate allocation concealment were only accomplished in a low percentage of PT trials (39.7% and 11.5%, respectively). In addition only 8.9% of the trials (n=35) had both appropriate sequence generation and appropriate allocation concealment despite the fact that both of the aforementioned risk of bias domains can be always appropriately performed when conducting an RCT in any field.57 In addition, trials with inappropriate concealment of allocation had an overestimation of treatment effects when compared with trials with adequate concealment of allocation. However, no difference in treatment effects in trials with appropriate or inappropriate sequence generation was found. These results confirm previous results obtained by several meta-epidemiological studies investigating the influence of sequence generation and allocation concealment on several areas of medicine using dichotomous outcomes.7 ,8 ,11 ,12 ,58 In addition, the pooled estimate obtained from our study and that of Nuesch et al,1 indicated that trials with inappropriate allocation concealment had a more beneficial effect than those with appropriate allocation concealment.

    These results have important implications for the research community in general as well as for the discipline of PT. To our knowledge; this study is the first of its kind conducted in PT that examined continuous outcomes. Thus, it provides novel evidence in a very specific area of health. Most of the previous studies were looking at medical areas, such as pregnancy and childbirth, circulatory conditions, infectious disease, surgery and mental health trials. All of these medical areas certainly differ from PT with respect to: type of intervention (being mostly drugs), type of outcomes used and specific area of study.

    First, it is not surprising that a large number of trials did not clearly report either randomisation sequence or allocation concealment or both. Our findings showed that approximately 62% and 70% of studies were classified as ‘unclear/not reported’ in their reporting of sequence generation and allocation concealment, respectively. This has also been observed in a meta-epidemiological study investigating meta-analyses before 2005.2 The study found a 64% and 69% of ‘unclear’ reporting of sequence generation and allocation concealment, respectively.2 Thus, despite the efforts made to improve reporting of RCTs such as introducing the CONSORT Statement, there is still inappropriate reporting and implementation of these important characteristics in RCTs. This is in agreement with results reported by Moseley et al,16 who found little or no effect of the CONSORT Statement on the quality of reports of physiotherapy trials. Quality of reporting has always been an issue when trying to understand the association between trial characteristics and treatments effects since most of authors do not properly report their methods. Although inappropriate reporting does not always reflect on weak trial conduct,59 some studies such as Pildal et al,12 found that most trials with unclear allocation concealment had also unclear allocation concealment in their protocol. Therefore, quality of reporting is related to assessment of risk of bias, but the direction of this relationship is still unclear. Future studies should assess how, how much and in which direction trial reporting affects treatment effects.

    The magnitude and direction of the effect found in our study is consistent and almost identical with a previously published study performed in osteoarthritic trials investigating allocation concealment. In our study, the average bias associated with lack of allocation concealment corresponds to ¼ or ½ of a typical treatment effect found in PT interventions.60–63 Furthermore, the overestimation of a treatment effect was even higher when only focusing the analysis to trials with clearly inappropriate methods of allocation concealment. As stated by others,2 ,8 if allocation concealment is not accomplished, it is more likely that the results of the trial are inaccurate because, intentionally or unintentionally, investigators could have interfered with assigning participants to different groups, favouring the effect of the intervention. In addition, allocation concealment could be a proxy measure for other aspects of trial design besides selection bias.2 ,8

    Random sequence generation was found not to be significantly associated with distortion of treatment effects in this study. Thus, it appears that random sequence generation does not have such a strong influence on treatment effects in comparison with concealment of allocation based on the results of the present study. Other meta-epidemiological studies have found similar results. 8 ,11 ,64 One of these studies, performed by Balk et al,64 has been questioned because of the high heterogeneity between trials included in meta-analysis. This could introduce a higher noise to be able to detect any effect of the trials characteristics. The other studies reported the same results argue that sequence generation is still an important characteristic that ensures rigour in the trial by allowing a trial to be concealed.8 ,11 In addition, other authors have suggested that the relationship between certain methodological characteristics and treatment effects varies according to each health area64 ,65 and more work should be carried out in other health areas to confirm these results.64 ,65

    Our stratified analysis for sequence generation and allocation concealment showed that none of the meta-analysis characteristics presented with significant interaction. This is somehow contrary to other meta-epidemiological studies. For example, Wood et al,3 and Savovic et al,2 found that there was an overestimation of treatment effects that ranged between 22% and 31% for subjective outcomes. We followed exactly the same definition for determining outcomes as ‘objective and subjective’ used by Wood et al,3 to facilitate comparisons and avoid misclassifications. Thus, it seems that for PT area, there is no evidence of bias regarding random sequence generation or allocation concealment when subjective or objective outcome are used. This could be attributed to the fact that both domains can be performed in a trial regardless of the outcome analysed. Thus, whether the outcome is objective or subjective does not affect whether it is easier or not to predict patient's prognosis at the time of recruitment.

    Study limitations

    We analysed the influence of random sequence generation and allocation concealment on treatment effect estimates. It could be possible that other biases interact with these biases and have an influence in the treatment effects. This should include a multivariate analysis where several biases should be taken into consideration. Future research should look into this. However, such analysis would require meta-analyses with a large number of included studies, which are extremely rare; this makes such an analysis hard to be performed.

    We limited our analysis to trials describing a true control group (ie, group receiving no treatment, or a waiting list), or placebo intervention as well as those studies in which the direction of expected treatment effect was evident. In this way, we could anticipate the direction of the bias in analyses. Combining effects of all trials without clear direction of what would be the effect (ie, in case of 2 similar active interventions) would have increased noise in the analyses and heterogeneity limiting our ability to find any significant effect of trials characteristics.66 However, limiting our analyses to these trials, we also reduced the power of our study.

    Conclusions

    Trials with inappropriate concealment of allocation had an overestimation of treatment effects when compared with trials with adequate concealment of allocation. Our results suggest that when evaluating risk of bias of primary RCTs in PT area, systematic reviewers and clinicians implementing trial findings into their clinical practice should pay attention to these characteristics, especially allocation concealment since it can be associated with an exaggeration of a treatment effect. Systematic reviewers should perform sensitivity analysis including trials with low risk of bias in these domains as primary analysis and/or in combination with less restrictive analyses. In addition, appropriate methods of sequence generation as well as allocation concealment should be implemented. Authors and editors should make sure that allocation concealment and random sequence generation are properly reported in trial reports.

    Acknowledgments

    In addition, the authors of this study thank the Alberta Research Center for Health Evidence (ARCHE) at the University of Alberta and all research assistants who helped with data collection.

    References

    1. ↵
      1. Nuesch E,
      2. Reichenbach S,
      3. Trelle S, et al
      . The importance of allocation concealment and patient blinding in osteoarthritis trials: a meta-epidemiologic study. Arthritis Rheum 2009;61:1633–41. doi:10.1002/art.24894
      OpenUrlCrossRefPubMedWeb of Science
    2. ↵
      1. Savovic J,
      2. Jones HE,
      3. Altman DG, et al
      . Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med 2012;157:429–38. doi:10.7326/0003-4819-157-6-201209180-00537
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      1. Wood L,
      2. Egger M,
      3. Gluud LL, et al
      . Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 2008;336:601–5. doi:10.1136/bmj.39465.451748.AD
      OpenUrlAbstract/FREE Full Text
    4. ↵
      1. Falagas ME,
      2. Grigori T,
      3. Ioannidou E
      . A systematic review of trends in the methodological quality of randomized controlled trials in various research fields. J Clin Epidemiol 2009;62:227–31, 231.e229. doi:10.1016/j.jclinepi.2008.07.012
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. Aickin M
      . Randomization, balance, and the validity and efficiency of design-adaptive allocation methods. J Stat Plann Inference 2001;94:97–119. doi:10.1016/S0378-3758(00)00228-7
      OpenUrl
    6. ↵
      1. Bausell RB,
      2. Lee WL,
      3. Soeken KL, et al
      . Larger effect sizes were associated with higher quality ratings in complementary and alternative medicine randomized controlled trials. J Clin Epidemiol 2004;57:438–46. doi:10.1016/j.jclinepi.2003.11.001
      OpenUrlCrossRefPubMed
    7. ↵
      1. Kjaergard LL,
      2. Villumsen J,
      3. Gluud C
      . Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 2001;135:982–9. doi:10.7326/0003-4819-135-11-200112040-00010
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Schulz KF,
      2. Chalmers I,
      3. Hayes RJ, et al
      . Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408–12. doi:10.1001/jama.1995.03520290060030
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Siersma V,
      2. Als-Nielsen B,
      3. Chen W, et al
      . Multivariable modelling for meta-epidemiological assessment of the association between trial quality and treatment effects estimated in randomized clinical trials. Stat Med 2007;26:2745–58. doi:10.1002/sim.2752
      OpenUrlCrossRefPubMedWeb of Science
    10. ↵
      1. Als-Nielsen B,
      2. Gluud L,
      3. Gluud C
      . Methodological quality and treatment effects in randomised trials: a review of six empirical studies. Paper presented at: 12th Cochrane Colloquium; 2–6 October 2004, Ottawa, Ontario, Canada.
    11. ↵
      1. Moher D,
      2. Pham B,
      3. Jones A, et al
      . Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? [see comment]. Lancet 1998;352:609–13. doi:10.1016/S0140-6736(98)01085-X
      OpenUrlCrossRefPubMedWeb of Science
    12. ↵
      1. Pildal J,
      2. Hrobjartsson A,
      3. Jorgensen KJ, et al
      . Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 2007;36:847–57. doi:10.1093/ije/dym087
      OpenUrlAbstract/FREE Full Text
    13. ↵
      1. Hewitt CE,
      2. Kumaravel B,
      3. Dumville JC, et al
      . Assessing the impact of attrition in randomized controlled trials. J Clin Epidemiol 2010;63:1264–70. doi:10.1016/j.jclinepi.2010.01.010
      OpenUrlCrossRefPubMed
    14. ↵
      1. Kjaergard LL,
      2. Als-Nielsen B
      . Association between competing interests and authors’ conclusions: Epidemiological study of randomised clinical trials published in the BMJ. BMJ 2002;325:249–52. doi:10.1136/bmj.325.7358.249
      OpenUrlAbstract/FREE Full Text
    15. ↵
      1. Trowman R,
      2. Dumville JC,
      3. Torgerson DJ, et al
      . The impact of trial baseline imbalances should be considered in systematic reviews: a methodological case study. J Clin Epidemiol 2007;60: 1229–33. doi:10.1016/j.jclinepi.2007.03.014
      OpenUrlCrossRefPubMed
    16. ↵
      1. Moseley AM,
      2. Herbert RD,
      3. Maher CG, et al
      . Reported quality of randomized controlled trials of physiotherapy interventions has improved over time. J Clin Epidemiol 2011;64:594–601. doi:10.1016/j.jclinepi.2010.08.009
      OpenUrlCrossRefPubMed
    17. ↵
      1. Armijo-Olivo S,
      2. Cummings G,
      3. Fuentes CJ, et al
      . Identifying items to assess methodological quality in physical therapy trials: a factor analysis. Phys Ther 2014;94:1272–84. doi:10.2522/ptj.20130464
      OpenUrlAbstract/FREE Full Text
    18. ↵
      1. da Costa B,
      2. Resta N,
      3. Beckett B, et al
      . Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a study protocol. Sys Rev 2014;3:1–6. doi:10.1186/2046-4053-3-144
      OpenUrl
    19. ↵
      World Confederation for Physical Therapy. Position statement: standards of physical therapy practice. World Confederation for Physical Therapy, 2011.
    20. ↵
      1. Higgins JP,
      2. Altman DG,
      3. Goetzsche PC, et al
      ., Cochrane Bias Methods Group; Cochrane Statistical Methods Group. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928. doi:10.1136/bmj.d5928
      OpenUrlFREE Full Text
    21. ↵
      1. Sterne JA,
      2. Juni P,
      3. Schulz KF, et al
      . Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research. Stat Med 2002;21:1513–24. doi:10.1002/sim.1184
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. Cohen J
      . Chapter 1: the concepts of power analysis. In: Cohen J, ed. Statistical power analysis for the behavioral sciences. 2nd edn. Hillsdale, NJ: Academic Press, INC, 1988:1–17.
    23. ↵
      1. Reichenbach S,
      2. Sterchi R,
      3. Scherer M, et al
      . Meta-analysis: chondroitin for osteoarthritis of the knee or hip. Ann Intern Med 2007;146:580–90. doi:10.7326/0003-4819-146-8-200704170-00009
      OpenUrlCrossRefPubMedWeb of Science
    24. ↵
      1. Nuesch E,
      2. Trelle S,
      3. Reichenbach S, et al
      . The effects of excluding patients from the analysis in randomised controlled trials: meta-epidemiological study. BMJ 2009;339:679–83. doi:10.1136/bmj.b3244
      OpenUrl
    25. ↵
      1. DerSimonian R,
      2. Laird N
      . Meta-analysis in clinical trials. Control Clin Trials 1986;7:177–88. doi:10.1016/0197-2456(86)90046-2
      OpenUrlCrossRefPubMedWeb of Science
    26. ↵
      1. Spittle AJ,
      2. Orton J,
      3. Doyle LW, et al
      . Early developmental intervention programs post hospital discharge to prevent motor and cognitive impairments in preterm infants. Cochrane Database Syst Rev 2007;(2):CD005495. Chichester: John Wiley & Sons, Ltd.
    27. ↵
      1. Fransen M,
      2. McConnell S,
      3. Hernandez-Molina G, et al
      . Exercise for osteoarthritis of the hip. Cochrane Database Syst Rev 2009;(3):CD007912. Chichester: John Wiley & Sons, Ltd. doi:10.1002/14651858.CD007912
    28. ↵
      1. Handoll HH,
      2. Cameron ID,
      3. Mak JC, et al
      . Multidisciplinary rehabilitation for older people with hip fractures. Cochrane Database Syst Rev 2009;(4):CD007125. Chichester: John Wiley & Sons, Ltd, Chichester. doi:10.1002/14651858.CD007125.pub2
    29. ↵
      1. Harvey LA,
      2. Brosseau L,
      3. Herbert RD
      . Continuous passive motion following total knee arthroplasty in people with arthritis. Cochrane Database Syst Rev 2010;(3):CD004260. Chichester: John Wiley & Sons, Ltd. doi:10.1002/14651858.CD004260.pub2
    30. ↵
      1. Katalinic OM,
      2. Harvey LA,
      3. Herbert RD, et al
      . Stretch for the treatment and prevention of contractures. Cochrane Database Syst Rev 2010;(9):CD007455. Chichester: John Wiley & Sons, Ltd, Chichester. doi:10.1002/14651858.CD007455.pub2
    31. ↵
      1. Manheimer E,
      2. Cheng K,
      3. Linde K, et al
      . Acupuncture for peripheral joint osteoarthritis. Cochrane Database Syst Rev 2010;(1):CD001977. John Wiley& Sons, Ltd, Chichester, UK. doi:10.1002/14651858.CD001977.pub2
    32. ↵
      1. Ostelo RW,
      2. Costa LO,
      3. Maher CG, et al
      . Rehabilitation after lumbar disc surgery. Cochrane Database Syst Rev 2008;(4):CD003007. John Wiley & Sons, Ltd. Chichester, UK. doi:10.1002/14651858.CD003007.pub2
    33. ↵
      1. Rutjes AW,
      2. Nüesch E,
      3. Sterchi R, et al
      . Therapeutic ultrasound for osteoarthritis of the knee or hip. Cochrane Database Syst Rev 2010;(1):CD003132. John Wiley & Sons, Ltd, Chichester. doi:10.1002/14651858.CD003132.pub2
    34. ↵
      1. Rutjes AW,
      2. Nüesch E,
      3. Sterchi R, et al
      . Transcutaneous electrostimulation for osteoarthritis of the knee. Cochrane Database Syst Rev 2009;(4):CD002823. John Wiley & Sons, Ltd, Chichester. doi:10.1002/14651858.CD002823.pub2
    35. ↵
      1. Schaafsma F,
      2. Schonstein E,
      3. Whelan KM, et al
      . Physical conditioning programs for improving work outcomes in workers with back pain. Cochrane Database Syst Rev 2010;(1):CD001822. John Wiley & Sons, Ltd, Chichester, UK. doi:10.1002/14651858.CD007131.pub2
    36. ↵
      1. Davies P,
      2. Taylor F,
      3. Beswick A, et al
      . Promoting patient uptake and adherence in cardiac rehabilitation. Cochrane Database Syst Rev 2010;(7):CD007131. John Wiley & Sons, Ltd, Chichester, UK.
    37. ↵
      1. Effing T,
      2. Monninkhof EM,
      3. van der Valk PD, et al
      . Self-management education for patients with chronic obstructive pulmonary disease. Cochrane Database Syst Rev 2007;(4):CD002990. John Wiley & Sons, Ltd, Chichester, UK.
    38. ↵
      1. Puhan M,
      2. Scharplatz M,
      3. Troosters T, et al
      . Pulmonary rehabilitation following exacerbations of chronic obstructive pulmonary disease. Cochrane Database Syst Rev 2009;(1):CD005305. John Wiley & Sons, Ltd, Chichester, UK. doi:10.1002/14651858.CD005305.pub2
    39. ↵
      1. Taylor RS,
      2. Dalal H,
      3. Jolly K, et al
      . Home-based versus centre-based cardiac rehabilitation. Cochrane Database Syst Rev 2010;(1):CD007130. John Wiley & Sons, Ltd, Chichester, UK. doi:10.1002/14651858.CD007130.pub2
    40. ↵
      1. Main E,
      2. Prasad A,
      3. Schans C
      . Conventional chest physiotherapy compared to other airway clearance techniques for cystic fibrosis. Cochrane Database Syst Rev 2005;(1):CD002011. John Wiley & Sons, Ltd, Chichester, UK.
    41. ↵
      1. O'Brien K,
      2. Nixon S,
      3. Tynan AM, et al
      . Aerobic exercise interventions for adults living with HIV/AIDS. Cochrane Database Syst Rev 2010;(8):CD001796. John Wiley & Sons, Ltd, Chichester, UK.
    42. ↵
      1. Jolliffe JA,
      2. Rees K,
      3. Taylor RS, et al
      . Exercise-based rehabilitation for coronary heart disease. Cochrane Database Syst Rev 2001;(1):CD001800. John Wiley & Sons, Ltd, Chichester, UK.
    43. ↵
      1. Shaw K,
      2. Gennat H,
      3. O'Rourke P, et al
      . Exercise for overweight or obesity. Cochrane Database Syst Rev 2006;(4):CD003817. John Wiley & Sons, Ltd, Chichester, UK.
    44. ↵
      1. Lacasse Y,
      2. Goldstein R,
      3. Lasserson TJ, et al
      . Pulmonary rehabilitation for chronic obstructive pulmonary disease. Cochrane Database Syst Rev 2006;(4):CD003793. John Wiley & Sons, Ltd, Chichester, UK.
    45. ↵
      1. Sirtori V,
      2. Corbetta D,
      3. Moja L, et al
      . Constraint-induced movement therapy for upper extremities in stroke patients. Cochrane Database Syst Rev 2009;(4):CD004433. John Wiley & Sons, Ltd, Chichester, UK. doi:10.1002/14651858.CD004433.pub2
    46. ↵
      1. States RA,
      2. Pappas E,
      3. Salem Y
      . Overground physical therapy gait training for chronic stroke patients with mobility deficits. Cochrane Database Syst Rev 2009;(3):CD006075. John Wiley & Sons, Ltd, Chichester, UK. doi:10.1002/14651858.CD006075.pub2
    47. ↵
      1. Mead GE,
      2. Morley W,
      3. Campbell P, et al
      . Exercise for depression. Cochrane Database Syst Rev 2009;(3):CD004366. John Wiley & Sons, Ltd, Chichester, UK. doi:10.1002/14651858.CD004366.pub4
    48. ↵
      1. Pollock A,
      2. Baer G,
      3. Pomeroy V, et al
      . Physiotherapy treatment approaches for the recovery of postural control and lower limb function following stroke. Cochrane Database Syst Rev 2007;(1):CD001920. John Wiley & Sons, Ltd, Chichester, UK.
    49. ↵
      1. Mehrholz J,
      2. Friis R,
      3. Kugler J, et al
      . Treadmill training for patients with Parkinson's disease. Cochrane Database Syst Rev 2010;(1):CD007830. John Wiley & Sons, Ltd, Chichester, UK. doi:10.1002/14651858.CD007830.pub2
    50. ↵
      1. Woodford H,
      2. Price C
      . EMG biofeedback for the recovery of motor function after stroke. Cochrane Database Syst Rev 2007;(2):CD004585. John Wiley & Sons, Ltd, Chichester, UK.
    51. ↵
      1. Watson L,
      2. Ellis B,
      3. Leng GC
      . Exercise for intermittent claudication. Cochrane Database Syst Rev 2008;(4):CD000990. John Wiley & Sons, Ltd, Chichester, UK. doi:10.1002/14651858.CD000990.pub2
    52. ↵
      1. Kramer MS,
      2. McDonald SW
      . Aerobic exercise for women during pregnancy. Cochrane Database Syst Rev 2006;(3):CD000180. John Wiley & Sons, Ltd, Chichester, UK.
    53. ↵
      1. Orozco LJ,
      2. Buchleitner AM,
      3. Gimenez-Perez G, et al
      . Exercise or exercise and diet for preventing type 2 diabetes mellitus. Cochrane Database Syst Rev 2008;(3):CD003054. John Wiley & Sons, Ltd, Chichester, UK. doi:10.1002/14651858.CD003054.pub3
    54. ↵
      1. Bendermacher BL,
      2. Willigendael EM,
      3. Teijink JA, et al
      . Supervised exercise therapy versus non-supervised exercise therapy for intermittent claudication. Cochrane Database Syst Rev 2006;(2):CD005263. John Wiley & Sons, Ltd, Chichester, UK.
    55. ↵
      1. Forster A,
      2. Lambley R,
      3. Hardy J, et al
      . Rehabilitation for older people in long-term care. Cochrane Database Syst Rev 2009;(1):CD004294. John.Wiley & Sons, Ltd, Chichester, UK. doi:10.1002/14651858.CD004294.pub2
    56. ↵
      1. Cramp F,
      2. Daniel J
      . Exercise for the management of cancer-related fatigue in adults. Cochrane Database Syst Rev 2008;(2):CD006145. John Wiley & Sons, Ltd, Chichester, UK. doi:10.1002/14651858.CD006145.pub2
    57. ↵
      1. Jadad AR
      . Bias in randomized controlled trials. In: Jadad AR, Enkin M, eds. Randomized controlled trials: questions, answers, and musings. 2nd edn. Malden, MA: Blackwell Pub, 2007:29–47.
    58. ↵
      1. Egger M,
      2. Juni P,
      3. Bartlett C, et al
      . How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess 2003;7:1–76.
      OpenUrlPubMed
    59. ↵
      1. Soares HP,
      2. Daniels S,
      3. Kumar A, et al.
      , Radiation Therapy Oncology Group. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ 2004;328: 22–4. doi:10.1136/bmj.328.7430.22
      OpenUrlAbstract/FREE Full Text
    60. ↵
      1. Abdul Latif L,
      2. Daud Amadera JE,
      3. Pimentel D, et al
      . Sample size calculation in physical medicine and rehabilitation: a systematic review of reporting, characteristics, and results in randomized controlled trials. Arch Phys Med Rehabil 2011;92:306–15. doi:10.1016/j.apmr.2010.10.003
      OpenUrlCrossRefPubMed
    61. ↵
      1. Dorstyn D,
      2. Mathias J,
      3. Denson L
      . Applications of telecounselling in spinal cord injury rehabilitation: a systematic review with effect sizes. Clin Rehabil 2013;27:1072–83. doi:10.1177/0269215513488001
      OpenUrlAbstract/FREE Full Text
    62. ↵
      1. Barker AL,
      2. Talevski J,
      3. Morello RT, et al
      . Effectiveness of aquatic exercise for musculoskeletal conditions: a meta-analysis. Arch Phys Med Rehabil 2014;95:1776–86. doi:10.1016/j.apmr.2014.04.005
      OpenUrlPubMed
    63. ↵
      1. Pollock A,
      2. Baer G,
      3. Langhorne P, et al
      . Physiotherapy treatment approaches for the recovery of postural control and lower limb function following stroke: a systematic review. Clin Rehabil 2007;21:395–410. doi:10.1177/0269215507073438
      OpenUrlAbstract/FREE Full Text
    64. ↵
      1. Balk EM,
      2. Bonis PA,
      3. Moskowitz H, et al
      . Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA 2002;287:2973–82. doi:10.1001/jama.287.22.2973
      OpenUrlCrossRefPubMedWeb of Science
    65. ↵
      1. Verhagen AP,
      2. de Vet HC,
      3. Willemsen S, et al
      . A meta-regression analysis shows no impact of design characteristics on outcome in trials on tension-type headaches. J Clin Epidemiol 2008;61:813–18. doi:10.1016/j.jclinepi.2007.10.006
      OpenUrlCrossRefPubMed
    66. ↵
      1. Hempel S
      . Detection of associations between trial quality and effect sizes. 2012; http://www.ncbi.nlm.nih.gov/books/NBK84249/

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • Contributors SA-O provided concept/idea/research design and writing. SA-O, CH, JF and HS performed the experiments. SA-O, BRdaC, GGC, CH, JF and HS were responsible for interpretation of the results and review of the final manuscript. SA-O, BRdaC analysed the data. All authors critically revised and provided final approval of the version to be published.

    • Funding This project is funded by the Canadian Institutes of Health Research (CIHR), Alberta Innovates Health Solutions through a knowledge translation initiative grant, the Knowledge Translation (KT) Canada research Stipend programme, and the Physiotherapy Foundation of Canada (PFC) through a B E Schnurr Memorial Fund Award.

    • Competing interests SA-O is supported by the Canadian Institutes of Health Research (CIHR) through a full-time Banting fellowship, the Alberta Innovates Health Solution through an incentive award, the STIHR Training Programme from Knowledge Translation (KT) Canada, the University of Alberta, and by the Faculty of Rehabilitation Medicine, at the University of Alberta, through the ‘Music and Motion Fellowship’. GGC holds a Centennial Professorship at the University of Alberta (2013–2020). JF is supported by the Government of Chile, University of Alberta through a Dissertation fellowship and the University Catholic of Maule. HS is supported through a Clinician Fellowship Award by Alberta Innovates—Health Solutions (AIHS), the Honorary Izaak Walton Killam Memorial Scholarship by the University of Alberta and the Honorary WCHRI Award by the Women and Children's Health Research Institute (WCHRI).

    • Ethics approval This Research protocol has been approved by the Ethics Board of the University of Alberta (Pro00038172).

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement No additional data are available.