Methodological characteristics and treatment effect sizes in oral health randomised controlled trials: Is there a relationship? Protocol for a meta-epidemiological study
=========================================================================================================================================================================

* Humam Saltaji
* Susan Armijo-Olivo
* Greta G Cummings
* Maryam Amin
* Carlos Flores-Mir

## Abstract

**Introduction** It is fundamental that randomised controlled trials (RCTs) are properly conducted in order to reach well-supported conclusions. However, there is emerging evidence that RCTs are subject to biases which can overestimate or underestimate the true treatment effect, due to flaws in the study design characteristics of such trials. The extent to which this holds true in oral health RCTs, which have some unique design characteristics compared to RCTs in other health fields, is unclear. As such, we aim to examine the empirical evidence quantifying the extent of bias associated with methodological and non-methodological characteristics in oral health RCTs.

**Methods and analysis** We plan to perform a meta-epidemiological study, where a sample size of 60 meta-analyses (MAs) including approximately 600 RCTs will be selected. The MAs will be randomly obtained from the Oral Health Database of Systematic Reviews using a random number table; and will be considered for inclusion if they include a minimum of five RCTs, and examine a therapeutic intervention related to one of the recognised dental specialties. RCTs identified in selected MAs will be subsequently included if their study design includes a comparison between an intervention group and a placebo group or another intervention group. Data will be extracted from selected trials included in MAs based on a number of methodological and non-methodological characteristics. Moreover, the risk of bias will be assessed using the Cochrane Risk of Bias tool. Effect size estimates and measures of variability for the main outcome will be extracted from each RCT included in selected MAs, and a two-level analysis will be conducted using a meta-meta-analytic approach with a random effects model to allow for intra-MA and inter-MA heterogeneity.

**Ethics and dissemination** The intended audiences of the findings will include dental clinicians, oral health researchers, policymakers and graduate students. The aforementioned will be introduced to the findings through workshops, seminars, round table discussions and targeted individual meetings. Other opportunities for knowledge transfer will be pursued such as key dental conferences. Finally, the results will be published as a scientific report in a dental peer-reviewed journal.

*   Oral & Maxillofacial Surgery
*   Oral Medicine

### Strengths and limitations of this study

*   This meta-epidemiological study is the first of its kind conducted in oral health research, and will most likely have important implications for oral health research, dental practice decision-making and oral health policy.

*   This work will have an important impact on the quality of future oral health RCTs and systematic reviews by providing new and important insights about potential biases that exist in RCTs as well as factors associated with bias in oral health RCTs.

*   It will provide a research framework when appraising, reporting and conducting oral health RCTs.

## Introduction

Current scientific knowledge for clinical research is based on randomised control trials (RCTs) that have been synthesised in systematic reviews (SRs) and meta-analyses (MAs), which together comprise the ‘gold standard’ of scientific evidence.1 ,2 The abundance of RCTs is continually increasing with about 50 new clinical trials published every month in the field of dentistry.3 Since these sources are considered the highest level of evidence for the efficacy of treatment interventions, the information gathered from them is used to guide clinical practice and policy decisions.4 In the field of oral health research, several investigations have assessed the methodological/reporting quality of oral health RCTs,5–8 and examined important aspects related to the conduct and reporting of these trials such as: clustering effects,9 reporting statistical findings,10 sample size justification11 and randomisation process.12 ,13 Nonetheless, the quality of these forms of evidence has not yet been fully scrutinised to identify design flaws and their impact on treatment effect estimates in the field of oral health research.3 ,14

Ideally, RCTs should be properly conducted and accurately reported in order to reach well-supported conclusions for decision–making that are both valid and generalisable to patients who will receive the interventions in clinical practice.4 ,15 However, evidence is emerging that some RCTs are biased and overestimate the magnitude of the effect size due to flaws in their design and/or reported study characteristics.15 ,16 These studies will likely skew the overall conclusions of MAs once pooled, possibly leading to faulty treatment decisions.2 ,17 Generally, the poor methodological quality of these RCTs has resulted in the tendency to exaggerate or overestimate the true treatment effect (effect size).18 Among the flawed characteristics observed in previously published reports and found to have an impact on the true treatment effect were: the lack of randomisation and concealment,19 ,20 inadequate blinding16 ,20 ,21 and industrial funding,22 although not all the studies confirmed these associations.23 ,24 This could lead clinicians to the implementation of treatment choices, which might be inappropriate or ineffective and have negative effects on treatment outcomes in dental practices.3

The assumption of the association between trial quality and true effect size is derived from published ‘meta-epidemiological’ studies, which are investigations that quantify the extent of bias in the effect sizes related to trial quality in a group of meta-analyses.25 There are a few meta-epidemiological investigations that have been conducted in the field of medicine16 ,18 ,23–26; however, the value of the conclusions of some of these investigations to other healthcare fields is limited by numerous factors including: the examination of quality items that were related to reporting quality and not methodological quality or bias,23 ,25 ,26 the failure to examine continuous outcomes (which occurred in the majority of studies)16 ,23 ,25 based on a preference for evaluating dichotomous outcomes which could limit applying their conclusions to RCTs with continuous outcomes, and the presentation of inconsistent findings regarding the methodological criteria that are associated with effect size.16 ,23

More importantly, meta-epidemiological studies reported that bias in effect size associated with methodological characteristics may vary between medical fields or different areas of health research27 and differ based on the type of intervention.20 ,25 To our current knowledge, no meta-epidemiological study has been conducted in any of the nine specialised dental fields that examined the extent of bias related to the quality of oral health RCTs. It is unclear to what extent this holds true in oral health RCTs, which have some unique design characteristics compared with RCTs in other health fields, such as: difficulty in applying blinding, use of a broad range of different interventions (surgical, non-surgical, drug and non-drug),28 use of multiple outcomes, common use of split-mouth and crossover designs and clustering effects,9 ,29 which make the evaluation of these trials more challenging compared to trials in other health areas.

As such, the purpose of the proposed study is to provide a first step in the development of a research framework for appraising, reporting and conducting RCTs in oral health research. The objectives are to: (1) examine the empirical evidence for associations between methodological trial characteristics (eg, adequacy of randomisation, adequacy of allocation concealment, baseline comparability, blinding of assessors and participants, similarity of cointerventions, adequacy of compliance to the treatment, among others) and treatment effect estimates (effect sizes) in oral health RCTs and (2) determine if other non-methodological study characteristics (eg, the nature of intervention, specialty, type of outcomes, number of centres, type of funding, among others) are associated with effect sizes in oral health RCTs.

The hypothesis of the proposed study is that there is no difference between treatment effect estimates (effect sizes) for oral health trials meeting certain methodological quality characteristics versus trials not meeting those quality characteristics, such as: adequacy of randomisation, adequacy of allocation concealment, baseline comparability, blinding of assessors and participants, similarity of cointerventions, similarity of outcome assessment, description of withdrawals and adequacy of compliance to the treatment, among others; and that trials with different non-methodological characteristics such as nature of intervention, type of outcomes, study design, number of centres, type of funding, sample size and speciality, among others, will not have different treatment effect estimates (effect sizes).

## Methods and analysis

### Design

A meta-epidemiological study.

### Study selection

#### Selection of MAs

MAs/SRs will be included if they fulfil the following inclusion criteria: 

1.  Reports should be therapeutic oral health MAs defined as reviews that examined therapeutic interventions related to dental/oral diseases as defined by the American Dental Association (ADA) scope of practice.28 ,30 Reports will be considered as MAs if they explicitly identified and summarised evidence from several published reports through quantitative analyses.31 ,32

2.  MAs should include a minimum of five RCTs and provide quantitative data of treatment estimates.

3.  The MA should examine at least one continuous outcome.

4.  MAs should be full-length reports.

#### Selection of RCTs

All RCTs included in selected MAs will be eligible if they meet the following inclusion–exclusion criteria: 

1.  The study design is reported to be an RCT;33

2.  Comparison is between an intervention and a placebo or another intervention;

3.  RCTs evaluated a therapeutic intervention related to one of the dental specialties defined by the ADA;30

4.  RCTs will be excluded if the results are reported in a way that does not allow for effect sizes to be calculated.

### Sample size calculation

Previously published meta-epidemiological studies have been reported to be underpowered because of their small sample sizes and their highly heterogeneous samples.27 Accordingly, it has been suggested compiling a set of RCTs that are specific to clinical fields to decrease heterogeneity and improve power of meta-epidemiological reports. Our study will focus on oral health RCTs published in the recognised nine dental specialties. Furthermore, a minimum of 60 MAs containing approximately 600 RCTs will be assessed for this meta-epidemiological study. Given the previous reports,34 ,35 it could be anticipated that a difference in effect sizes of at least 0.15 will be obtained between trials with and without selected quality domains. This magnitude of difference has been argued to correspond to one-quarter to one-half of the typical treatment effect found for interventions in areas similar to dentistry. Thus, this difference should also be relevant to the field of dentistry.

As such, we planned a sample size of 60 MAs expecting that a sample of 600 RCTs would come from these MAs. To the best of our knowledge, this number of RCTs will represent the largest sample size examined in a meta-epidemiological study aimed at examining bias related to trial quality using continuous outcomes. The sample of MAs will be selected from the Oral Health Database of Systematic Reviews,28 developed by the authors and include all of the oral health SRs published between 1999 and 2012, encompassing the nine dental specialties defined by the ADA.30 This database contains 153 MAs (39 Cochrane and 114 non-Cochrane), out of the 1188 SRs included in the database, which potentially meet the inclusion–exclusion criteria for this study. Figure 1 provides further details of the SRs identified in the database and the number of MAs potentially meeting the eligibility criteria for this study, within each of the nine dental specialties and for Cochrane and non-Cochrane SRs separately.

![Figure 1](http://bmjopen.bmj.com/https://bmjopen.bmj.com/content/bmjopen/4/2/e004527/F1.medium.gif)

[Figure 1](http://bmjopen.bmj.com/content/4/2/e004527/F1)

Figure 1 
Systematic reviews identified in the database and the number of MAs potentially meeting the eligibility criteria for this study.

### Data extraction

A data extraction template will be designed in Microsoft ACCESS and pilot tested. With regard to assessors, a panel of assessors from varied health research backgrounds will perform data extraction. One of the team members will perform the training for all assessors and will make sure that all of them have a clear understanding of the data extraction process. Training of these assessors will be carried out through 10–15 separate articles, not included in the set of articles to be reviewed. Each of the 10–15 training articles will be independently reviewed by all the members of the review panel and then discussed by the panel. In order to ensure good agreement between the assessors, the training exercise will be repeated to address any issues identified in the first exercise. For actual data extraction, two assessors will independently complete data extraction with a consensus meeting utilised to resolve any disagreement between the assessors. If a consensus could not be achieved, then the two assessors will consult with a third assessor (HS or SA-O) to achieve full consensus, and only consensus answers will be used for all analyses. Data will be extracted on the following items.

#### Non-methodological characteristics

Dental speciality (eg, dental public health, endodontics, oral medicine and oral pathology, oral and maxillofacial radiology, oral and maxillofacial surgery, orthodontics and dentofacial orthopaedics, pediatric dentistry, periodontics, prosthodontics and restorative dentistry), year of publication, source of funding (eg, industry, government, foundation and academic), type of intervention (eg, drug, surgical, device, dental material, psychological, educational and policy), number of randomised groups, number of centres (eg, multicentre and single centre), study design (eg, parallel, crossover and factorial) and type of outcome (eg, subjective and objective).

#### Methodological characteristics

Methodological characteristics will be based on preliminary work performed by our research team and will be extracted from commonly used tools to evaluate the methodological quality of RCT in health research,36 such as: adequacy of randomisation, adequacy of allocation concealment, baseline comparability, blinding of assessors and participants, similarity of cointerventions, similarity of outcome assessment, description of withdrawals and adequacy of compliance to the treatment. Guidelines for decision-making will be formed based on the previous work of our team, in order to increase consistency.36 ,37

#### Risk of bias

Risk of bias will be assessed using the Cochrane risk of bias tool as ‘high’, ‘low’ or ‘unclear’ and will follow the guidelines of the Cochrane Collaboration for scoring clinical trials.33

#### Treatment effect estimates

Treatment effect estimates, measures of variability (SDs and 95% CIs) and respective sample sizes will be extracted for the main outcome.

### Data analysis

Descriptive statistics (means and SDs) will be extracted for one continuous outcome per RCT. RCTs that are included in more than one MA will be eligible for inclusion in the study only once, and will be extracted from the MA with the fewer number of RCTs. STATA statistical software V.12 will be employed to perform the planned statistical analyses.

We propose to use a two-level meta-meta-analytic approach with the use of a random effects model. This analysis will permit the evaluation of the heterogeneity intra-MAs and inter-MAs.38 The first step will consist in obtaining the standardised effect size estimates for the primary outcome of each trial using the guidelines established by Cohen.39 The second step will involve pooling the results of the previous analysis, using a combined difference approach, to demonstrate the different components of MAs across all MAs. Moreover, the data acquired will be used to evaluate certain components of the methodological assessment—such as allocation concealment, randomisation, blinding, etc—and will subsequently be used to divide the data set into two groups: one group having adequately addressed the said component, and the other not addressing it. Thus, for each MA, we will conduct metaregression techniques to derive the difference between pooled estimates from trials with (eg, allocation concealment) and without the characteristic of interest. Formal tests of interaction will be performed separately for each MA based on z scores for estimated differences in effect sizes between trials with and without the characteristic of interest and the corresponding SEs. Therefore, two pooled effect sizes will be calculated for each MA. The effect sizes at this stage will be combined using the DerSimonian and Laird random effect models to allow for appropriate inter-MA heterogeneity assessment.40 P values will be two sided. Analysis will be performed by a statistician specialised in the meta-epidemiological approach.

## Discussion

The findings of the proposed research could most likely have important implications for oral health research, dental practice decision-making and oral health policy. The proposed research will be the first meta-epidemiological study that provides empirical evidence regarding biases related to the quality of RCTs in the field of dentistry. This work, in combination with some of the current knowledge the oral health community already has, should have an important impact on the quality of future oral health RCTs, SRs and MAs by providing new and important insights about potential biases that exist in RCTs as well as factors associated with bias in oral health RCTs. Additionally, it will provide an improved framework when conducting, appraising and reporting oral health RCTs.

More importantly, this additional information will update dental professionals about proper, evidence-based decision-making when treating patients and could assist guideline developers and policymakers in making informed decisions about the implementation of dental interventions. Finally, the outcomes generated from this work should most likely be of value for developing and disseminating future research framework for the conduct and reporting of oral health RCTs.

Dissemination of the developed framework will be achieved through an array of means to maximise exposure. The intended audiences will include dental clinicians, oral health researchers, policymakers and graduate students. The aforementioned persons will be introduced to this research framework through workshops, seminars, round table discussions and targeted individual meetings. Moreover, key organisations will be used to strengthen the dissemination strategy, such as the International Association for Dental Research (IADR), the American Dental Association (ADA), the Canadian Dental Association (CDA) and the Cochrane Bias Methods Group. Other opportunities for knowledge transfer will be pursued such as key conferences (eg, the annual meeting of the International Association for Dental Research). Finally, the results of this study will be published as a scientific report in a dental peer-reviewed journal.

## Acknowledgments

The authors would like to thank Dr Lisa Hartling and Dr Paul Major for their contributions to the initial planning of this study. Dr Humam Saltaji is supported through a Clinician Fellowship Award by Alberta Innovates—Health Solutions (AIHS), the Honorary Izaak Walton Killam Memorial Scholarship by the University of Alberta, Alpha Omega Foundation of Canada Grant, and the Honorary WCHRI Award by the Women and Children's Health Research Institute (WCHRI). Dr Susan Armijo-Olivo is supported by the Canadian Institutes of Health Research (CIHR) through a full-time Banting fellowship, the Alberta Innovates Health solution through an incentive award, the STIHR Training Programme from Knowledge Translation (KT) Canada, and the University of Alberta. Dr Greta Cummings is funded both provincially and nationally and holds a Population Health Investigator award from the Alberta Heritage Foundation for Medical Research (2006–2013). In July 2013, she holds a Centennial Professorship at the University of Alberta (2013–2020).

## Footnotes

*   Contributors HS SA-O MA GGC CF-M conceived and designed the study; HS drafted the manuscript and integrated critical feedback from all of the other authors. All of the authors approved the final version of the manuscript.

*   Funding This project is funded by the Alpha Omega Foundation of Canada and Alberta Innovates—Health Solutions (AIHS) through a Clinician Fellowship Award.

*   Competing interests None.

*   Provenance and peer review Not commissioned; externally peer reviewed.

*   Data sharing statement No additional data are available.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/3.0/](http://creativecommons.org/licenses/by-nc/3.0/)

## References

1.  Altman DG, Schulz KF, Moher D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001;134:663–94.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.7326/0003-4819-134-8-200104170-00012&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=11304107&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000168177500006&link_type=ISI) 

2.  Gluud LL. Bias in clinical intervention research. Am J Epidemiol 2006;163:493–501.
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYW1qZXBpZCI7czo1OiJyZXNpZCI7czo5OiIxNjMvNi80OTMiO3M6NDoiYXRvbSI7czoyNToiL2Jtam9wZW4vNC8yL2UwMDQ1MjcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

3.  Needleman I, Worthington H, Moher D, et al. Improving the completeness and transparency of reports of randomized trials in oral health: the CONSORT Statement. Am J Dentist 2008;21:7–12.
    
    

4.  Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. Int J Surg 2012;10:28–55.
    
    

5.  Pandis N, Polychronopoulou A, Madianos P, et al. Reporting of research quality characteristics of studies published in 6 major clinical dental specialty journals. Jo Evid Based Dent Pract 2011;11:75–83.
    
    

6.  Seehra J, Wright NS, Polychronopoulou A, et al. Reporting quality of abstracts of randomized controlled trials published in dental specialty journals. J Evid Based Dent Pract 2013;13:1–8.
    
    

7.  Pandis N, Polychronopoulou A, Eliades T. An assessment of quality characteristics of randomised control trials published in dental journals. J Dent 2010;38:713–21.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.jdent.2010.05.014&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=20546823&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000281500100004&link_type=ISI) 

8.  Dumbrigue HB, Al-Bayat MI, Ng CC, et al. Assessment of bias in methodology for randomized controlled trials published on implant dentistry. J Prosthodont 2006;15:257–63.
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=16827739&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 

9.  Koletsi D, Pandis N, Polychronopoulou A, et al. Does published orthodontic research account for clustering effects during statistical data analysis? Eur J Orthod 2012;34:287–92.
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiZW9ydGhvIjtzOjU6InJlc2lkIjtzOjg6IjM0LzMvMjg3IjtzOjQ6ImF0b20iO3M6MjU6Ii9ibWpvcGVuLzQvMi9lMDA0NTI3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

10. Polychronopoulou A, Pandis N, Eliades T. Appropriateness of reporting statistical results in orthodontics: the dominance of P values over confidence intervals. Eur J Orthod 2011;33:22–5.
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiZW9ydGhvIjtzOjU6InJlc2lkIjtzOjc6IjMzLzEvMjIiO3M6NDoiYXRvbSI7czoyNToiL2Jtam9wZW4vNC8yL2UwMDQ1MjcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

11. Koletsi D, Pandis N, Fleming PS. Sample size in orthodontic randomized controlled trials: are numbers justified? Eur J Orthod 2013;1:1.
    
    

12. Koletsi D, Pandis N, Polychronopoulou A, et al. What's in a title? An assessment of whether randomized controlled trial in a title means that it is one. Am J Orthod Dentofacial Orthop 2012;141:679–85.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.ajodo.2011.12.020&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=22640669&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 

13. Koletsi D, Pandis N, Polychronopoulou A, et al. Mislabeling controlled clinical trials (CCTs) as ‘randomized clinical trials (RCTs)’ in dental specialty journals. J Evid Based Dent Pract 2012;12:124–30.
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=22935275&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 

14. Fenwick J, Needleman IG, Moles DR. The effect of bias on the magnitude of clinical outcomes in periodontology: a pilot study. J Clin Periodontol 2008;35:775–82.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1111/j.1600-051X.2008.01291.x&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=18840153&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 

15. Moher D, Cook DJ, Jadad AR, et al. Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses. Health Technol Assess 1999;3:i–iv, 1–98.
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=10374081&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 

16. Moher D, Pham B, Jones A, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998;352:609–13.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/S0140-6736(98)01085-X&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=9746022&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000075567400010&link_type=ISI) 

17. Balk EM, Bonis PA, Moskowitz H, et al. Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA 2002;287:2973–82.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1001/jama.287.22.2973&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=12052127&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000176128300027&link_type=ISI) 

18. van Tulder MW, Suttorp M, Morton S, et al. Empirical evidence of an association between internal validity and effect size in randomized controlled trials of low-back pain. Spine (Phila Pa 1976) 2009;34:1685–92.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1097/BRS.0b013e3181ab6a78&link_type=DOI) 

19. Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 2001;135:982–9.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.7326/0003-4819-135-11-200112040-00010&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=11730399&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000172520000005&link_type=ISI) 

20. Egger M, Juni P, Bartlett C, et al. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess 2003;7:1–76.
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=12583822&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 

21. Savovic J, Jones HE, Altman DG, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med 2012;157:429–38.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.7326/0003-4819-157-6-201209180-00537&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=22945832&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000308912800017&link_type=ISI) 

22. Lexchin J, Bero LA, Djulbegovic B, et al. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 2003;326:1167–70.
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMjYvNzQwMC8xMTY3IjtzOjQ6ImF0b20iO3M6MjU6Ii9ibWpvcGVuLzQvMi9lMDA0NTI3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

23. Balk EM, Bonis PA, Moskowitz H, et al. Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. JAMA 2002;287:2973–82.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1001/jama.287.22.2973&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=12052127&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000176128300027&link_type=ISI) 

24. Siersma V, Als-Nielsen B, Chen W, et al. Multivariable modelling for meta-epidemiological assessment of the association between trial quality and treatment effects estimated in randomized clinical trials. Stat Med 2007;26:2745–58.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1002/sim.2752&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=17117373&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000247058500001&link_type=ISI) 

25. Wood L, Egger M, Gluud LL, et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 2008;336:601–5.
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMzYvNzY0NC82MDEiO3M6NDoiYXRvbSI7czoyNToiL2Jtam9wZW4vNC8yL2UwMDQ1MjcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

26. Pildal J, Hrobjartsson A, Jorgensen KJ, et al. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 2007;36:847–57.
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiaW50amVwaWQiO3M6NToicmVzaWQiO3M6ODoiMzYvNC84NDciO3M6NDoiYXRvbSI7czoyNToiL2Jtam9wZW4vNC8yL2UwMDQ1MjcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

27. Hempel S, Miles J, Suttorp MJ, et al. Detection of associations between trial quality and effect sizes. Rockville, MD: Agency for Healthcare Research and Quality (US), 2012.
    
    

28. Saltaji H, Cummings GG, Armijo-Olivo S, et al. A Descriptive Analysis of Oral Health Systematic Reviews Published 1991–2012: cross Sectional Study. PLoS ONE 2013;8:e74545.
    
    

29. Fleming PS, Koletsi D, Polychronopoulou A, et al. Are clustering effects accounted for in statistical analysis in leading dental specialty journals? J Dent 2013;41:265–70.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/j.jdent.2012.11.012&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=23201411&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 

30. American Dental Association. Specialty Definitions. [http://www.ada.org/495.aspx](http://www.ada.org/495.aspx) (accessed April 2012).
    
    

31. Major MP, Major PW, Flores-Mir C. Benchmarking of reported search and selection methods of systematic reviews by dental speciality. Evid Based Dent 2007;8:66–70.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1038/sj.ebd.6400504&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=17891119&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 

32. Glenny AM, Esposito M, Coulthard P, et al. The assessment of systematic reviews in dentistry. Eur J Oral Sci 2003;111: 85–92.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1034/j.1600-0722.2003.00013.x&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=12648258&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 

33. 1.  Higgins JPT, 
    2.  Green S
    
    , eds. Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.0. The Cochrane Collaboration, 2008. [http://www.cochrane-handbook.org](http://www.cochrane-handbook.org)
    
    

34. Nuesch E, Trelle S, Reichenbach S, et al. The effects of excluding patients from the analysis in randomised controlled trials: meta-epidemiological study. BMJ 2009;339:b3244.
    
    [Abstract/FREE Full Text](http://bmjopen.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzMzkvc2VwMDdfMS9iMzI0NCI7czo0OiJhdG9tIjtzOjI1OiIvYm1qb3Blbi80LzIvZTAwNDUyNy5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

35. Nuesch E, Reichenbach S, Trelle S, et al. The importance of allocation concealment and patient blinding in osteoarthritis trials: a meta-epidemiologic study. Arthritis Rheum 2009;61:1633–41.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1002/art.24894&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=19950329&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 

36. Armijo-Olivo S, Fuentes J, Ospina M, et al. Inconsistency in the items included in tools used in general health research and physical therapy to evaluate the methodological quality of randomized controlled trials: a descriptive analysis. BMC Med Res Methodol 2013;13:116.
    
    

37. Armijo-Olivo S, Fuentes J, Rogers T, et al. How should we evaluate the risk of bias of physical therapy trials? a psychometric and meta-epidemiological approach towards developing guidelines for the design, conduct, and reporting of RCTs in Physical Therapy (PT) area: a study protocol. Syst Rev 2013;2:88.
    
    

38. Sterne JA, Juni P, Schulz KF, et al. Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research. Stat Med 2002;21:1513–24.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1002/sim.1184&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=12111917&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=000176016900003&link_type=ISI) 

39. 1.  Cohen J
    
    Cohen J. Chapter 1: the concepts of power analysis. In: Cohen J, ed. Statistical power analysis for the behavioral sciences. 2nd edn. Hillsdale, NJ: Academic Press, INC, 1988:1–17.
    
    

40. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986;7:177–88.
    
    [CrossRef](http://bmjopen.bmj.com/lookup/external-ref?access_num=10.1016/0197-2456(86)90046-2&link_type=DOI) 
    
    [PubMed](http://bmjopen.bmj.com/lookup/external-ref?access_num=3802833&link_type=MED&atom=%2Fbmjopen%2F4%2F2%2Fe004527.atom) 
    
    [Web of Science](http://bmjopen.bmj.com/lookup/external-ref?access_num=A1986F013900001&link_type=ISI)