You are here

Article Text

Article menu

Download PDFPDF

XML
Research methods
Research
Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review
  1. Matthew J Page1,2,
  2. Joanne E McKenzie1,
  3. Julian P T Higgins2
  1. 1 School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
  2. 2 Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
  1. Correspondence to Dr Matthew J Page; matthew.page{at}monash.edu

Abstract

Background Several scales, checklists and domain-based tools for assessing risk of reporting biases exist, but it is unclear how much they vary in content and guidance. We conducted a systematic review of the content and measurement properties of such tools.

Methods We searched for potentially relevant articles in Ovid MEDLINE, Ovid Embase, Ovid PsycINFO and Google Scholar from inception to February 2017. One author screened all titles, abstracts and full text articles, and collected data on tool characteristics.

Results We identified 18 tools that include an assessment of the risk of reporting bias. Tools varied in regard to the type of reporting bias assessed (eg, bias due to selective publication, bias due to selective non-reporting), and the level of assessment (eg, for the study as a whole, a particular result within a study or a particular synthesis of studies). Various criteria are used across tools to designate a synthesis as being at ‘high’ risk of bias due to selective publication (eg, evidence of funnel plot asymmetry, use of non-comprehensive searches). However, the relative weight assigned to each criterion in the overall judgement is unclear for most of these tools. Tools for assessing risk of bias due to selective non-reporting guide users to assess a study, or an outcome within a study, as ‘high’ risk of bias if no results are reported for an outcome. However, assessing the corresponding risk of bias in a synthesis that is missing the non-reported outcomes is outside the scope of most of these tools. Inter-rater agreement estimates were available for five tools.

Conclusion There are several limitations of existing tools for assessing risk of reporting biases, in terms of their scope, guidance for reaching risk of bias judgements and measurement properties. Development and evaluation of a new, comprehensive tool could help overcome present limitations.

  • publication bias
  • bias (epidemiology)
  • review literature as topic
  • checklist

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

https://doi.org/10.1136/bmjopen-2017-019703

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • Tools for assessing risk of reporting biases, and studies evaluating their measurement properties, were identified by searching several relevant databases using a search string developed in conjunction with an information specialist.

    • Detailed information on the content and measurement properties of existing tools was collected, providing readers with pertinent information to help decide which tools to use in evidence syntheses.

    • Screening of articles and data collection were performed by one author only, so it is possible that some relevant articles were missed, or that errors in data collection were made.

    • The search of grey literature was not comprehensive, so it is possible that there are other tools for assessing risk of reporting biases, and unpublished studies evaluating measurement properties, that were omitted from this review.

    Background 

    The credibility of evidence syntheses can be compromised by reporting biases, which arise when dissemination of research findings is influenced by the nature of the results.1 For example, there may be bias due to selective publication, where a study is only published if the findings are considered interesting (also known as publication bias).2 In addition, bias due to selective non-reporting may occur, where findings (eg, estimates of intervention efficacy or an association between exposure and outcome) that are statistically non-significant are not reported or are partially reported in a paper (eg, stating only that ‘P>0.05’).3 Alternatively, there may be bias in selection of the reported result, where authors perform multiple analyses for a particular outcome/association, yet only report the result which yielded the most favourable effect estimate.4 Evidence from cohorts of clinical trials followed from inception suggest that biased dissemination is common. Specifically, on average, half of all trials are not published,1 5 trials with statistically significant results are twice as likely to be published5 and a third of trials have outcomes that are omitted, added or modified between protocol and publication.6

    Audits of systematic review conduct suggest that most systematic reviewers do not assess risk of reporting biases.7–10 For example, in a cross-sectional study of 300 systematic reviews indexed in MEDLINE in February 2014,7 the risk of bias due to selective publication was not considered in 56% of reviews. A common reason for not doing so was that the small number of included studies, or inability to perform a meta-analysis, precluded the use of funnel plots. Only 19% of reviews included a search of a trial registry to identify completed but unpublished trials or prespecified but non-reported outcomes, and only 7% included a search of another source of data disseminated outside of journal articles. The risk of bias due to selective non-reporting in the included studies was assessed in only 24% of reviews.7 Another study showed that authors of Cochrane reviews routinely record whether any outcomes that were measured were not reported in the included trials, yet rarely consider if such non-reporting could have biased the results of a synthesis.11

    Previous researchers have summarised the characteristics of tools designed to assess various sources of bias in randomised trials,12–14 non-randomised studies of interventions (NRSI),14 15 diagnostic test accuracy studies16 and systematic reviews.14 17 Others have summarised the performance of statistical methods developed to detect or adjust for reporting biases.18–20 However, no prior review has focused specifically on tools (ie, structured instruments such as scales, checklists or domain-based tools) for assessing the risk of reporting biases. A particular challenge when assessing risk of reporting biases is that existing tools vary in their level of assessment. For example, tools for assessing risk of bias due to selective publication direct assessments at the level of the synthesis, whereas tools for assessing risk of bias due to selective non-reporting within studies can direct assessments at the level of the individual study, at the level of the synthesis or at both levels. It is unclear how many tools are available to assess different types of reporting bias, and what level they direct assessments at. It is also unclear whether criteria for reaching risk of bias judgements are consistent across existing tools. Therefore, the aim of this research was to conduct a systematic review of the content and measurement properties of such tools.

    Methods

    Protocol

    Methods for this systematic review were prespecified in a protocol which was uploaded to the Open Science Framework in February 2017 (https://osf.io/9ea22/).

    Eligibility criteria

    Papers were included if the authors described a tool that was designed for use by individuals performing evidence syntheses to assess risk of reporting biases in the included studies or in their synthesis of studies. Tools could assess any type of reporting bias, including bias due to selective publication, bias due to selective non-reporting or bias in selection of the reported result. Tools could assess the risk of reporting biases in any type of study (eg, randomised trial of intervention, diagnostic test accuracy study, observational study estimating prevalence of an exposure) and in any type of result (eg, estimate of intervention efficacy or harm, estimate of diagnostic accuracy, association between exposure and outcome). Eligible tools could take any form, including scales, checklists and domain-based tools. To be considered a scale, each item had to have a numeric score attached to it, so that an overall summary score could be calculated.12 To be considered a checklist, the tool had to include multiple questions, but the developers’ intention was not to attach a numerical score to each response, or to calculate an overall score.13 Domain-based tools were those that required users to judge risk of bias or quality within specific domains, and to record the information on which each judgement was based.21

    Tools with a broad scope, for example, to assess multiple sources of bias or the overall quality of the body of evidence, were eligible if one of the items covered risk of reporting bias. Multidimensional tools with a statistical component were also eligible (eg, those that require users to respond to a set of questions about the comprehensiveness of the search, as well as to perform statistical tests for funnel plot asymmetry). In addition, any studies that evaluated the measurement properties of existing tools (eg, construct validity, inter-rater agreement, time taken to complete assessments) were eligible for inclusion. Papers were eligible regardless of the date or format of publication, but were limited to those written in English.

    The following were ineligible:

    • articles or book chapters providing guidance on how to address reporting biases, but which do not include a structured tool that can be applied by users (eg, the 2011 Cochrane Handbook chapter on reporting biases22);

    • tools developed or modified for use in one particular systematic review;

    • tools designed to appraise published systematic reviews, such as the Risk Of Bias In Systematic reviews (ROBIS) tool23 or A MeaSurement Tool to Assess systematic Reviews (AMSTAR)24;

    • articles that focus on the development or evaluation of statistical methods to detect or adjust for reporting biases, as these have been reviewed elsewhere.18–20

    Search methods

    On 9 February 2017, one author (MJP) searched for potentially relevant records in Ovid MEDLINE (January 1946 to February 2017), Ovid Embase (January 1980 to February 2017) and Ovid PsycINFO (January 1806 to February 2017). The search strategies included terms relating to reporting bias which were combined with a search string used previously by Whiting et al to identify risk of bias/quality assessment tools17 (see full Boolean search strategies in online supplementary table S1).

    Supplementary file 1

    To capture any tools not published by formal academic publishers, we searched Google Scholar using the phrase ‘reporting bias tool OR risk of bias’. One author (MJP) screened the titles of the first 300 records, as recommended by Haddaway et al.25 To capture any papers that may have been missed by all searches, one author (MJP) screened the references of included articles. In April 2017, the same author emailed the list of included tools to 15 individuals with expertise in reporting biases and risk of bias assessment, and asked if they were aware of any other tools we had not identified.

    Study selection and data collection

    One author (MJP) screened all titles and abstracts retrieved by the searches. The same author screened any full-text articles retrieved. One author (MJP) collected data from included papers using a standardised data-collection form. The following data on included tools were collected:

    • type of tool (scale, checklist or domain-based tool);

    • types of reporting bias addressed by the tool;

    • level of assessment (ie, whether users direct assessments at the synthesis or at the individual studies included in the synthesis);

    • whether the tool is designed for general use (generic) or targets specific study designs or topic areas (specific);

    • items included in the tool;

    • how items within the tool are rated;

    • methods used to develop the tool (eg, Delphi study, expert consensus meeting);

    • availability of guidance to assist with completion of the tool (eg, guidance manual).

    The following data from studies evaluating measurement properties of an included tool were collected:

    • tool evaluated

    • measurement properties evaluated (eg, inter-rater agreement)

    • number of syntheses/studies evaluated

    • publication year of syntheses/studies evaluated

    • areas of healthcare addressed by syntheses/studies evaluated

    • number of assessors

    • estimate (and precision) of psychometric statistics (eg, weighted kappa; κ).

    Data analysis

    We summarised the characteristics of included tools in tables. We calculated the median (IQR) number of items across all tools, and tabulated the frequency of different criteria used in tools to denote a judgement of ‘high’ risk of reporting bias. We summarised estimates of psychometric statistics, such as weighted κ to estimate inter-rater agreement,26 by reporting the range of values across studies. For studies reporting weighted κ, we categorised agreement according to the system proposed by Landis and Koch,27 as poor (0.00), slight (0.01–0.20), fair (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80) or almost perfect (0.81–1.00).

    Results

    In total, 5554 records were identified from the searches, of which we retrieved 165 for full-text screening (figure 1). The inclusion criteria were met by 42 reports summarising 18 tools (table 1) and 17 studies evaluating the measurement properties of tools.3 4 21 28–66 A list of excluded papers is presented in online supplementary table S2. No additional tools were identified by the 15 experts contacted.

    Supplementary file 2

    View this table:
    Table 1

    List of included tools

    Figure 1
    Figure 1

    Flow diagram of identification, screening and inclusion of studies. aRecords identified from Ovid MEDLINE, Ovid Embase, Ovid PsycINFO and Google Scholar. bRecords identified from screening references of included articles. SR, systematic review.

    General characteristics of included tools

    Nearly all of the included tools (16/18; 89%) were domain-based, where users judge risk of bias or quality within specific domains (table 2; individual characteristics of each tool are presented in online supplementary table S3). All tools were designed for generic rather than specific use. Five tools focused solely on the risk of reporting biases3 28 29 47 48; the remainder addressed reporting biases and other sources of bias/methodological quality (eg, problems with randomisation, lack of blinding). Half of the tools (9/18; 50%) addressed only one type of reporting bias (eg, bias due to selective non-reporting only). Tools varied in regard to the study design that they assessed (ie, randomised trial, non-randomised study of an intervention, laboratory animal experiment). The publication year of the tools ranged from 1998 to 2016 (the earliest was the Downs-Black tool,31 a 27-item tool assessing multiple sources of bias, one of which focuses on risk of bias in the selection of the reported result).

    Supplementary file 3

    View this table:
    Table 2

    Summary of general characteristics of included tools

    Assessments for half of the tools (9/18; 50%) are directed at an individual study (eg, tool is used to assess whether any outcomes in a study were not reported). In 5/18 (28%) tools, assessments are directed at a specific outcome or result within a study (eg, tool is used to assess whether a particular outcome in a study, such as pain, was not reported). In a few tools (4/18; 22%), assessments are directed at a specific synthesis (eg, tool is used to assess whether a particular synthesis, such as a meta-analysis of studies examining pain as an outcome, is missing unpublished studies).

    The content of the included tools was informed by various sources of data. The most common included a literature review of items used in existing tools or a literature review of empirical evidence of bias (9/18; 50%), ideas generated at an expert consensus meeting (8/18; 44%) and pilot feedback on a preliminary version of the tool (7/18; 39%). The most common type of guidance available for the tools was a brief annotation per item/response option (9/18; 50%). A detailed guidance manual is available for four (22%) tools.

    Tool content

    Four tools include items for assessing risk of bias due to both selective publication and selective non-reporting.29 33 45 49 One of these tools (the AHRQ tool for evaluating the risk of reporting bias29) directs users to assess a particular synthesis, where a single risk of bias judgement is made based on information about unpublished studies and under-reported outcomes. In the other three tools (the GRADE framework, and two others which are based on GRADE),33 45 49 the different sources of reporting bias are assessed in separate domains (bias due to selective non-reporting is considered in a ‘study limitations (risk of bias)’ domain, while bias due to selective publication is considered in a ‘publication bias’ domain).

    Five tools21 28 43 44 47 guide users to assess risk of bias due to both selective non-reporting and selection of the reported result (ie, problems with outcomes/results that are not reported and those that are reported, respectively). Four of these tools, which include the Cochrane risk of bias tool for randomised trials21 and three others which are based on the Cochrane tool,43 44 47 direct assessments at the study level. That is, a whole study is rated at ‘high’ risk of reporting bias if any outcome/result in the study has been omitted, or fully reported, on the basis of the findings.

    Some of the tools designed to assess the risk of bias due to selective non-reporting ask users to assess, for particular outcomes of interest, whether the outcome was not reported or only partially reported in the study on the basis of its results (eg, Outcome Reporting Bias In Trials (ORBIT) tools,3 48 the AHRQ outcome reporting bias framework,28 and GRADE.34 This allows users to perform multiple outcome-level assessments of the risk of reporting bias (rather than one assessment for the study as a whole). In total, 15 tools include a mechanism for assessing risk of bias due to selective non-reporting in studies, but assessing the corresponding risk of bias in a synthesis that is missing the non-reported outcomes is not within the scope of 11 of these tools.3 21 28 30 38 43 44 47 48 51 52

    A variety of criteria are used in existing tools to inform a judgement of ‘high’ risk of bias due to selective publication (table 3), selective non-reporting (table 4), and selection of the reported result (table 5; more detail is provided in online supplementary table S4). In the four tools with an assessment of risk of bias due to selective publication, ‘high’ risk criteria include evidence of funnel plot asymmetry, discrepancies between published and unpublished studies, use of non-comprehensive searches and presence of small, ‘positive’ studies with for-profit interest (table 3). However, not all of these criteria appear in all tools (only evidence of funnel plot asymmetry does), and the relative weight assigned to each criterion in the overall risk of reporting bias judgement is clear for only one tool (the Semi-Automated Quality Assessment Tool; SAQAT).45 46

    Supplementary file 4

    View this table:
    Table 3

    Criteria used in existing tools to inform a judgement of ‘high’ risk of bias due to selective publication

    View this table:
    Table 4

    Criteria used in existing tools to inform a judgement of ‘high’ risk of bias due to selective non-reporting

    View this table:
    Table 5

    Criteria used in existing tools to inform a judgement of ‘high’ risk of bias in selection of the reported result

    All 15 tools with an assessment of the risk of bias due to selective non-reporting suggest that the risk of bias is ‘high’ when it is clear that an outcome was measured but no results were reported (table 4). Fewer of these tools (n=8; 53%) also recommend a ‘high’ risk judgement when results for an outcome are partially reported (eg, it is stated that the result was non-significant, but no effect estimate or summary statistics are presented).

    The eight tools that include an assessment of the risk of bias in selection of the reported result recommend various criteria for a ‘high’ risk judgement (table 5). These include when some outcomes that were not prespecified are added post hoc (in 4 (50%) tools), or when it is likely that the reported result for a particular outcome has been selected, on the basis of the findings, from among multiple outcome measurements or analyses within the outcome domain (in 2 (25%) tools).

    General characteristics of studies evaluating measurement properties of included tools

    Despite identifying 17 studies that evaluated measurement properties of an included tool, psychometric statistics for the risk of reporting bias component were available only from 12 studies43 44 54–60 62 64 66 (the other five studies include only data on properties of the multidimensional tool as a whole31 53 61 63 65; online supplementary table S5). Nearly all 12 studies (11; 92%) evaluated inter-rater agreement between two assessors; eight of these studies reported weighted κ values, but only two described the weighting scheme.55 62 Eleven studies43 44 54–60 64 66 evaluated the measurement properties of tools for assessing risk of bias in a study due to selective non-reporting or risk of bias in selection of the reported result; in these 11 studies, a median of 40 (IQR 32–109) studies were assessed. One study62 evaluated a tool for assessing risk of bias in a synthesis due to selective publication, in which 44 syntheses were assessed. In the studies evaluating inter-rater agreement, all involved two assessors.

    Supplementary file 5

    Results of evaluation studies

    Five studies54 56–58 60 included data on the inter-rater agreement of assessments of risk of bias due to selective non-reporting using the Cochrane risk of bias tool for randomised trials21 (table 6). Weighted κ values in four studies54 56–58 ranged from 0.13 to 0.50 (sample size ranged from 87 to 163 studies), suggesting slight to moderate agreement.27 In the other study,60 the per cent agreement in selective non-reporting assessments in trials that were included in two different Cochrane reviews was low (43% of judgements were in agreement). Two other studies found that inter-rater agreement of selective non-reporting assessments were substantial for SYRCLE’s RoB tool (κ=0.62, n=32),43 but poor for the RoBANS tool (κ=0, n=39).44 There was substantial agreement between raters in the assessment of risk of bias due to selective publication using the SAQAT (κ=0.63, n=29).62 The inter-rater agreement of assessments of risk of bias in selection of the reported result using the ROBINS-I tool4 was moderate for NRSI included in a review of the effect of cyclooxygenase-2 inhibitors on cardiovascular events (κ=0.45, n=21), and substantial for NRSI included in a review of the effect of thiazolidinediones on cardiovascular events (κ=0.78, n=16).55

    View this table:
    Table 6

    Reported measurement properties of tools with an assessment of the risk of reporting bias

    Discussion

    From a systematic search of the literature, we identified 18 tools designed for use by individuals performing evidence syntheses to assess risk of reporting biases in the included studies or in their synthesis of studies. The tools varied with regard to the type of reporting bias assessed (eg, bias due to selective publication, bias due to selective non-reporting), and the level of assessment (eg, for the study as a whole, a particular outcome within a study or a particular synthesis of studies). Various criteria are used across tools to designate a synthesis as being at ‘high’ risk of bias due to selective publication (eg, evidence of funnel plot asymmetry, use of non-comprehensive searches). However, the relative weight assigned to each criterion in the overall judgement is not clear for most of these tools. Tools for assessing risk of bias due to selective non-reporting guide users to assess a study, or an outcome within a study, as ‘high’ risk of bias if no results are reported for an outcome. However, assessing the corresponding risk of bias in a synthesis that is missing the non-reported outcomes is outside the scope of most of these tools. Inter-rater agreement estimates were available for five tools,4 21 43 44 62 and ranged from poor to substantial; however, the sample sizes of most evaluations were small, and few described the weighting scheme used to calculate κ.

    Strengths and limitations

    There are several strengths of this research. Methods were conducted in accordance with a systematic review protocol (https://osf.io/9ea22/). Published articles were identified by searching several relevant databases using a search string developed in conjunction with an information specialist,17 and by contacting experts to identify tools missed by the search. Detailed information on the content and measurement properties of existing tools was collected, providing readers with pertinent information to help decide which tools to use in future reviews. However, the findings need to be considered in light of some limitations. Screening of articles and data collection were performed by one author only. It is therefore possible that some relevant articles were missed, or that errors in data collection were made. The search for unpublished tools was not comprehensive (only Google Scholar was searched), so it is possible that other tools for assessing risk of reporting biases exist. Further, restricting the search to articles in English was done to expedite the review process, but may have resulted in loss of information about tools written in other languages, and additional evidence on measurement properties of tools.

    Comparison with other studies

    Other systematic reviews of risk of bias tools12–17 have restricted inclusion to tools developed for particular study designs (eg, randomised trials, diagnostic test accuracy studies), where the authors recorded all the sources of bias addressed. A different approach was taken in the current review, where all tools (regardless of study design) that address a particular source of bias were examined. By focusing on one source of bias only, the analysis of included items and criteria for risk of bias judgements was more detailed than that recorded previously. Some of the existing reviews of tools15 considered tools that were developed or modified in the context of a specific systematic review. However, such tools were excluded from the current review as they are unlikely to have been developed systematically,15 67 and are difficult to find (all systematic reviews conducted during a particular period would need to have been examined for the search to be considered exhaustive).

    Explanations and implications

    Of the 18 tools identified, only four (22%) included a mechanism for assessing risk of bias due to selective publication, which is the type of reporting bias that has been investigated by methodologists most often.2 This is perhaps unsurprising given that hundreds of statistical methods to ‘detect’ or ‘adjust’ for bias due to selective publication have been developed.18 These statistical methods may be considered by methodologists and systematic reviewers as the tools of choice for assessing this type of bias. However, application of these statistical methods without considering other factors (eg, existence of registered but unpublished studies, conflicts of interest that may influence investigators to not disseminate studies with unfavourable results) is not sufficiently comprehensive, and could lead to incorrect conclusions about the risk of bias due to selective publication. Further, there are many limitations of these statistical approaches, in terms of their underlying assumptions, statistical power, which is often low because most meta-analyses include few studies,7 and the need for specialist statistical software to apply them.19 68 These factors may have limited their use in practice and potentially explain why a large number of systematic reviewers currently ignore the risk of bias due to selective publication.7–9 69

    Our analysis suggests that the factors that need to be considered to assess risk of reporting biases adequately (eg, comprehensiveness of the search, amount of data missing from the synthesis due to unpublished studies and under-reported outcomes) are fragmented. A similar problem was occurring a decade ago with the assessment of risk of bias in randomised trials. Some authors assessed only problems with randomisation, while others focused on whether trials were not ‘double blinded’ or had any missing participant data.70 It was not until all the important bias domains were brought together into a structured, domain-based tool to assess the risk of bias in randomised trials,21 that systematic reviewers started to consider risk of bias in trials comprehensively. A similar initiative to link all the components needed to judge the risk of reporting biases into a comprehensive new tool may improve the credibility of evidence syntheses.

    In particular, there is an emergent need for a new tool to assess the risk that a synthesis is affected by reporting biases. This tool could guide users to consider risk of bias in a synthesis due to both selective publication and selective non-reporting, given that both practices lead to the same consequence: evidence missing from the synthesis.11 Such a tool would complement recently developed tools for assessing risk of bias within studies (RoB 2.041 and ROBINS-I4 which include a domain for assessing the risk of bias in selection of the reported result, but no mechanism to assess risk of bias due to selective non-reporting). Careful thought would need to be given as to how to weigh up various pieces of information underpinning the risk of bias judgement. For example, users will need guidance on how evidence of known, unpublished studies (as identified from trial registries, protocols or regulatory documents) should be considered alongside evidence that is more speculative (eg, funnel plots suggesting that studies may be missing). Further, guidance for the tool will need to emphasise the value of seeking documents other than published journal articles (eg, protocols) to inform risk of bias judgements. Preparation of a detailed guidance manual may enhance the usability of the tool, minimise misinterpretation and increase reliability in assessments. Once developed, evaluations of the measurement properties of the tool, such as inter-rater agreement and construct validity, should be conducted to explore whether modifications to the tool are necessary.

    Conclusions

    There are several limitations of existing tools for assessing risk of reporting biases in studies or syntheses of studies, in terms of their scope, guidance for reaching risk of bias judgements and measurement properties. Development and evaluation of a new, comprehensive tool could help overcome present limitations.

    References

    1. 1.↵
      2. Chan AW ,
      3. Song F ,
      4. Vickers A , et al
      . Increasing value and reducing waste: addressing inaccessible research. Lancet 2014;383:257–66.doi:10.1016/S0140-6736(13)62296-5
      OpenUrlCrossRefPubMedWeb of Science
    2. 2.↵
      2. Song F ,
      3. Parekh S ,
      4. Hooper L , et al
      . Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess 2010;14:8.doi:10.3310/hta14080
      OpenUrl
    3. 3.↵
      2. Kirkham JJ ,
      3. Dwan KM ,
      4. Altman DG , et al
      . The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 2010;340:c365.doi:10.1136/bmj.c365
      OpenUrlAbstract/FREE Full Text
    4. 4.↵
      2. Sterne JA ,
      3. Hernán MA ,
      4. Reeves BC , et al
      . ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016;355:i4919.doi:10.1136/bmj.i4919
      OpenUrlFREE Full Text
    5. 5.↵
      2. Schmucker C ,
      3. Schell LK ,
      4. Portalupi S , et al
      . Extent of non-publication in cohorts of studies approved by research ethics committees or included in trial registries. PLoS One 2014;9:e114023.doi:10.1371/journal.pone.0114023
    6. 6.↵
      2. Jones CW ,
      3. Keil LG ,
      4. Holland WC , et al
      . Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med 2015;13:282.doi:10.1186/s12916-015-0520-3
      OpenUrlCrossRefPubMed
    7. 7.↵
      2. Page MJ ,
      3. Shamseer L ,
      4. Altman DG , et al
      . Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med 2016;13:e1002028.doi:10.1371/journal.pmed.1002028
    8. 8.↵
      2. Koletsi D ,
      3. Valla K ,
      4. Fleming PS , et al
      . Assessment of publication bias required improvement in oral health systematic reviews. J Clin Epidemiol 2016;76:118–24.doi:10.1016/j.jclinepi.2016.02.019
      OpenUrl
    9. 9.↵
      2. Hedin RJ ,
      3. Umberham BA ,
      4. Detweiler BN , et al
      . Publication Bias and Nonreporting Found in Majority of Systematic Reviews and Meta-analyses in Anesthesiology Journals. Anesth Analg 2016;123:1018–25.doi:10.1213/ANE.0000000000001452
      OpenUrl
    10. 10.↵
      2. Ziai H ,
      3. Zhang R ,
      4. Chan AW , et al
      . Search for unpublished data by systematic reviewers: an audit. BMJ Open 2017;7:e017737.doi:10.1136/bmjopen-2017-017737
    11. 11.↵
      2. Page MJ ,
      3. Higgins JP
      . Rethinking the assessment of risk of bias due to selective reporting: a cross-sectional study. Syst Rev 2016;5:108.doi:10.1186/s13643-016-0289-2
      OpenUrl
    12. 12.↵
      2. Moher D ,
      3. Jadad AR ,
      4. Nichol G , et al
      . Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials 1995;16:62–73.doi:10.1016/0197-2456(94)00031-W
      OpenUrlCrossRefPubMedWeb of Science
    13. 13.↵
      2. Olivo SA ,
      3. Macedo LG ,
      4. Gadotti IC , et al
      . Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther 2008;88:156–75.doi:10.2522/ptj.20070147
      OpenUrlAbstract/FREE Full Text
    14. 14.↵
      2. Bai A ,
      3. Shukla VK ,
      4. Bak G , et al
      . Quality Assessment Tools Project Report. Ottawa: Canadian Agency for Drugs and Technologies in Health, 2012.
    15. 15.↵
      2. Sanderson S ,
      3. Tatt ID ,
      4. Higgins JP
      . Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol 2007;36:666–76.doi:10.1093/ije/dym018
      OpenUrlCrossRefPubMedWeb of Science
    16. 16.↵
      2. Whiting P ,
      3. Rutjes AW ,
      4. Dinnes J , et al
      . A systematic review finds that diagnostic reviews fail to incorporate quality despite available tools. J Clin Epidemiol 2005;58:1–12.doi:10.1016/j.jclinepi.2004.04.008
      OpenUrlCrossRefPubMedWeb of Science
    17. 17.↵
      2. Whiting P ,
      3. Davies P ,
      4. Savovic J , et al
      . Evidence to inform the development of ROBIS, a new tool to assess the risk of bias in systematic reviews, September 2013. 2013 https://www.researchgate.net/publication/303312018_Evidence_to_inform_the_development_of_ROBIS_a_new_tool_to_assess_the_risk_of_bias_in_systematic_reviews (accessed 1 Aug 2017).
    18. 18.↵
      2. Mueller KF ,
      3. Meerpohl JJ ,
      4. Briel M , et al
      . Methods for detecting, quantifying, and adjusting for dissemination bias in meta-analysis are described. J Clin Epidemiol 2016;80:25–33.doi:10.1016/j.jclinepi.2016.04.015
      OpenUrl
    19. 19.↵
      2. Jin ZC ,
      3. Zhou XH ,
      4. He J
      . Statistical methods for dealing with publication bias in meta-analysis. Stat Med 2015;34:343–60.doi:10.1002/sim.6342
      OpenUrlCrossRefPubMed
    20. 20.↵
      2. Sterne JA ,
      3. Sutton AJ ,
      4. Ioannidis JP , et al
      . Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ 2011;343:d4002.doi:10.1136/bmj.d4002
      OpenUrlFREE Full Text
    21. 21.↵
      2. Higgins JP ,
      3. Altman DG ,
      4. Gøtzsche PC , et al
      . The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928.doi:10.1136/bmj.d5928
      OpenUrlFREE Full Text
    22. 22.↵
      2. Sterne JAC ,
      3. Egger M ,
      4. Moher D
      . Chapter 10: Addressing reporting biases. In: Higgins JPT , Green S , eds. Cochrane handbook for systematic reviews of interventions Version 510. Chichester, UK: John Wiley & Sons, 2011.
    23. 23.↵
      2. Whiting P ,
      3. Savović J ,
      4. Higgins JP , et al
      . ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 2016;69:225–34.doi:10.1016/j.jclinepi.2015.06.005
      OpenUrlCrossRefPubMed
    24. 24.↵
      2. Shea BJ ,
      3. Grimshaw JM ,
      4. Wells GA , et al
      . Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10.doi:10.1186/1471-2288-7-10
      OpenUrlCrossRefPubMed
    25. 25.↵
      2. Haddaway NR ,
      3. Collins AM ,
      4. Coughlin D , et al
      . The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching. PLoS One 2015;10:e0138237.doi:10.1371/journal.pone.0138237
    26. 26.↵
      2. Cohen J
      . A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:37–46.doi:10.1177/001316446002000104
      OpenUrlCrossRefWeb of Science
    27. 27.↵
      2. Landis JR ,
      3. Koch GG
      . The measurement of observer agreement for categorical data. Biometrics 1977;33:159–74.
      OpenUrlCrossRefPubMedWeb of Science
    28. 28.↵
      2. Balshem H ,
      3. Stevens A ,
      4. Ansari M , et al
      . Finding grey literature evidence and assessing for outcome and analysis reporting biases when comparing medical interventions: AHRQ and the Effective Health Care Program. (Prepared by the Oregon Health and Science University and the University of Ottawa Evidence-based Practice Centers under Contract Nos. 290-2007-10057-I and 290-2007-10059-I.) AHRQ Publication No. 13(14)-EHC096-EF. Rockville, MD: Agency for Healthcare Research and Quality, 2013.
    29. 29.↵
      2. Berkman ND ,
      3. Lohr KN ,
      4. Ansari M , et al
      . Chapter 15 Appendix A: A Tool for Evaluating the Risk of Reporting Bias (in Chapter 15: Grading the Strength of a Body of Evidence When Assessing Health Care Interventions for the Effective Health Care Program of the Agency for Healthcare Research and Quality: An Update). Methods Guide for Comparative Effectiveness Reviews (Prepared by the RTI-UNC Evidence-based Practice Center under Contract No. 290-2007-10056-I). AHRQ Publication No. 13(14)-EHC130-EF. Rockville, MD: Agency for Healthcare Research and Quality, 2013.
    30. 30.↵
      2. Downes MJ ,
      3. Brennan ML ,
      4. Williams HC , et al
      . Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS). BMJ Open 2016;6:e011458.doi:10.1136/bmjopen-2016-011458
    31. 31.↵
      2. Downs SH ,
      3. Black N
      . The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health 1998;52:377–84.
      OpenUrlAbstract
    32. 32.↵
      2. Dwan K ,
      3. Gamble C ,
      4. Kolamunnage-Dona R , et al
      . Assessing the potential for outcome reporting bias in a review: a tutorial. Trials 2010;11:52.doi:10.1186/1745-6215-11-52
    33. 33.↵
      2. Guyatt GH ,
      3. Oxman AD ,
      4. Vist GE , et al
      . GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924–6.doi:10.1136/bmj.39489.470347.AD
      OpenUrlFREE Full Text
    34. 34.↵
      2. Guyatt GH ,
      3. Oxman AD ,
      4. Vist G , et al
      . GRADE guidelines: 4. Rating the quality of evidence--study limitations (risk of bias). J Clin Epidemiol 2011;64:407–15.doi:10.1016/j.jclinepi.2010.07.017
      OpenUrlCrossRefPubMedWeb of Science
    35. 35.↵
      2. Guyatt GH ,
      3. Oxman AD ,
      4. Montori V , et al
      . GRADE guidelines: 5. Rating the quality of evidence--publication bias. J Clin Epidemiol 2011;64:1277–82.doi:10.1016/j.jclinepi.2011.01.011
      OpenUrlCrossRefPubMed
    36. 36.↵
      2. Schünemann H ,
      3. Brożek J ,
      4. Guyatt G , et al
      . Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach. http://gdt.guidelinedevelopment.org/app/handbook/handbook.html.
    37. 37.↵
      2. Santesso N ,
      3. Carrasco-Labra A ,
      4. Langendam M , et al
      . Improving GRADE evidence tables part 3: detailed guidance for explanatory footnotes supports creating and understanding GRADE certainty in the evidence judgments. J Clin Epidemiol 2016;74.doi:10.1016/j.jclinepi.2015.12.006
    38. 38.↵
      2. Hayden JA ,
      3. van der Windt DA ,
      4. Cartwright JL , et al
      . Assessing bias in studies of prognostic factors. Ann Intern Med 2013;158:280–6.doi:10.7326/0003-4819-158-4-201302190-00009
      OpenUrlCrossRefPubMedWeb of Science
    39. 39.↵
      2. Higgins JPT ,
      3. Altman DG ,
      4. Sterne JAC
      . Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT , Green S , eds. Cochrane Handbook for Systematic Reviews of Interventions. Chichester (UK: John Wiley & Sons, 2008:187–241.
    40. 40.↵
      2. Higgins JPT ,
      3. Altman DG ,
      4. Sterne JAC
      . Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT , Green S , eds. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. London: The Cochrane Collaboration,  2011.
    41. 41.↵
      2. Higgins JPT ,
      3. Savović J ,
      4. Page MJ , et al
      . Revised Cochrane risk of bias tool for randomized trials (RoB 2.0), Version 20. 2016 https://sites.google.com/site/riskofbiastool/ (accessed 19 Sep 2017).
    42. 42.↵
      2. Higgins JPT ,
      3. Sterne JAC ,
      4. Savović J , et al
      . A revised tool for assessing risk of bias in randomized trials. Cochrane Methods Cochrane Database of Systematic Reviews 2016;10(Suppl 1):29–31.
      OpenUrl
    43. 43.↵
      2. Hooijmans CR ,
      3. Rovers MM ,
      4. de Vries RB , et al
      . SYRCLE’s risk of bias tool for animal studies. BMC Med Res Methodol 2014;14:43.doi:10.1186/1471-2288-14-43
      OpenUrlCrossRefPubMed
    44. 44.↵
      2. Kim SY ,
      3. Park JE ,
      4. Lee YJ , et al
      . Testing a tool for assessing the risk of bias for nonrandomized studies showed moderate reliability and promising validity. J Clin Epidemiol 2013;66:408–14.doi:10.1016/j.jclinepi.2012.09.016
      OpenUrlCrossRefPubMed
    45. 45.↵
      2. Meader N ,
      3. King K ,
      4. Llewellyn A , et al
      . A checklist designed to aid consistency and reproducibility of GRADE assessments: development and pilot validation. Syst Rev 2014;3:82.doi:10.1186/2046-4053-3-82
      OpenUrl
    46. 46.↵
      2. Stewart GB ,
      3. Higgins JP ,
      4. Schünemann H , et al
      . The use of Bayesian networks to assess the quality of evidence from research synthesis: 1. PLoS One 2015;10:e0114497.doi:10.1371/journal.pone.0114497
    47. 47.↵
      2. Reid EK ,
      3. Tejani AM ,
      4. Huan LN , et al
      . Managing the incidence of selective reporting bias: a survey of Cochrane review groups. Syst Rev 2015;4:85.doi:10.1186/s13643-015-0070-y
      OpenUrlCrossRefPubMed
    48. 48.↵
      2. Saini P ,
      3. Loke YK ,
      4. Gamble C , et al
      . Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews. BMJ 2014;349:g6501.
      OpenUrlAbstract/FREE Full Text
    49. 49.↵
      2. Salanti G ,
      3. Del Giovane C ,
      4. Chaimani A , et al
      . Evaluating the quality of evidence from a network meta-analysis. PLoS One 2014;9:e99682.doi:10.1371/journal.pone.0099682
    50. 50.↵
      2. Higgins JP ,
      3. Del Giovane C ,
      4. Chaimani A , et al
      . Evaluating the Quality of Evidence from a Network Meta-Analysis. Value Health 2014;17:A324.doi:10.1016/j.jval.2014.08.572
      OpenUrl
    51. 51.↵
      2. Viswanathan M ,
      3. Berkman ND
      . Development of the RTI item bank on risk of bias and precision of observational studies. J Clin Epidemiol 2012;65:163–78.doi:10.1016/j.jclinepi.2011.05.008
      OpenUrlCrossRefPubMed
    52. 52.↵
      2. Viswanathan M ,
      3. Berkman ND ,
      4. Dryden DM , et al
      . AHRQ Methods for Effective Health Care. Assessing Risk of Bias and Confounding in Observational Studies of Interventions or Exposures: Further Development of the RTI Item Bank. Rockville (MD: Agency for Healthcare Research and Quality (US), 2013.
    53. 53.↵
      2. Armijo-Olivo S ,
      3. Stiles CR ,
      4. Hagen NA , et al
      . Assessment of study quality for systematic reviews: a comparison of the Cochrane Collaboration Risk of Bias Tool and the Effective Public Health Practice Project Quality Assessment Tool: methodological research. J Eval Clin Pract 2012;18:12–18.doi:10.1111/j.1365-2753.2010.01516.x
      OpenUrlCrossRefPubMed
    54. 54.↵
      2. Armijo-Olivo S ,
      3. Ospina M ,
      4. da Costa BR , et al
      . Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials. PLoS One 2014;9:e96920.doi:10.1371/journal.pone.0096920
    55. 55.↵
      2. Bilandzic A ,
      3. Fitzpatrick T ,
      4. Rosella L , et al
      . Risk of Bias in Systematic Reviews of Non-Randomized Studies of Adverse Cardiovascular Effects of Thiazolidinediones and Cyclooxygenase-2 Inhibitors: Application of a New Cochrane Risk of Bias Tool. PLoS Med 2016;13:e1001987.doi:10.1371/journal.pmed.1001987
      OpenUrlPubMed
    56. 56.↵
      2. Hartling L ,
      3. Ospina M ,
      4. Liang Y , et al
      . Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ 2009;339:b4012.
      OpenUrlAbstract/FREE Full Text
    57. 57.↵
      2. Hartling L ,
      3. Bond K ,
      4. Vandermeer B , et al
      . Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma. PLoS One 2011;6:e17242.doi:10.1371/journal.pone.0017242
    58. 58.↵
      2. Hartling L ,
      3. Hamm M ,
      4. Milne A , et al
      . AHRQ Methods for Effective Health Care. Validity and Inter-Rater Reliability Testing of Quality Assessment Instruments. Rockville (MD: Agency for Healthcare Research and Quality (US), 2012.
    59. 59.↵
      2. Hartling L ,
      3. Hamm MP ,
      4. Milne A , et al
      . Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol 2013;66:973–81.doi:10.1016/j.jclinepi.2012.07.005
      OpenUrlCrossRefPubMed
    60. 60.↵
      2. Jordan VM ,
      3. Lensen SF ,
      4. Farquhar CM
      . There were large discrepancies in risk of bias tool judgments when a randomized controlled trial appeared in more than one systematic review. J Clin Epidemiol 2017;81:72–6.doi:10.1016/j.jclinepi.2016.08.012
      OpenUrl
    61. 61.↵
      2. Kumar A ,
      3. Miladinovic B ,
      4. Guyatt GH , et al
      . GRADE guidelines system is reproducible when instructions are clearly operationalized even among the guidelines panel members with limited experience with GRADE. J Clin Epidemiol 2016;75:115–8.doi:10.1016/j.jclinepi.2015.11.020
      OpenUrl
    62. 62.↵
      2. Llewellyn A ,
      3. Whittington C ,
      4. Stewart G , et al
      . The Use of Bayesian Networks to Assess the Quality of Evidence from Research Synthesis: 2. Inter-Rater Reliability and Comparison with Standard GRADE Assessment. PLoS One 2015;10:e0123511.doi:10.1371/journal.pone.0123511
    63. 63.↵
      2. Mustafa RA ,
      3. Santesso N ,
      4. Brozek J , et al
      . The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses. J Clin Epidemiol 2013;66:736–42.doi:10.1016/j.jclinepi.2013.02.004
      OpenUrlCrossRefPubMed
    64. 64.↵
      2. Norris SL ,
      3. Holmer HK ,
      4. Ogden LA , et al
      . AHRQ Methods for Effective Health Care. Selective Outcome Reporting as a Source of Bias in Reviews of Comparative Effectiveness. Rockville (MD): Agency for Healthcare Research and Quality (US), 2012.
    65. 65.↵
      2. O’Connor SR ,
      3. Tully MA ,
      4. Ryan B , et al
      . Failure of a numerical quality assessment scale to identify potential risk of bias in a systematic review: a comparison study. BMC Res Notes 2015;8:224.doi:10.1186/s13104-015-1181-1
      OpenUrl
    66. 66.↵
      2. Vale CL ,
      3. Tierney JF ,
      4. Burdett S
      . Can trial quality be reliably assessed from published reports of cancer trials: evaluation of risk of bias assessments in systematic reviews. BMJ 2013;346:f1798.doi:10.1136/bmj.f1798
      OpenUrlAbstract/FREE Full Text
    67. 67.↵
      2. Whiting PF ,
      3. Rutjes AW ,
      4. Westwood ME , et al
      . A systematic review classifies sources of bias and variation in diagnostic test accuracy studies. J Clin Epidemiol 2013;66:1093–104.doi:10.1016/j.jclinepi.2013.05.014
      OpenUrlCrossRefPubMed
    68. 68.↵
      2. Sterne JAC ,
      3. Egger M ,
      4. Moher D , et al
      . Chapter 10: Addressing reporting biases. In: Higgins JPT , Churchill R , Chandler J , eds. Cochrane Handbook for Systematic Reviews of Interventions version 5.2.0. Chichester, UK: The Cochrane Collaboration, 2017.
    69. 69.↵
      2. Atakpo P ,
      3. Vassar M
      . Publication bias in dermatology systematic reviews and meta-analyses. J Dermatol Sci 2016;82:69–74.doi:10.1016/j.jdermsci.2016.02.005
      OpenUrl
    70. 70.↵
      2. Lundh A ,
      3. Gøtzsche PC
      . Recommendations by Cochrane Review Groups for assessment of the risk of bias in studies. BMC Med Res Methodol 2008;8:22.doi:10.1186/1471-2288-8-22
      OpenUrlCrossRefPubMed

    Footnotes

    • Contributors MJP conceived and designed the study, collected data, analysed the data and wrote the first draft of the article. JEM and JPTH provided input on the study design and contributed to revisions of the article. All authors approved the final version of the submitted article.

    • Funding MJP is supported by an Australian National Health and Medical Research Council (NHMRC) Early Career Fellowship (1088535). JEM is supported by an NHMRC Australian Public Health Fellowship (1072366). JPTH is funded in part by Cancer Research UK Programme Grant C18281/A19169; is a member of the MRC Integrative Epidemiology Unit at the University of Bristol, which is supported by the UK Medical Research Council and the University of Bristol (grant MC_UU_12013/9); and is a member of the MRC ConDuCT-II Hub (Collaboration and innovation for Difficult and Complex randomised controlled Trials In Invasive procedures; grant MR/K025643/1).

    • Competing interests JPTH led or participated in the development of four of the included tools (the current Cochrane risk of bias tool for randomised trials, the RoB 2.0 tool for assessing risk of bias in randomised trials, the ROBINS-I tool for assessing risk of bias in non-randomised studies of interventions and the framework for assessing quality of evidence from a network meta-analysis). MJP participated in the development of one of the included tools (the RoB 2.0 tool for assessing risk of bias in randomised trials). All authors are participating in the development of a new tool for assessing risk of reporting biases in systematic reviews.

    • Patient consent Not required.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement The study protocol, data collection form, and the raw data and statistical analysis code for this study are available on the Open Science Framework: https://osf.io/3jdaa/