Table 3

Criteria used in existing tools to inform a judgement of ‘high’ risk of bias due to selective publication

‘High’ risk of bias criteria proposed in existing toolsAHRQ RRBGRADESAQATNMA-QualityTotal, n (%)
Assessment directed at a specific synthesis (eg, meta-analysis)
 Evidence of funnel plot asymmetry (based on visual inspection of funnel plot or statistical test for funnel plot asymmetry)4 (100)
 Smaller studies tend to demonstrate more favourable results (based on visual assessment, without funnel plot)1 (25)
 Clinical decision would differ for estimates from a fixed-effect versus a random-effects model because the findings from a fixed-effect model are closer to the null1 (25)
 Substantial heterogeneity in the meta-analysis cannot be explained by some clinical or methodological factor1 (25)
 At least one study is affected by non-publication or non-accessibility1 (25)
 Presence of small (often ‘positive’) studies with for-profit interest in the synthesis2 (50)
 Presence of early studies (ie, set of small, ‘positive’ trials addressing a novel therapy) in the synthesis2 (50)
 Discrepancy in findings between published and unpublished trials3 (75)
 Search strategies were not comprehensive3 (75)
 Methods to identify all available evidence were not comprehensive2 (50)
 Grey literature were not searched1 (25)
 Restrictions to study selection on the basis of language were applied1 (25)
 Industry influence may apply to studies included in the synthesis1 (25)
  • AHRQ RRB, AHRQ tool for evaluating the risk of reporting bias29; GRADE, GRADE rating of quality of evidence34–37; NMA-Quality, Framework for evaluating the quality of evidence from a network meta-analysis49; SAQAT, Semi-Automated Quality Assessment Tool.45 46