Reported measurement properties of tools with an assessment of the risk of reporting bias
| Study ID | Tool | Measurement property | Sample size | Areas of healthcare addressed | Weighted kappa (95% CI) | Weighting scheme | Interpretation of kappa* |
| Armijo-Olivo et al 54 | RoB 1.0 | Inter-rater agreement of assessments of risk of bias due to selective non-reporting (between two external reviewers) | 87 | Musculoskeletal, cardiorespiratory, neurological and gynaecological conditions. | 0.5 (CI not reported) | Not described | Moderate agreement |
| Armijo-Olivo et al 54 | RoB 1.0 | Inter-rater agreement of assessments of risk of bias due to selective non-reporting (between two external reviewers and Cochrane reviewers) | 87 | See above | 0.13 (CI not reported) | Not described | Slight agreement |
| Hartling et al 56 | RoB 1.0 | Inter-rater agreement of assessments of risk of bias due to selective non-reporting | 163 | Child health | 0.13 (95% CI −0.05 to 0.31) | Not described | Slight agreement |
| Hartling et al 57 | RoB 1.0 | Inter-rater agreement of assessments of risk of bias due to selective non-reporting | 107 | Asthma | 0.4 (95% CI 0.14 to 0.67) | Not described | Fair agreement |
| Hartling et al 58 59 | RoB 1.0 | Inter-rater agreement of assessments of risk of bias due to selective non-reporting (between two reviewers, all trials) | 124 | Varied | 0.27 (95% CI 0.06 to 0.49) | Not described | Fair agreement |
| Hartling et al 58 59 | RoB 1.0 | Inter-rater agreement of assessments of risk of bias due to selective non-reporting (between pairs of reviewers across different centres, all trials) | 30 | Varied | 0.08 (95% CI −0.09 to 0.26) | Not described | Slight agreement |
| Jordan et al 60 | RoB 1.0 | Inter-rater agreement of assessments of risk of bias due to selective non-reporting (between judgements of trials appearing in two SRs) | 28 | Subfertility | Not reported† | Not applicable | Not applicable |
| Vale et al 66 | RoB 1.0 | Agreement between selective non-reporting assessments performed using published article only versus published article and data collected during the individual participant data process | 95 | Cancer pain | Not reported† | Not applicable | Not applicable |
| Hoojimans et al 43 | SYRCLE RoB | Inter-rater agreement of assessments of risk of bias due to selective non-reporting | 32 | Animal studies (not specified) | 0.62 (CI not reported) | Not described | Substantial agreement |
| Kim et al 44 | RoBANS | Inter-rater agreement of assessments of risk of bias due to selective non-reporting | 39 | Depression, myocardial infarction, postpartum haemorrhage, chronic non-cancer pain | 0 (CI not reported) | Not described | Poor agreement |
| Llewellyn et al 62 | SAQAT | Inter-rater agreement of assessments of risk of bias due to selective publication (between two SAQAT raters) | 29 | Varied | 0.63 (95% CI 0.17 to 1) | Quadratic | Substantial agreement |
| Llewellyn et al 62 | SAQAT | Inter-rater agreement of assessments of risk of bias due to selective publication (between one rater using SAQAT and one using the standard GRADE approach) | 15 | Varied | Not reported† | Not applicable | Not applicable |
| Norris et al 64 | ORBIT-I | Inter-rater agreement of ORBIT-I classifications of risk of bias due to selective non-reporting | 40 | Varied | Not calculated, as too little variation in judgements | Not applicable | Not applicable |
| Bilandzic et al 55 | ROBINS-I | Inter-rater agreement of assessments of risk of bias in selection of the reported result | 16 | Thiazolidinediones and cardiovascular events | 0.78 (CI not reported) | Linear | Substantial agreement |
| Bilandzic et al 55 | ROBINS-I | Inter-rater agreement of assessments of risk of bias in selection of the reported result | 21 | COX-2 inhibitors and cardiovascular events | 0.45 (CI not reported) | Linear | Moderate agreement |
*Interpretation of kappa based on categorisation system defined by Landis et al.27
†Data presented as per cent agreement, not weighted kappa.
COX-2, cyclooxygenase-2; ORBIT-I, Outcome Reporting Bias In Trials classification system for benefit outcomes3 32; RoB 1.0, Cochrane risk of bias tool for randomised trials21 39 40; RoBANS, Risk of Bias Assessment Tool for Nonrandomized Studies44; ROBINS-I, Risk Of Bias In Non-randomized Studies of Interventions tool4; SAQAT, Semi-Automated Quality Assessment Tool45 46; SRs, systematic reviews; SYRCLE RoB, SYstematic Review Centre for Laboratory animal Experimentation risk of bias tool.43