Crude and adjusted reliability for evaluating consultations for different numbers of raters using GCRS (transformed 0–10 data)
| Number of raters | Crude reliability* (95% CI) | Reliability adjusted for evaluator bias* (95% CI) |
|---|---|---|
| 1 | 0.58 (0.28 to 0.66) | 0.65 (0.36 to 0.71) |
| 2 | 0.73 (0.44 to 0.79) | 0.78 (0.53 to 0.83) |
| 3 | 0.80 (0.54 to 0.85) | 0.85 (0.63 to 0.88) |
| 4 | 0.85 (0.61 to 0.88) | 0.88 (0.69 to 0.91) |
| 5 | 0.87 (0.66 to 0.91) | 0.90 (0.74 to 0.93) |
| 6 | 0.89 (0.70 to 0.92) | 0.92 (0.77 to 0.94) |
| 7 | 0.91 (0.73 to 0.93) | 0.93 (0.80 to 0.95) |
| 8 | 0.92 (0.76 to 0.94) | 0.94 (0.82 to 0.95) |
| 9 | 0.93 (0.78 to 0.95) | 0.94 (0.84 to 0.96) |
| 10 | 0.93 (0.80 to 0.95) | 0.95 (0.85 to 0.96) |
*Calculated from the estimated SDs shown in table 1.
GCRS, Global Consultation Rating Scale.