Objective: Although Australia and the UK have both made efforts to systematize outcome measurement in mental health services, surprisingly little attention has been paid to how best to analyse routine outcome data in order to determine how services are performing.
Methods: Outcome data collected in acute inpatient and ambulatory mental health services across Australia during the 2006-2007 financial year were used. three approaches to measuring effectiveness were explored: effect size (ES); the reliable change index (RCI); and standard error of measurement (SEM).
Results: The most conservative results were produced by the RCI and the least conservative by the medium ES statistic and the SEM. By way of example, only 38.0% of inpatient admission-discharge periods of care showed significant improvement for adults when the RCI was used, whereas 67.4% and 72.9% did so when the medium ES and the SEM statistics were used, respectively.
Conclusions: In any routine outcome measurement exercise, the degree of effectiveness demonstrated by services will depend on the specific statistical indicator used to judge effectiveness. Routine outcome measurement has the potential to answer a range of crucial performance-related questions, but only if the same metric is used. Discussion of the appropriate statistical approach to take to facilitate cross-service, cross-area and even cross-national comparisons warrants attention.