Self-reported versus clinician-rated symptoms of depression as outcome measures in psychotherapy research on depression: A meta-analysis

Abstract

It is not well-known whether self-report measures and clinician-rated instruments for depression result in comparable outcomes in research on psychotherapy. We conducted a meta-analysis in which randomized controlled trials were included examining the effects of psychotherapy for adult depression. Only studies were included in which both a self-report and a clinician-rated instrument were used. We calculated the effect size (Hedges' g) based on the self-report measures, the effect size based on the clinician-rated instruments, and the difference between these two effect sizes (Δg). A total of 48 studies including a total of 2462 participants was included in the meta-analysis. The differential effect size was Δg = 0.20 (95% CI: 0.10–0.30), indicating that clinician-rated instruments resulted in a significantly higher effect size than self-report instruments from the same studies. When we limited the effect size analysis to those studies comparing the HRSD with the BDI, the differential effect was somewhat smaller, but still statistically significant (Δg = 0.15; 95% CI: 0.03–0.27). This meta-analysis has made it clear that clinician-rated and self-report measures of improvement following psychotherapy for depression are not equivalent. Different symptoms may be more suitable for self-report or ratings by clinicians and in clinical trials it is probably best to include both.

Introduction

Both self-report and clinician-rated symptom scales of depression are used widely as outcome measures in psychotherapy research. While the correlation between self-report and clinician-rated instruments usually is strong and statistically significant (Domken, Scott, & Kelly, 1994), the agreement in individual cases often is not (Rush et al., 2006). These discrepancies could be due to differences in item content for the clinician and self-report scales, different weightings of symptom items, clinician or patient biases in describing symptomatology, limited insight of the patient (and/or clinician), cognitive deficits, clinical factors such as depression subtypes, severity, comorbidity, and personality traits, and patient demographic characteristics, such as education, ethnicity, and socioeconomic factors (Corruble et al., 1999, Rush et al., 2006).

Several earlier meta-analytic studies of treatments for depression have examined whether or not self-reported and clinician-rated scales result in comparable outcomes (Edwards et al., 1984, Greenberg et al., 1992, Lambert et al., 1986). Most of these studies found that effect sizes based on self-reported measures were smaller than effect sizes based on clinician-rated scales, and this was true for psychological as well as pharmacological treatments. These studies were, however, based on small numbers of studies, and did not use modern meta-analytic techniques to test whether the effect sizes differed significantly from each other. They also did not examine possible moderators of differential effect sizes, levels of heterogeneity, and possible publication bias. More recent meta-analyses have also examined the effects of studies based on self-reported and clinician-rated scales (Pinquart et al., 2006, Pinquart et al., 2007). These studies confirmed that clinician-rated instruments resulted in larger effect sizes than self-report scales, both in pharmacotherapy and psychotherapy. These studies did not, however, test whether the differences between the two types of outcome measures were significant, and did not examine possible moderators either. In the current paper, we aim to examine this difference between the two types of outcome measures in more depth by comparing the effects of clinician-rated measures with those of self-rated measures within the same studies.

Whether or not self-report and clinician-rated scales result in comparable outcomes is an important question, in particular in relation to publication and policy-making standards. Usually clinician-rated scales are considered to be the gold standard in depression outcome research. If, however, self-report scales are just as good as clinician-rated scales, this would greatly benefit both clinical practice and clinical trial research (Rush et al., 2006). Furthermore, if self-reported outcomes result in lower symptom improvement levels than clinician-rated outcomes as is suggested by earlier research, it may be better to use self-reported outcomes in meta-analytic research, as these are more conservative and have less chance of resulting in an overestimation of outcomes. On the other hand, self-report instruments may be less sensitive to change.

The Hamilton Rating Scale for Depression (HRSD; Hamilton, 1960) is by far the most used clinician-rated instrument in depression treatment studies (Bagby et al., 2004, Williams, 2001). The original HRSD has 21 items, although usually only the first 17 are used to assess the severity of depression. There are, however, also versions with 24 items (Williams, 2001) and 26 items (Butler & Waelde, 2009) available. Although the interrater reliability is good in most studies (Bagby et al., 2004), internal consistency is found to be insufficient in some studies. Despite these limitations, the HRSD remains the gold standard among the clinician-rated measures for depression. Other important clinician-rated instruments that have been developed include the Montgomery–Asberg Depression Scale (MADRS; Montgomery & Asberg, 1979), which was designed to be sensitive to treatment effects and does not rely on physical symptoms as heavily as the HRSD, and in older studies the Raskin Scale (Raskin, 1988).

The best known and most used self-report measure of depression is the Beck Depression Inventory (BDI; Beck, Ward, Mendelson, Mock, & Erbaugh, 1961), and its revised version, the BDI-II (Beck et al., 1996). The BDI and the BDI-II consist of 21 items with 4 answer options each. The reliability and validity of the BDI and BDI-II are generally considered to be good (Burt & Ishak, 2002, Richter et al., 1998), although it has been suggested that it is too exclusively based on a cognitive model (Hagen, 2007). Other self-report measures that are much used include the Center for Epidemiological Studies Depression Scale (CES-D; Radloff, 1977), which is often used in epidemiological research, the Zung Self-Rating Depression Scale (Zung, 1965), the Geriatric Depression Scale (GDS, Yesavage et al., 1983), which was specifically designed to measure depressive symptoms in older adults, and the Edinburgh Postpartum Depression Scale (EPDS; Cox, Holden, & Sagovsky, 1987), used to measure symptoms of postpartum depression.

We decided to conduct a new meta-analytic study in which we compared the two types of outcomes in studies which included both self-report and clinician-rated scales. This allowed us to compare the two directly and test whether they result in comparable outcomes. We wanted to compare self-report and clinician-rated instruments in a homogeneous set of studies. We chose to examine this in the acute treatment of depression, because this is a more or less homogeneous area of research, because it comprises a considerable number of studies, allowing us to examine subsets of studies and moderators, and because it is a clinically relevant area of research.