How many studies are in the file drawer? An estimate from the family/marital psychotherapy literature

https://doi.org/10.1016/0272-7358(89)90013-5Get rights and content

Abstract

There has been much speculation but little data about the number of unpublished studies located in “file drawers.” If the number is high, and if those studies yield systematically different effects, then literature reviews that ignore them might yield biased results. As part of an ongoing meta-analysis of the effectiveness of marital/family therapies, the authors asked 519 randomly sampled members of five relevant organizations if they had file drawer studies. A total of 375 respondents yielded three such unpublished studies that would otherwise have qualified for inclusion in the meta-analysis. Resulting population estimates suggest that there may be almost as many family/marital psychotherapy studies in the file drawer as there are published studies and dissertations. However, because so few file drawer studies were actually obtained, very large confidence intervals surrounded the estimate of the magnitude of effect sizes. Hence the degree of bias resulting from the file drawer problem is still in doubt. Available evidence in this and other studies suggests that the conservative conclusion is to assume that population effect sizes are only 70–90% as large as those computed from published studies.

References (28)

  • M. Frankel

    Sampling theory

  • R. Rosenthal

    The “file drawer problem” and tolerance for null results

    Psychological Bulletin

    (1979)
  • M.L. Smith

    Publication bias and meta-analysis

    Evaluation in Education

    (1980)
  • American Association for Marriage and Family Therapy

    Directory of clinical members, 1984–1985

    (1985)
  • American Family Therapy Association

    Membership directory

    (1986)
  • American Psychological Association

    Directory of the American Psychological Association

    (1985)
  • Association for Advancement of Behavior Therapy

    1984 membership directory

    (1984)
  • E.R. Babbie

    Survey research methods

    (1973)
  • R.L. Bednar et al.

    Systems of family treatment: Substance or semantics?

    Annual Review of Psychology

    (1988)
  • J. Cohen

    Statistical power analysis for the behavioral sciences

    (1977)
  • N.B. Epstein et al.

    Family dysfunction in bipolar disorder: Description and pilot treatment study

  • L.L. Feinauer

    Family therapy pilot study scores on miscellaneous dependent variables

    (1987)
  • A.G. Greenwald

    Consequences of prejudice against the null hypothesis

    Psychological Bulletin

    (1975)
  • A.S. Gurman et al.

    Research on marital and family therapies

  • Cited by (39)

    View all citing articles on Scopus

    This work was supported in part by a grant to the first author from the National Institute of Mental Health (No. 2-R01-MH41097-02). A previous version of this manuscript was presented at the American Evaluation Association, Boston, October 1987.

    1

    The authors would like to thank Jeffrey Berman, Thomas D. Cook, William Follette and Robert Rosenthal for helpful comments on the previous version. We would especially like to thank Martin Frankel for computing confidence intervals for those instances in the present data where the exact binomial probabilities were required.

    View full text