Skip to main content
Log in

Impartial Judgment by the “Gatekeepers” of Science: Fallibility and Accountability in the Peer Review Process

Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

High publication demands and the low acceptance rate of peer review journals place the journal editors and their reviewers in a powerful position. Journal reviewers have a vital role not only in influencing the journal editor's publication decisions, but also in the very nature and direction of scientific research. Because of their influence in peer review outcomes, journal reviewers are aptly described as the “gate keepers of science.” In this article we describe several pitfalls that can impede reviewers' impartial judgement. These include such issues as confirmatory bias, the negative results bias (the file drawer problem), the Matthew effect, the Doctor Fox effect, and gender, race, theoretical orientation, and “political correctness.” We argue that procedures currently used by many professional journals, such as blind or masked review, may not completely alleviate the effects of these pitfalls. Instead, we suggest that increasing reviewers' awareness of the pitfalls, accountability, and vigilance can improve fairness in the peer review process. The ultimate responsibilities belong to the journal editors who are confronted with the difficult task of satisfying journal readers, contributors, reviewers, and owners. We recommend that the journal editors conduct periodic internal and external evaluations of their journals' peer review process and outcomes, with participation of reviewers, contributors, readers and owners.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Albert, T. (1997). Why bother with peer review? Lancet 350: 822.

    Article  Google Scholar 

  • Altman, D.G., Chalmers, I. & Herxheimer. A. (1994). Is there a case for an international medical scientific press council? Journal of the American Medical Association 272: 166–167.

    Article  PubMed  Google Scholar 

  • Angell, M. (1999). The journal and its owner: Resolving the crisis. New England Journal of Medicine 341: 752.

    Article  PubMed  Google Scholar 

  • APA (2001). Summary report of journal operations, 2000. American Psychologist 56: 693–694.

    Google Scholar 

  • Armstrong, S.J. (October 25, 1996). We need to rethink the editorial role of peer reviewers. The Chronicle of Higher Education 43(9): B3.

    Google Scholar 

  • Bailar, J.C. & Patterson, K. (1985). Journal peer review: The need for a research agenda. New England Journal of Medicine 312: 654–657.

    PubMed  Google Scholar 

  • Bartko, J.J. (1966). The intra-class correlation coefficient as a measure of reliability. Psychological Reports 19: 3–11.

    PubMed  Google Scholar 

  • Begley, S. (1992, September 14). Is science censored? Newsweek, p. 63.

  • Belsky, J. (1986). Infant day care: A cause for concern? Zero to Three 10: 22–24.

    Google Scholar 

  • Belsky, J. (1987, April). Science, social policy and day care: A personal odyssey. Paper presented at the biennial meeting of the Society for Research in Child Development, New York.

    Google Scholar 

  • Black, N., Van Rooyen, S. et al. (1998).What makes a good reviewer and a good review for a general medical journal. Journal of the American Medical Association 280: 231–233.

    Article  PubMed  Google Scholar 

  • Bloom, F.E. (1999). The importance of reviewers. Science 283: 789.

    Article  PubMed  Google Scholar 

  • Caelleigh, A.S., Hojat, M. et al. (September 2001). Effects of reviewers' gender on assessments of a gender-related standardized manuscript. Presented at the Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain.

    Google Scholar 

  • Callaham, M.L., Baxt, W.G. et al. (1998). Reliability of editors' subjective quality ratings of peer reviews of manuscripts. Journal of the American medical Association 280: 229–231.

    Article  PubMed  Google Scholar 

  • Callaham, M.L., Wears, R.L. et al. (1998). Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. Journal of the American Medical Association 280: 254–257.

    Article  PubMed  Google Scholar 

  • Campion, E.W., Curfman, G.D. & Drazen, J.M. (2000). Tracking the peer-review process. The New England Journal of Medicine 343: 1485–1486.

    Article  PubMed  Google Scholar 

  • Cho, M.K., Justice, A.C. et al. (1998). Masking author identity in peer review. Journal of the American Medical Association 280: 243–245.

    Article  PubMed  Google Scholar 

  • Cicchetti, D.V. (1980). Reliability of reviews for the American Psychologist: A biostatistical assessment of the data. American Psychologist 35: 300–305.

    Article  Google Scholar 

  • Colman, A.M. (1979). Editorial role in author-referee disagreements. Bulletin of the British Psychological Society 32: 390–391.

    Google Scholar 

  • Crandall, R. (1990). Author, reviewer and editorial values: Politics in the working mother research? Journal of Social Behavior and Personality 5: 489–491.

    Google Scholar 

  • Crandall, R. (1982). Editorial responsibilities in manuscript review. The Behavioral and Brain Sciences 5: 207–208.

    Google Scholar 

  • Crandall, R. (1978). Interrater agreement on manuscripts is not so bad! American Psychologist 33: 623–624.

    Article  Google Scholar 

  • Crane, D. (1967). The gatekeepers of science: Some factors affecting the selection of articles for scientific journals. The American Sociologist 32: 195–201.

    Google Scholar 

  • DeGrazia, A. (1963). The scientific reception system and Dr. Velikovsky. American Behavioral Scientist 7: 38–56.

    Google Scholar 

  • Dickersin, K. (1990). The existence of publication bias and risk factors for its occurrence. Journal of the American Medical Association 263: 1385–1389.

    Article  PubMed  Google Scholar 

  • Dividoff, F., DeAngelis, C.D. et al. (2001). Sponsorship, authorship, and accountability. Journal of American Medical Association 286: 1232–1233.

    Article  Google Scholar 

  • Drucker, P.F. (1994). Political correctness and American academe. Society 32: 58–63.

    Google Scholar 

  • Eagley, A.H. (1995a). The science and politics of comparing women and men. American Psychologist 50: 145–158.

    Article  Google Scholar 

  • Eagley, A.H. (1995b). Reflections on the commentators' views. American Psychologist 50: 169–171.

    Article  Google Scholar 

  • Fisher, M., Friedman, S.B. & Strauss, B. (1994). The effects of blinding on acceptance of research papers by peer review. Journal of the American Medical Association 272: 143–146.

    Article  PubMed  Google Scholar 

  • Fontanarosa, P.B., Glass, R.M. & De Angelis (2000). Thanking authors, peer reviewers, and readers – constancy in a time of change. Journal of the American Medical Association 283: 2016–2017.

    Article  Google Scholar 

  • Gannon, L., Luchetta, T. et al. (1992). Sex bias in psychological research: Progress or complacency? American Psychologist 47: 389–396.

    Article  PubMed  Google Scholar 

  • Garfunkel, J.M., Ulshen, M.H. et al. (1994). Effect of institutional prestige on reviewers' recommendations and editorial decisions. Journal of the American Medical Association 272: 137–138.

    Article  PubMed  Google Scholar 

  • Garrison, G.E. & Kobor, C.P. (2002). Weathering a political storm: A contextual perspective on a psychological research controversy. American Psychologist 57: 165–175.

    PubMed  Google Scholar 

  • Gilbert, J.R., Williams, E.S. & Lundberg, G.D. (1994). Is there gender bias in JAMA's peer review process? Journal of the American Medical Association 272: 139–142.

    Article  PubMed  Google Scholar 

  • Godlee, F., Gale, C. Martyn, C.N. (1998). Effect on the quality of peer review of blinding reviewers and asking themto sign their names. Journal of the AmericanMedical Association 280: 237–240.

    Article  Google Scholar 

  • Goldbeck-Wood, S. (1999). Evidence on peer review – Scientific quality control or smoke screen? British Medical Journal 318: 44–45.

    PubMed  Google Scholar 

  • Goldberg, P. (1968). Are some women prejudiced against women? Trans-Action 5: 28–30.

    Google Scholar 

  • Gottfredson, S.D. (1978). Evaluating psychological research reports: Dimensions, reliability, and correlates of quality judgments. American Psychologist 33: 920–934.

    Article  Google Scholar 

  • Greenwald, A.G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin 82: 1–20.

    Google Scholar 

  • Hall, J.A. (1979). Author review of reviewers. American Psychologist 34: 798.

    Google Scholar 

  • Hare-Mustin, R. & Maracek, J. (1988). The meaning of difference: Gender theory, postmodernism, and psychology. American Psychologist 43: 355–364.

    Article  Google Scholar 

  • Hare-Mustin, R. & Maracek J. (1990). Making a difference: Psychology and the Construction of Gender. New Haven, CT: Yale University Press.

    Google Scholar 

  • Hendrick, C. (1976). Editorial comment. Personality and Social Psychology Bulletin 2: 207–208.

    Google Scholar 

  • Herrnstein, R.J. & Murray, C. (1994). The Bell Curve: Intelligence and Class Structure in American Life. New York: The Free Press.

    Google Scholar 

  • Hoffman, L.W. (1990). Bias and social responsibility in the study of maternal employment. In C.B. Fisher & W.W. Tryon (eds.), Ethics in Applied Developmental Psychology: Emergency Issues in an Emergency Field (Annual Advances in Applied Developmental Psychology, Vol. 4) (pp. 253–271). Norwood, NJ: Ablex.

    Google Scholar 

  • Hojat, M. (1993a). A mother's love: What children will not receive in day-care centers. (Interview). The Family in America 8: 1–7.

    Google Scholar 

  • Hojat, M. (1993b). Abandoning research on consequences of nonmaternal care: A disservice to the science. Journal of Social Behavior and Personality 8: 5–8.

    Google Scholar 

  • Hojat, M. (1993c). The world declaration of the rights of the child: Anticipated challenges. Psychological Reports 72: 1011–1022.

    PubMed  Google Scholar 

  • Hojat, M. (1990). Can affectional ties be purchased? Comments on working mothers and their families. Journal of Social Behavior and Personality 5: 493–502.

    Google Scholar 

  • Hojat, M. (1995). Developmental pathways to violence: A psychodynamic paradigm. Peace Psychology Review 1: 176–195.

    Google Scholar 

  • Hojat, M. An empirical study of possible biases due to reviewers' theoretical perspective, gender, and political correctness of the manuscript in journal peer review process, unpublished manuscript.

  • Hoover, J.D. & Howard, L.A. (1995). The political correctness controversy revisited: Retreat from argumentation and reaffirmation of critical dialogue. American Behavioral Scientist 38: 963–975.

    Google Scholar 

  • Horrobin, D.F. (1982). Peer review: A philosophically faulty concept which is proving disastrous for science. The Behavioral and Brain Sciences 5: 217–218.

    Google Scholar 

  • Horton, R. (1998). The journal ombudsperson: A step toward scientific press oversight. Journal of the American Medical Association 280: 298–299.

    Article  PubMed  Google Scholar 

  • Horton, R. (1996). The Lancet's ombudsman. Lancet 348: 6.

    Article  Google Scholar 

  • Howard, L. & Wilkinson, G. (1998). Peer review and editorial decision-making. British Journal of Psychiatry 173: 110–113.

    PubMed  Google Scholar 

  • Hunt, E. (1971). Psychological publications. American Psychologist 26: 311.

    Google Scholar 

  • Jefferson, T. & Godlee, F. (eds.) (1999). Peer Review in Health Care. London, UK: British Journal Publishing Group.

    Google Scholar 

  • Justice, A.C., Cho, M.K. et al. (1998). Does masking author identity improve peer review quality? A randomized controlled trail. Journal of the American Medical Association 280: 240–242.

    Article  PubMed  Google Scholar 

  • Karen, R. (1994). Becoming Attached. New York: Warner Books.

    Google Scholar 

  • Kassirer, J.P. & Campion, E.W. (1994). Peer review: Crude and understudied, but indispensable. Journal of the American Medical Association 272: 96–97.

    Article  PubMed  Google Scholar 

  • Kennedy, D. (2002). Publish or not publish. Science 295: 1793.

    Article  PubMed  Google Scholar 

  • King, D.W., McDonald, D.D. & Rodereer, M.K. (1981). Scientific Journals in the United States: Their Production, Use, and Economics. Stroudsburg, PA: Hutchinson & Ross.

    Google Scholar 

  • Laband, D.N. & Piette,M.J. (1994). A citation analysis of the impact of blinded peer review. Journal of the American 272: 147–149.

    Google Scholar 

  • Lilienfeld, S.O. (2002).When worlds collide: Social science, politics, and the Rind et al. (1998) child sexual abuse meta-analysis. American Psychologist 57: 176–188.

    Google Scholar 

  • Lock, S. (1985). A Difficult Balance: Editorial Peer Review in Medicine. Philadelphia: ISI Press.

    Google Scholar 

  • Lock, S. (ed.) (1991). The Future of Medical Journals. Plymouth, UK: British Medical Journal.

    Google Scholar 

  • Loury, G.C. (1994). Self-censorship in public discourse: A theory of political correctness and related phenomena. Rationality and Society 6: 428–461.

    Google Scholar 

  • Mahoney, M. (1977). Publication prejudices: An experimental study of confirmatory bias in peer review system. Cognitive Therapy and Research 1: 161–175.

    Article  Google Scholar 

  • Mahoney, M.J. (1987). Scientific publication and knowledge politics. Journal of Social Behavior and Personality 2(Part 1): 165–176.

    Google Scholar 

  • McNutt, R.A., Evans, A.T. et al. (1990). The effects of blinding on the quality of peer review. Journal of the American Medical Association 263: 1371–1376.

    Article  PubMed  Google Scholar 

  • Merton, R.K. (1968). The Matthew effect in science. Science 159: 56–63.

    PubMed  Google Scholar 

  • Misakian, A.L. & Bero, L.A. (1998). Publication bias and research on passive smoking: Comparison of published and unpublished studies. Journal of the American Medical Association 280: 250–253.

    Article  PubMed  Google Scholar 

  • Moore, M. (1978). Discrimination or favoritism? Sex bias in book reviews. American Psychologist 33: 936–938.

    Article  Google Scholar 

  • Naftulin, D.H., Ware, Jr., J.E. & Donnelly, F.A. (1973). The doctor Fox lecture: A paradigm of educational seduction. Journal of Medical Education 48: 630–635.

    PubMed  Google Scholar 

  • Nylenna, M., Riis, P. & Karlsson, Y. (1994). Multiple blinded reviews of the same two manuscript. Journal of the American Medical Association 272: 149–151.

    Article  PubMed  Google Scholar 

  • Olson, C.M. (1990). Peer review of biomedical literature. American Journal of Emergency Medicine 8: 356–358.

    Article  PubMed  Google Scholar 

  • Over, R. (1982). What is the source of bias in peer review? The Behavioral and Brain Sciences 5: 229–230.

    Google Scholar 

  • Oxman A.D., Guyatt, G.H. & Singer, J. (1991). Agreement among reviewers of review articles. Journal of Clinical Epidemiology 44: 91–98.

    Article  PubMed  Google Scholar 

  • Peters, D.P. & Ceci, S.J. (1982). Peer review practices of psychological journals: The fate of published articles, submitted again. The Behavioral and Brain Sciences 5: 187–195.

    Google Scholar 

  • Pierie, J.P., Walvoort, H.C. & Overbeke, A.J.P. (1996). Readers' evaluation of effect of peer review and editing on quality of articles in the Nederlands Tijdschrift voor Geneeskunde. Lancet 348: 1480–1483.

    Article  PubMed  Google Scholar 

  • Popenoe, D. (1993a). American family decline, 1960-1990: A review and appraisal. Journal of Marriage and the Family 55: 527–532.

    Google Scholar 

  • Popenoe, D. (1993b). The national family wars. Journal of Marriage and the Family 55: 533–553.

    Google Scholar 

  • Rennie, D. (1998). Peer review in Prague. Journal of the American Medical Association 280: 214–215.

    Article  PubMed  Google Scholar 

  • Riger, S. (1992). Epistemological debates, feminist voices: Science, social values, and the study of women. American Psychologist 47: 730–740.

    Article  Google Scholar 

  • Riis, P. (1992). New paradigm in journalogy. Journal of Internal Medicine 232: 207–213.

    PubMed  Google Scholar 

  • Rind, B., Tromovich, P. & Bauserman, R. (1998). A meta-analytic examination of assumed properties of child sexual abuse using college samples. Psychological Bulletin 124: 22–53.

    Article  PubMed  Google Scholar 

  • Rosenberg, R.N. & Anderson, E.R., Jr. (1999). Editorial governance of the Journal of the American Medical Association: A report. Journal of the American Medical Association 281: 2239–2242.

    Article  PubMed  Google Scholar 

  • Rosenthal, R. (1979). The “file drawer problem” and tolerance of null results. Psychological Bulletin 86: 541–638.

    Google Scholar 

  • Rubin, P.H. (1994). The assault on the first amendment: Public choice and political correctness. Cato Journal 14: 23–36.

    Google Scholar 

  • Scarr, S. & Weber, B.L.R. (1978). The reliability of reviewers for the American Psychologist. American Psychologist 33: 935.

    Google Scholar 

  • Scott, W.A. (1974). Interreferee agreement on some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology. American Psychologist 31: 799–804.

    Google Scholar 

  • Sharp, D.W. (1990). What can and should be done to reduce publication bias? Journal of the American Medical Association 263: 1390–1391.

    Article  PubMed  Google Scholar 

  • Shea, J.A., Caelleigh, A.S. et al. (2001). Review process and publication decision. Academic Medicine 76: 911–916.

    Google Scholar 

  • Silverstein, L.B. (1991). Transforming the debate about child care and maternal employment. American Psychologist 46: 1025–1032.

    Article  PubMed  Google Scholar 

  • Smith, R. (1999). Opening of BMJ peer review. British Medical Journal 318: 4–5.

    PubMed  Google Scholar 

  • Stark-Adamec, C. (1993). Social science and scientific responsibility. Canadian Journal of Physics 71: 192–196.

    Google Scholar 

  • Stephenson, J. (1997). Medical journals turn gaze inward to examine process of peer review. Journal of the American Medical Association 278: 1389–1391.

    Article  PubMed  Google Scholar 

  • Sterling, T.D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance or vise versa. Journal American Statistical Association 54: 30–34.

    Google Scholar 

  • Susser, M. & Northridge, M.E. (1996). Editor's note: Reviewing for the journal. American Journal of Public Health 86: 161.

    Google Scholar 

  • Sweitzer, B.J. & Cullen, D.J. (1994). How well does a journal's review process function? A survey of authors' opinion. Journal of the American Medical Association 272: 152–153.

    Article  PubMed  Google Scholar 

  • Van Rooyen S., Godlee, H. et al. (1998). Effect of open peer review on quality of quality of reviews and reviewers recommendation: A randomised trail. British Medical Journal 317: 23–27.

    Google Scholar 

  • Van Rooyen, S., Godlee, H. et al. (1998). Effect of blinding and unmaksing on the quality of peer review: A clinical trail. Journal of American Medical Association 280: 234–237.

    Article  Google Scholar 

  • Ware, J.E. & Williams, R.G. (1975). The Dr. Fox effect: A study of lecturer effectiveness and ratings of instruction. Journal of Medical Education 50: 149–156.

    PubMed  Google Scholar 

  • Watkins, M.W. (1979). Chance and interrater agreement on manuscript. American Psychologist 34: 796–798.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Mohammadreza Hojat.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hojat, M., Gonnella, J.S. & Caelleigh, A.S. Impartial Judgment by the “Gatekeepers” of Science: Fallibility and Accountability in the Peer Review Process. Adv Health Sci Educ Theory Pract 8, 75–96 (2003). https://doi.org/10.1023/A:1022670432373

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1022670432373

Navigation