Purpose: Scientific journals rely on peer review to maintain the high quality and standards of papers accepted for publication. The purpose of this study was to explore the strengths and weaknesses of medical education reports by analyzing the ratings and written comments given by external reviewers.
Method: The author conducted a content analysis of reviewers' comments on 151 research manuscripts submitted to the 1997 and 1998 Research in Medical Education conference proceedings. The negative comments on 123 manuscripts that received "questionable, probably exclude" or "definitely exclude" overall ratings from at least one reviewer were evaluated. A similar analysis was performed on reviewers' positive comments for 28 manuscripts recommended unanimously for acceptance.
Results: On average, four peers (4.1, SD = 0.97, range = 2-6) reviewed each manuscript. Of those recommended for exclusion, a mean of 2.3 reviewers recommended exclusion and each reviewer wrote a mean of 8.1 (SD = 5.7) reasons. The top ten reasons for rejection were: inappropriate or incomplete statistics; overinterpretation of results; inappropriate or suboptimal instrumentation; sample too small or biased; text difficult to follow; insufficient problem statement; inaccurate or inconsistent data reported; incomplete, inaccurate, or outdated review of the literature; insufficient data presented; and defective tables or figures. The main strengths noted in accepted manuscripts were the importance or timeliness of the problem studies, excellence of writing, and soundness of study design.
Conclusion: While overstating the results and applying the wrong statistics can be fixed, other problems that the reviewers identified (ignoring the literature, designing poor studies, choosing inappropriate instruments, and writing poor manuscripts) are likely to be fatal flaws warranting rejection.