Frequency of discrepancies in retracted clinical trial reports versus unretracted reports: blinded case-control study

Principal findings and implications

This study indicates that the presence of discrepancies in a study report should not be assumed to be meaningless. Discrepancies are significantly more common in retracted rather than unretracted articles.

Peer reviewers and other readers may benefit from this knowledge because it is notoriously difficult for them to evaluate the reliability of a trial’s findings. It is already known27 28 29 that the presence of certain features of study design such as blinding, formal enrolment, and automated documentation of results can substantially affect reported effect size. Our study goes beyond this to suggest that identification of discrepancies, even by scientists without particular scientific specialism in the field, might provide an early alarm of unreliability. When doubt exists, it may be practical to repeat some types of scientific study. This is usually not practical for clinical trials, on grounds of time and expense, so there is additional value in readers being able to gauge the reliability of existing reports.

Although the presence of discrepancies seems sensitive to serious problems within trial reports, it is not specific, in that there are many trial reports in good standing with discrepancies. We do not propose that any particular discrepancy threshold should be used as an absolute level for identifying unreliable trial reports. However, it might help to identify trial reports, in which additional documentation from authors might be important to provide reassurance that a study has been reported reliably.

Journals could help in additional ways. Providing a post publication forum for readers to share knowledge of discrepancies is important, because, as our study shows, one reader may spot only a subset of the discrepancies noticed by multiple readers. We believe this finding highlights the difficulty of the task and the likely benefits of crowd sourcing when examining papers after publication. In our study, six retracted trials and seven unretracted trials had a letter to the editor published or a critical editorial raising concerns. Only one of these letters in each group mentioned any discrepancies.

A journal could plan an automatic escalation protocol that would minimise consumption of editorial time. Once it receives a list of discrepancies, it could publish them immediately and request an online supplement of individual patient data from the paper’s authors, if such a supplement was not already provided in the original publication. The journal could publish the time in days and hours from request to receipt. In honestly conducted trials with innocent errors (for example, honest, simple transcription errors), these would be identified as such and quickly corrected. Readers might draw their own conclusions if the dataset is delayed or unavailable.7 30

We propose this approach of requesting the raw data for two reasons. Firstly, if the authors were asked to rerun the analyses or present an explanation instead, this could take time to conduct and even more time to achieve agreement between authors. By contrast, the raw data can simply be released by the corresponding author, as there should be no debate. Secondly, such a policy would encourage researchers with nothing to hide to provide the full data as an online supplement in the original publication without waiting for discrepancies to be identified.

Without such protocols and related amendments, journal reviewers and editors must individually find and evaluate the significance of discrepancies. An alternative is that the many eyes of readers could be harnessed in crowd sourced analytical capacity, who would know that their observations contribute to science. Journals could respond at an administrative level without consuming scarce editorial time. Authors would also know that publication would provide genuine scrutiny, not routinely provided or even intended by prepublication peer review.31 An alternative mechanism for readers to communicate discrepancies to other readers would be an annotation system such as PubMed Commons or Pubpeer. This circumvents the system of letters to the editor, which is becoming unfit for this purpose because of word count limits and short six week limitation periods in some journals.10

Who would do the work of analysing the raw data? Meta-analysts are likely to have time and motivation for this, but so would any reader who wanted to find out the correct answer efficiently. The currently practiced approach, which is to write a letter to the editor, is ineffective. For example, even asking about the registered primary endpoint of a trial,32 in which the data seemed to be inexplicably missing from the publication,33 may yield an unrevealing reply.34 Worse, if the replacement endpoint is different between abstract (and shareholder prospectus35) versus individual patient data,36 the mathematical impossibility can be stonewalled. Worst of all, statistical experts in the field37 can fail to notice this and instead highlight the queries as being “responsibly rebutted.”38

Study limitations

We recognise that even our list of discrepancies (errors) may itself contain errors. Moreover, we have not attempted to establish the mechanism for the discrepancies. We have no way of knowing where each lies on the spectrum from innocent administrative error to intentional fabrication. The strength of our non-judgmental approach is that the presence of discrepancies, rather than any inferred mechanism, is the signal that a trial may be unsound. Author provision of raw data would allow readers to judge the importance of the identified discrepancies and assist appropriate resolution.

We chose our controls to be the preceding clinical trial in the same journal. Our reasoning was that controlling for journal editorial processes, impact factor,14 readership, and the journal’s postpublication policy was the priority. Instead of using a fixed protocol to identify the control report, it might have been preferable to use individual judgment to select an unretracted report matched for subject matter. However, we considered that attempting to do this would open the study to bias in such selection. Although our study is not able to confidently state whether the observed pattern is changing over time, each control report was very close in time to its counterpart retracted report.

The overall number of discrepancies (of any type) was significantly different between retracted and unretracted trial reports. For every type of discrepancy, the individual count for that type was numerically higher in retracted than unretracted trial reports; for three types (arithmetical errors, factual discrepancies, and missed P values), this difference between retracted and unretracted reports was statistically significant. This reanalysis that separated the types of discrepancy did not have power to adequately test each type individually. Whether some types of discrepancy are particularly strong markers of trial report unreliability independently therefore remains uncertain. It is also unknown whether some types of discrepancy imply a mechanism (for example, fabrication) and should be considered especially concerning. Alternatively, some readers might be particularly concerned by discrepancies that have immediate therapeutic implications for patients (for example, miscalculations of therapeutic effect size or significance).

In addition, the sample size was constrained by resources because of the time taken to identify, verify, and collate the 479 discrepancies in 100 trial reports, and was not based on a formal power calculation. Future researchers could address other such trial reports or even reassess the trial reports we analysed.

Misclassified clinical trials—subgroup analysis of pairs of reports meeting a stricter definition of “clinical trial”

During the study, it became apparent that some of the publications identified during the PubMed search would not generally be considered clinical trials of a therapeutic intervention. We performed an additional reanalysis that adopted an aggressive strategy of removing all pairs of clinical trials listed in PubMed where one of the pair was not actually a clinical trial. This reselection process left 68 trial reports in 34 pairs, where both would be generally recognised as clinical trials. Web appendix 4 shows all the figures redrawn for this subgroup. The pattern observed in the 100 trial reports classified as clinical trials in PubMed remained evident in this subgroup of trials with a therapeutic intervention.

Of 335 discrepancies in the 68 trial reports, 254 were in the 34 retracted reports and 81 were in the 34 unretracted reports. The overall numbers of discrepancies were 3.1-fold higher for the retracted reports than the unretracted reports. Individual discrepancy counts remained higher in retracted reports than unretracted reports (median 5 (interquartile range 3-8.75) v 0.5 (0-4.75); P<0.001).

Thirty two (94%) of 34 retracted reports and 17 (50%) of 34 unretracted reports contained discrepancies. Papers with a discrepancy were significantly more likely to be retracted than those without a discrepancy (odds ratio 16 (95% confidence interval 3.3 to 78), P<0.001). Using zero inflated negative binomial models, results of the univariable and multivariable analyses in this subgroup (web appendix 5) showed a trend similar to the larger dataset, but with wider confidence intervals, owing to a reduced sample size. A threshold of three or more discrepancies showed 76% sensitivity and 68% specificity for identifying retracted reports in this subgroup.