Jump to comment:

• Re:Pre-existing biases and failure to uphold scientific standards
  Lewis O J Killin
  Published on: 23 November 2015
• Dryad data now available
  Surayya Johar
  Published on: 23 November 2015
• Re:Pre-existing biases and failure to uphold scientific standards
  John A. Tucker
  Published on: 23 November 2015
• Re:Pre-existing biases and failure to uphold scientific standards
  Richard Sands
  Published on: 23 November 2015
• Pre-existing biases and failure to uphold scientific standards
  John A. Tucker
  Published on: 23 November 2015

• Published on: 23 November 2015

  Re:Pre-existing biases and failure to uphold scientific standards

  • Lewis O J Killin, PhD Student

  Thank you for your comments. Although we sense that your criticisms primarily relate to the editor's decision to publish our article, we think it would be helpful to address your underlying criticisms of the paper. Specifically, you argue that we cannot assert our findings are indicative of bias in industry-funded papers given the lack of statistical significance.

  We can only elucidate what we have already written about t...

  Show More

  Thank you for your comments. Although we sense that your criticisms primarily relate to the editor's decision to publish our article, we think it would be helpful to address your underlying criticisms of the paper. Specifically, you argue that we cannot assert our findings are indicative of bias in industry-funded papers given the lack of statistical significance.

  We can only elucidate what we have already written about this lack of significance.

  The analysis in question was to deduce whether there was a statistically significant effect of funding source on study heterogeneity. This analysis is affected by the overall heterogeneity there is to explain, and how many studies there are to account for potential differences. As we raise in our discussion, both of these variables are low; there is little heterogeneity to explain and few independent studies to do so. The power of the independent effect as encompassed by a handful of papers is evidenced by its wide confidence intervals, resulting in an overlap and, ultimately, a failure to reject the null hypothesis.

  In spite of a non-significant finding, we nonetheless observe a difference between the effect sizes reported by independent and industry-funded studies, favouring the latter. We conclude that this notable difference resonates with previous literature, and interpret it to this end and in line with the conclusions of past research on industry funding and bias.

  To reconcile this discrepancy, we have argued that power - or the lack of it - is a central issue of this study; its chief weakness. As you mention, there is a relatively long discussion about power, and the amount of space dedicated to it is a testament to its importance in the interpretation of our results.

  Two of the authors are editor-in-chief of scientific journals and, as such, usually give more weight to measures of effect size than p-values derived from tests of statistical significance. There is a cautionary paper on over-valuing such tests in a recent issue of Nature [1] supporting this approach.

  In our discussion, we have made the lack of significance clear, along with issues of power, but we have also highlighted the difference in effect sizes. Subsequently, readers of this article will be able to place their own confidence in our findings accordingly. As Nuzzo concludes, quoting Steven Goodman, "The numbers are where the scientific discussion should start, not end."

  1. Nuzzo, R. Scientific method: Statistical errors. P values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume. Nature 2014;506:150-152.

  Conflict of Interest:

  I am the first author of this paper. This response was edited and approved by all co-authors of the paper.

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top
• Published on: 23 November 2015

  Dryad data now available

  • Surayya Johar, Assistant Editor

  Data for this article is now available in the Dryad data repository (10.5061/dryad.1h18h) and can be viewed here http://datadryad.org/resource/doi:10.5061/dryad.1h18h.

  Conflict of Interest:

  BMJ Open staff member

  Conflict of Interest:
  None declared.

  • Back to top
• Published on: 23 November 2015

  Re:Pre-existing biases and failure to uphold scientific standards

  • John A. Tucker, Freelance Scientific Writer

  Dear Dr. Sands,

  Thank you for your response. I would like to note however, that my concern was not that this article failed "to reach a particular subjective threshold of perceived impact or citation count." It was rather that both the abstract and the paper itself drew conclusions that clearly were not supported by the results of the study. This is quite a different issue. To wit:

  The result: "Trial...

  Show More

  Dear Dr. Sands,

  Thank you for your response. I would like to note however, that my concern was not that this article failed "to reach a particular subjective threshold of perceived impact or citation count." It was rather that both the abstract and the paper itself drew conclusions that clearly were not supported by the results of the study. This is quite a different issue. To wit:

  The result: "Trials published by pharmaceutical companies reported a larger effect of donepezil on standardized cognitive tests than trials published by independent research groups (standardized mean difference =0.46, 95% CI 0.37 to 0.55 vs 0.33, 95% CI 0.18 to 0.48)"

  The 95% confidence intervals for these two outcomes clearly overlap, and thus the null hypothesis that the means are equal cannot be rejected according to standards that are essentially universal within science and medicine. Nonetheless, the abstract continues with the conclusion that "The effect sizer of donepezil on cognition is larger in industry-funded than independent trials". This statement is simply not supported by the data presented in the paper.

  The conclusion is not simply stated once, but repeated throughout the paper with a certainty that completely belies the investigator's inabiltiy to support their hypothesis with evidence. Continuing,

  "This meta-analysis has provided a distinct example of the established phenomenon that industry funding is associated with a higher favourable outcome for the industry's product" - this is a completely unsupported statement per the author's own data.

  "The pattern of effects reported here has also been observed in other meta-analyses" - However, no "effects" were seen in this study.

  "While the results of the current study confirm that donepezil is effective in terms of cognitive outcomes, it provides another instance of how a meta-analysis may amalgamate conflicting evidence and produce an overall inaccurate effect if funding sources are not considered" - No, because the results of the current study did not demonstrate any difference between industry and independently conducted trials.

  I could go on, but overall I count 4 paragraphs discussing the failure of the study to show a statistically signficant effect (one of which assumes the effect is real and blames the lack of statistical significance on sample size) followed by 12 paragraphs of conclusions that are completely unsupported by the data.

  I apologize for the sarcasm in my prior note, but it was intended to make a very important point. I have little doubt but that if BMJ Open received a manuscript from a pharmaceutical company describing an antidepressant trial in which the difference between the treatment and control outcomes on the primary endpoint was not statistically significant, the Journal would correctly not allow the authors to briefly acknowledge the failure of the trial to hit its endpoint, then go into a page long discussion of how promising the drug was for treating patients. We all know that is not acceptable.

  What is puzzling to me is why the staff of BMJ thought it was acceptable in this case.

  I'd also note that I did read the referee reviews, and at least one of the referees made it very clear that he did not understand the statistical issued involved. I think the appropriate thing to do would have been to replace this referee or add a third.

  Given the extreme departure from scientific norms that occurred in this case, I urge you to retract the paper.

  Respectfully,

  John Tucker, PhD.

  Conflict of Interest:

  None declared

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top
• Published on: 23 November 2015

  Re:Pre-existing biases and failure to uphold scientific standards

  • Richard Sands, Managing editor

  Dear Dr Tucker

  Thank you for your letter.

  Some of the papers we publish report results that are not statistically significant. BMJ Open does not reject papers if they fail to reach a particular subjective threshold of perceived impact or citation count. All published papers have been externally peer reviewed.

  As with all BMJ Open papers, the peer review history for this paper is online, so yo...

  Show More

  Dear Dr Tucker

  Thank you for your letter.

  Some of the papers we publish report results that are not statistically significant. BMJ Open does not reject papers if they fail to reach a particular subjective threshold of perceived impact or citation count. All published papers have been externally peer reviewed.

  As with all BMJ Open papers, the peer review history for this paper is online, so you can see that the paper went through two rounds of external peer review prior to publication. The limitations of the study are clearly stated and acknowledged more than once.

  We welcome post-publication review and critique. We notify authors of any responses that are published so they can respond if they wish. We've done so here albeit the comments are more about BMJ Open than the authors' work.

  Yours, Richard Sands

  Conflict of Interest:

  I am the managing editor of BMJ Open

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top
• Published on: 23 November 2015

  Pre-existing biases and failure to uphold scientific standards

  • John A. Tucker, Freelance Scientific Writer

  Dear Madam or Sir:

  My attention was recently drawn to "The effect of funding sources on donepezil randomized controlled trial outcome: a meta analysis", published in BMJ open on April 7, 2014. Having read the abstract 3 times, and perused the paper to ensure that I had not misunderstood, I am writing to ask what failure of the peer review system allowed this paper to be published in its present form.

  ...

  Show More

  Dear Madam or Sir:

  My attention was recently drawn to "The effect of funding sources on donepezil randomized controlled trial outcome: a meta analysis", published in BMJ open on April 7, 2014. Having read the abstract 3 times, and perused the paper to ensure that I had not misunderstood, I am writing to ask what failure of the peer review system allowed this paper to be published in its present form.

  Drs. Killen et al. undetake a traditional meta analysis comparing the efficacy of donepezil against an endpoint of cognitive function (measured using the standard MMSE and ADAS-cog scales), comparing the effect size obtained in industry-sponsored vs. non-industry sponsored trials. The investigators find a 95% confidence interval of 0.46 +/- 0.09 for the effect size in the industry-sponsored trials, and an effect size of 0.33 +/- 0.15 for the non-industry sponsored trials, for a difference of approximately 0.13 +/- 0.17 (p=0.13) Failing to achieve statistical significance in their first attempt at proving their hypothesis, they take a second run at the problem, ignoring the hazards of multiple hypothesis testing and still failing to demonstrate an effect. Limiting the trials to those of up to 12 weeks duration, they find a difference of 0.09 +/- 0.19, for a p value of 0.32.

  Having completely failed to demonstrate any statistically significant difference between the two groups, the authors conclude "The effect size of donepezil on cognition is larger in industry-funded than independent trials and this is not explained by the longer duration of industry-funded trials". Having failed to demonstrate their hypothesis, the journal and its referees nonetheless allow the authors to continue with a multi- paragraph tirade regarding the unreliability of industry sponsored research.

  I am curious as to exactly when BMJ began publishing papers making claims based on studies that failed to demonstrate a statistically significant effect. Can my company publish a paper in BMJ opening extolling at length about the benefits of our new drug which failed to show a statistically significant effect on outcomes in its clinical trial, or does this allowance only apply to those writing articles critical of industry?

  Respectfully,

  John Tucker, PhD

  Conflict of Interest:

  None declared

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top