Intended for healthcare professionals

Papers

Pharmaceutical industry sponsorship and research outcome and quality: systematic review

BMJ 2003; 326 doi: https://doi.org/10.1136/bmj.326.7400.1167 (Published 29 May 2003) Cite this as: BMJ 2003;326:1167
  1. Joel Lexchin, associate professor (joel.lexchin{at}utoronto.ca)1,
  2. Lisa A Bero, professor2,
  3. Benjamin Djulbegovic, associate professor3,
  4. Otavio Clark, chief of clinical oncology section4
  1. 1 School of Health Policy and Management, York University, Toronto, ON, Canada M3J 1P3
  2. 2 Department of Clinical Pharmacy and Institute for Health Policy Studies, University of California at San Francisco, San Francisco, CA 94118, USA
  3. 3 Interdisciplinary Oncology Program, H Lee Moffitt Cancer Center and Research Institute, University of South Florida, Tampa, FL 33612, USA
  4. 4 Instituto do Radium de Campinas, 13075–460 Campinas-SP, Brazil
  1. Correspondence to: J Lexchin
  • Accepted 15 April 2003

Abstract

Objective To investigate whether funding of drug studies by the pharmaceutical industry is associated with outcomes that are favourable to the funder and whether the methods of trials funded by pharmaceutical companies differ from the methods in trials with other sources of support.

Methods Medline (January 1966 to December 2002) and Embase (January 1980 to December 2002) searches were supplemented with material identified in the references and in the authors' personal files. Data were independently abstracted by three of the authors and disagreements were resolved by consensus.

Results 30 studies were included. Research funded by drug companies was less likely to be published than research funded by other sources. Studies sponsored by pharmaceutical companies were more likely to have outcomes favouring the sponsor than were studies with other sponsors (odds ratio 4.05; 95% confidence interval 2.98 to 5.51; 18 comparisons). None of the 13 studies that analysed methods reported that studies funded by industry was of poorer quality.

Conclusion Systematic bias favours products which are made by the company funding the research. Explanations include the selection of an inappropriate comparator to the product being investigated and publication bias.

Introduction

Clinical research sponsored by the pharmaceutical industry affects how doctors practise medicine.1 An increasing number of clinical trials at all stages in a product's life cycle are funded by the pharmaceutical industry,2 3 probably reflecting the fact that the pharmaceutical industry now spends more on medical research than do the National Institutes of Health in the United States.4 Most pharmacoeconomic studies are either done in-house by the drug companies or externally by consultants who are paid for by the company.5 6

Results that are unfavourable to the sponsor—that is, trials that find a drug is less clinically effective or cost effective or less safe than other drugs used to treat the same condition—can pose considerable financial risks to companies. Pressure to show that the drug causes a favourable outcome may result in biases in design, outcome, and reporting of industry sponsored research.7

A recent systematic review of the impact of financial conflicts on biomedical research found that studies financed by industry, although as rigorous as other studies, always found outcomes favourable to the sponsoring company.8 However, this review looked for papers published only in English, excluded reports in letters and abstracts, and looked at studies funded by other industries. We reviewed the relation between the source of funding of the research and the reported outcomes and investigated whether quality of the methods in studies funded by pharmaceutical companies differs from that in other studies.

Methods

Study selection

We included only studies that specifically stated that they analysed research sponsored by a pharmaceutical company, compared methodological quality or outcomes with studies with other sources of funding, and reported the results in quantitative terms. Outcomes of interest were conclusions about differences in drug effectiveness, adverse effects, cost outcomes, or publication status between industry funded trials and other trials. Work published in any language was eligible for inclusion.

Some studies analysed both pharmacological and non-pharmacological trials and combined research funded by drug companies and other industries into one group. In these cases, if most were non-pharmaceutical trials and were funded by other industries they were excluded.

Search strategy

We searched Medline from January 1966 to December 2002 using a combination of terms as both MESH subject headings (exploded) and key words (“clinical trials,” “conflict of interest,” “drug industry,” “financial support,” “publication bias” (subject heading only), “research design,” and “research support.”) We searched Embase from January 1980 to December 2002 using a combination of terms as subject headings (exploded) and key words (“clinical trials” (subject heading only), “drug industry,” “ethics,” “financial management,” “methodology,” and “ethics.” To find more studies, we scanned the reference lists from each of the articles and searched the Cochrane methodology register. We placed messages on two email drug discussion groups, contacted content experts, and searched our personal libraries. In cases where the reported results were incomplete, we contacted the lead author and asked for further details. A single author (JL) did the initial selection of studies and sent copies of each of these studies to the other three authors for validation of the inclusion criteria.

Data collection

From each study, we extracted the study design, type of research assessed in the study, design of research assessed in the study, search strategy used to locate research, time period covered, drug or drug class, disease, number of industry and non-industry funded articles analysed in each study, how industry funding was defined, criteria used to assess methodological quality of the research, results with respect to methodological quality or outcome of the research, and primary purpose of study.

We provide a critical description of each included study, but do not assess methodological quality (see table 1). Since our included studies had a variety of designs—that is, cohort collections of trials, meta-analyses, and economic studies—and since we included letters and abstracts with limited descriptions of methods, we had no valid and reliable quality assessment instrument available for assessing their methodological quality. We did not use a component approach to assess their quality since this approach applies to randomised controlled trials.9 10

Table 1

General characteristics of studies

View this table:

Three of us (LB, OC, JL), who were not blinded to study authors or results, independently abstracted information. We resolved disagreements by consensus.

On the basis of the rationale that funding does affect the direction of effect, we did a meta-analysis on the studies that reported the effects of funding on the outcome of either pharmacoeconomic analyses or clinical trials in cases where odds ratios could be computed. The homogeneity test showed that the effect size did not differ between the studies (P=0.17). Using a Mantel-Haenszel test, we constructed a pooled odds ratio.11 We used the program StatsDirect and considered P < 0.05 significant.

Results

Search results

The combined searches and other data sources found 3351 potential titles. We scanned titles and abstracts (where available) for mention of the pharmaceutical industry in either the title or the abstract or any suggestion that the study would deal with industry funding. We read 103 articles in full (eight in languages other than English); we retained 30 articles for analysis. Reasons for exclusion are detailed in the QUOROM statement (fig 1).

The studies by Friedberg et al23 and by Knox et al29 analysed the same set of 44 trials but looked at different aspects of the trials: conclusions about the usefulness of products in one case,23 and how the trials had been reported in the other.29 Only one article was duplicated in the two studies reported by the group including Chard, Tallon, and Dieppe13 18 (J Chard, personal communication, 2002). Seven of the nine articles in Kemmeren's27 meta-analysis of third generation oral contraceptives were also included in Vandenbroucke's meta-analysis.39 We found no other cases of double counting, but as some of the papers did not provide a full list of references we could not exclude the possibility of further overlap.

Characteristics of included studies

Table 1 gives the characteristics of the 30 studies included in this analysis.1241 Six were reviews of pharmacoeconomic reports,12 23 26 29 34 36 two reviewed meta-analyses and systematic reviews,18 25 and the remaining 22 analysed groups of clinical trials.1317 1922 24 27 28 3033 35 3741 Eleven papers mentioned that some trials were funded by industry but offered 22 further definition of industry funding.17 18 22 2426 28 34 36 39 40 In the other 15 papers the definition varied from a statement acknowledging industry funding in the article12 32 to a more comprehensive definition.35

Relationship between source of funding and outcome

A total of 26 of the 30 studies reported results on the association of the outcome of the research and the source of funding: six examined the effects on publication,17 21 24 31 33 37 five looked at the outcome of pharmacoeconomic studies,12 23 26 34 36 and 16 analysed the outcome of clinical trials and meta-analyses of clinical trials (table 2).1316 1820 22 24 27 30 32 3841

Table 2

Relation between source of funding and outcome of research

View this table:

Funding source and publication status

Research funded by drug companies was less likely to be published or presented than research funded by other sources (table 2). 17 21 Three studies looked at time to publication,24 31 37 and two of these found that company sponsored research took longer to be published than research with other sources of funding.24 31 Research funded by drug companies was also more likely to be published in the proceedings of symposiums than non-industry sponsored research.33

Funding source and economic outcomes

Pharmacoeconomic studies sponsored by the drug industry were more likely to report results favouring the sponsor's product than studies with other sources of funding in all five articles that examined this question.12 23 26 34 36 In three cases, however, the bias in favour of industry funded research depended on the particular question being posed23 26 or on where the pharmacoeconomic analyses were published.36

Funding source and outcomes of clinical trials and meta-analyses

Sixteen studies investigated the relationship between funding source and the outcomes of clinical trials and meta-analyses. Of these, 13 found that clinical trials and meta-analyses sponsored by drug companies favoured the product produced by the funder. Statistical significance for this finding was reported in eight of the 13 studies,13 14 16 19 20 38 40 41 and in another two there was a trend towards statistical significance.18 22 These studies covered a wide range of diseases, such as osteoarthritis of the knee,13 18 multiple myeloma,20 various psychiatric problems,32 40 Alzheimer's disease,30 and venous thromboembolism,39 and a wide range of drugs, such as tacrine,30 clozapine,40 third generation oral contraceptives,39 erythropoietin,19 antidepressants,22 and topical glucocorticosteroids.38 One study that found no difference looked at the outcome of trials of treatment for HIV and associated complications and in this case the trials were monitored by the National Institutes of Health.24 In one meta-analysis of third generation oral contraceptives,27 the risk of venous thromboembolism for non-industry funded research was higher than that for industry sponsored trials, although the increased risk for thromboembolic disease was significant in both cases. Another study found no difference in outcomes in research published in five leading medical journals.15

Figure 2 shows the individual odds ratios and summary odds ratio for 18 different comparisons (15 studies) of the outcomes of industry funded and non-industry funded studies—seven from pharmacoeconomic analyses and 11 from clinical trials or meta-analyses of clinical trials. The summary odds ratio was 4.05 (95% confidence interval 2.98 to 5.51).

Fig 2
Fig 2

Source of funding and outcome in pharmacoeconomic analyses, clinical trials, and meta-analyses of clinical trials of drug treatments; for references see bmj.com (*Favourable qualitative results; †Overstatement of quantitative results; ‡Reporting possibility of cost effectiveness or cost savings of prophylaxis in entire high risk infant population either in point estimates or sensitivity analysis; §Reporting cost effectiveness or cost savings in either entire high risk populations or specific infant subgroups compared across studies; ¶Analyses reported in general medical journals; **Analyses reported in Pharmacoeconomics)

Relationship between source of funding and methodologic quality

A total of 13 studies examined the relationship between the source of funding and the methodological quality of the research (table 3).1416 19 20 25 28 29 3135 None of the 13 reported that industry funded studies had poorer methodological quality. Of the nine that provided statistical analyses, four found that drug company sponsored research had better quality scores.14 28 31 33

Table 3

Relation between source of funding and methodological quality of research

View this table:

Nine of the studies on clinical trials used well established methods of assessing quality.14 15 19 20 24 28 29 31 35 The single study that reported on the methods of pharmacoeconomic analyses used commonly accepted criteria for assessing cost effectiveness.34

One study evaluated the appropriateness of the comparators in clinical trials and found that a greater proportion of industry sponsored studies compared innovative treatment to either placebo or no therapy than did studies sponsored by public resources (60% v 21%; P < 0.001).20

Discussion

Research sponsored by the drug industry was more likely to produce results favouring the product made by the company sponsoring the research than studies funded by other sources. The results apply across a wide range of disease states, drugs, and drug classes, over at least two decades and regardless of the type of research being assessed—pharmacoeconomic studies, clinical trials, or meta-analyses of clinical trials. The totality of the evidence reported in our meta-analysis of a subset of homogeneous studies suggests that there is some kind of systematic bias to the outcome of published research funded by the pharmaceutical industry.

Our results confirm and extend those reported by Bekelman et al.8 They identified only five studies that compared outcomes in research funded by pharmaceutical companies and other sources,14 16 20 23 41 and our study adds another 16 studies12 13 15 18 19 22 24 26 27 30 32 34 36 3840 Our results are also supported by Rochon and coworkers43 (we excluded this paper because all of the trials were sponsored by drug companies and were, therefore, not comparible with trials lacking company funding.) They found that trials supported by manufacturers of non-steroidal anti-inflammatory agents almost always reported that the sponsor's drug was as or more effective and less toxic than the comparison drug.

Explanations

At least four possible explanations exist for favourable results seen in industry sponsored research. Firstly, pharmaceutical companies may selectively fund trials on drugs that they consider to be superior to the competition. Data collected so far, however, indicate that researchers cannot predict results of trials in advance.44

Secondly, positive results could be the consequence of poor quality research conducted by industry. For example, low quality trials exaggerate the benefits of treatment by an average of 34%.45 46 We found that the research methods of trials sponsored by drug companies is at least as good as that of non-industry funded research and in many cases better. This does not guarantee the absence of bias in studies sponsored by the industry since outcome could be influenced by factors left out of quality scores, such as the question asked or the conduct or reporting of the study.7 47

Thirdly, selecting an appropriate comparator is a key issue in planning a clinical trial.7 20 44 In the study by Rochon et al, in most cases in which the doses of the study and comparator drugs were not equivalent, the drug given at the higher dose was that of the supporting manufacturer.43 As the authors saw, higher doses may bias the results in favour of effectiveness of the manufacturer's product. Safer also reports that in trials of psychiatric drugs the comparator drug is often given in doses outside the usual range or there is a rapid and substantial dose increase in the drug not manufactured by the sponsoring company.48 In another instance, research funded by the company marketing fluconazole compared it with oral amphotericin B, a drug known to be poorly absorbed, thereby creating a bias in favour of fluconazole.49 We did not consider who is finally responsible for the selection of the comparator—investigators, regulatory agencies, or sponsors.

Finally, our results suggest that publication bias may explain our finding of bias in favour of outcomes of research funded by industry. Although research sponsored by industry was less likely to be published than research with other sources of funding, the two studies with this finding did not specifically examine whether non-publication applied just to research with non-significant outcomes.17 21 In the past few years, manufacturers have attempted to prevent studies which are unfavourable to their products from being published in several high profile cases.5052

Massie and colleagues raise another possible source of publication bias.33 They showed that research funded by industry appears more often in symposiums. Studies in symposiums are known to lack peer review and to favour the sponsor's product.14 53 Although the methods of industry funded trials are at least equal to those in studies funded by other sources, the absence of peer review may result in an overly favourable interpretation of the results of a trial. Rochon and colleagues noted that claims of superiority for the sponsor's product were often not supported by the data.43

Limitations

Our study has several limitations, primarily the difficulty in locating research examining the effects of company sponsorship. Our Medline and Embase searches found only 13 of the papers that we included,1618 20 23 2730 32 38 40 41 and the remaining 17 came from a search of the authors' personal files, suggestions by outside experts, or scanning of reference lists.1215 19 21 22 2426 31 3337 39 The inability to critically evaluate the methodology in the abstracts and journal letters is another possible source of bias, and the conclusions of two of the letters that we included30 40 have been criticised.54 55

Methods in studies sponsored by industry were at least as good as in studies with other sources of funding. Conclusions about overall quality can be influenced by the instrument used,9 and some of the scales may have missed important criteria.

What is already known on this topic

When a pharmaceutical company funds research into drugs, studies are likely to produce results favourable to the sponsoring company's product

What this study adds

Research funded by drug companies was more likely to have outcomes that favour the sponsor's product than research funded by other sources

This cannot be explained by the reported quality of the methods in research sponsored by industry

The result may be due to inappropriate comparators or to publication bias

Research sponsored by the pharmaceutical industry is facing a number of challenges. Questions have been raised about the mismatch between the research agendas of the pharmaceutical industry and consumers of research.56 Meta-analysts are confronted with the problems of duplicate publication of data from company funded trials and the withholding of data.49 57 58

Leading medical journals recently decided to establish more rigorous criteria for the acceptance of research sponsored by industry; this is a step in the right direction towards increasing the credibility of studies paid for by drug companies.58 The revised CONSORT statement should also help improve the quality of clinical research.59 60 In addition, authors and editors should consider including a statement concerning prior beliefs of the investigators about the uncertainty of the treatments that are reported. Finally, all clinical trials should be registered prospectively as the only way to prevent publication bias.61 The proposal to do so which was put forward in 198662 has been periodically renewed,6365 but to this date has not been implemented.

Footnotes

  • We thank Jiri Chard, David Liebeskind, Paula Rochon, and José Sacristan for additional information and data about their studies.

    Contributors: JL conceived and planned the study, did the Medline search, extracted the data, and wrote the paper. LAB planned the study, extracted the data, and wrote the paper. BD planned the study, checked the data extraction process, and wrote the paper. OC extracted the data and wrote the paper. JL is guarantor.

  • Funding No additional funding.

  • Competing interests BD has been funded by several pharmaceutical companies to perform research and has received speaking honorariums.

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
  27. 27.
  28. 28.
  29. 29.
  30. 30.
  31. 31.
  32. 32.
  33. 33.
  34. 34.
  35. 35.
  36. 36.
  37. 37.
  38. 38.
  39. 39.
  40. 40.
  41. 41.
  42. 42.
  43. 43.
  44. 44.
  45. 45.
  46. 46.
  47. 47.
  48. 48.
  49. 49.
  50. 50.
  51. 51.
  52. 52.
  53. 53.
  54. 54.
  55. 55.
  56. 56.
  57. 57.
  58. 58.
  59. 59.
  60. 60.
  61. 61.
  62. 62.
  63. 63.
  64. 64.
  65. 65.
View Abstract