Article Text
PDF
Abstract
Objective To evaluate clinical trial registration, reporting and publication rates for new drugs by: (1) legal requirements and (2) the ethical standard that all human subjects research should be publicly accessible to contribute to generalisable knowledge.
Design Cross-sectional analysis of all clinical trials submitted to the Food and Drug Administration (FDA) for drugs approved in 2012, sponsored by large biopharmaceutical companies.
Data sources Information from Drugs@FDA, ClinicalTrials.gov, MEDLINE-indexed journals and drug company communications.
Main outcome measures Clinical trial registration and results reporting in ClinicalTrials.gov, publication in the medical literature, and compliance with the 2007 FDA Amendments Acts (FDAAA), analysed on the drug level.
Results The FDA approved 15 drugs sponsored by 10 large companies in 2012. We identified 318 relevant trials involving 99 599 research participants. Per drug, a median of 57% (IQR 32–83%) of trials were registered, 20% (IQR 12–28%) reported results in ClinicalTrials.gov, 56% (IQR 41–83%) were published, and 65% (IQR 41–83%) were either published or reported results. Almost half of all reviewed drugs had at least one undisclosed phase II or III trial. Per drug, a median of 17% (IQR 8–20%) of trials supporting FDA approvals were subject to FDAAA mandated public disclosure; of these, a median of 67% (IQR 0–100%) were FDAAA-compliant. 68% of research participants (67 629 of 99 599) participated in FDAAA-subject trials, with 51% (33 405 of 67 629) enrolled in non-compliant trials. Transparency varied widely among companies.
Conclusions Trial disclosures for new drugs remain below legal and ethics standards, with wide variation in practices among drugs and their sponsors. Best practices are emerging. 2 of our 10 reviewed companies disclosed all trials and complied with legal disclosure requirements for their 2012 approved drugs. Ranking new drugs on transparency criteria may improve compliance with legal and ethics standards and the quality of medical knowledge.
- MEDICAL ETHICS
- PUBLIC HEALTH
- GENERAL MEDICINE (see Internal Medicine)
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This study uniquely analyses the transparency of clinical trial information for new drugs, whereas other studies analyse transparency on the trial level. It also debuts an innovative strategy for reforming areas of low transparency.
This study uses Food and Drug Administration (FDA) databases as a key data source, because they characterise all clinical trials supporting new drug approvals. Prior studies evaluate the transparency of already registered trials in ClinicalTrials.gov, providing limited insights into the many unregistered studies.
This study takes a uniquely comprehensive approach, analysing five critical elements of transparency for trials of new drugs: (1) registration, (2) results reporting, (3) publication in a medical journal, (4) compliance with legal disclosure requirements and (5) adherence with the ethics standards enshrined in the Common Rule, Helsinki Declaration and elsewhere which state that all trials should be ‘designed to develop or contribute to generalisable knowledge’, that is, should be publicly accessible.
A main limitation for this study is the need to extend and repeat the analysis beyond the 15 drugs approved by the FDA in 2012 that were manufactured by large companies, to include drugs approved in other years and sponsored by other institutions. We are in the process of expanding the rankings to include drugs approved in other years as well as additional trial sponsors.
Introduction
For decades, many clinical trials have been publicly inaccessible, raising ethics, medical practice and population health concerns. While recent transparency efforts have improved practices, a significant portion of both commercially and publicly funded trials and trial results still remain inaccessible, because they are unregistered and their results are unreported in trial registries,1 ,2 or they are never published in the medical literature.3 ,4
Studies have shown that roughly 30–50% of clinical trials remain unpublished, often years after their completion,5 ,6 and most fail to meet baseline legal disclosure requirements, such as those established in the 2007 US Food and Drug Administration Amendments Act (FDAAA).7 Moreover, studies that are published by sponsors, journals and researchers tend to show favourable or statistically significant results.8–11 This selective trial dissemination can distort the medical evidence and challenge physicians, prescription guideline writers, payers and formulary decision-makers’ abilities to recommend and provide the right drugs for the right patients. It also represents a violation of the rights of human research subjects, as experimenting on humans is largely justified by its potential to contribute to generalisable knowledge (as stated in the 1981 US Common Rule). Furthermore, transparency may be essential to ensuring the integrity and trustworthiness of the clinical research enterprise.
Despite numerous major reform strategies, the transparency problem persists, raising questions of what more can be done. Efforts include the 1997 US Food and Drug Administration Modernization Act requiring the registration of drug trials for serious or life-threatening conditions, FDAAA requiring that select trials be registered and publicly report results, and the 2008 World Medical Association guidelines identifying trial registration and results reporting as an ethical obligation in the Declaration of Helsinki. The International Committee of Medical Journal Editors, Institute of Medicine, individual drug companies and their trade associations, the European Medicines Agency, WHO and Bill and Melinda Gates Foundation have also made efforts to improve transparency in clinical research. Recently, both the Department of Health and Human Services and National Institutes of Health (NIH) called for public comment on two new proposals to further expand access to clinical trial information. The DHHS proposal would substantially expand the scope of registration and results reporting requirements under FDAAA.12 ,13 The NIH proposal would require registration and results reporting for all NIH funded clinical trials, including phase I trials.14
To help understand the efficacy of these transparency efforts for new drugs and how to improve them, this paper examines whether clinical trials for drugs approved by the FDA in 2012, which were sponsored by large companies, were registered, reported, published in the medical literature and complied with legal transparency requirements established in FDAAA.
This study and approach are novel for at least five reasons. First, we evaluated the transparency around individual new drugs. Previous studies generally evaluate transparency on the trial level. We thought evaluating on the drug level could help make the transparency problem more understandable and proximate for stakeholders who consume, prescribe, reimburse, stockpile or otherwise regulate medicines and vaccines. Moreover, drug level transparency evaluations are critical to improving clinical practice. When a new drug enters the market, the trials we evaluated in our rating system contain the safety and efficacy profile for that drug, and all, or nearly all, available evidence to inform clinical practice.
Second, we used FDA approval packages as a key data source, because they characterise all clinical trials supporting new drug approvals. Prior studies evaluate the transparency of already registered trials in ClinicalTrials.gov, which provide limited insights into the many unregistered studies. Third, we focused on large companies because, as a group, they sponsor a significant portion of the trials conducted annually and the majority of new drug applications (NDAs) submitted to the FDA. Also, they were expected to have the infrastructure to comply with regulatory and ethics standards. Thus, we likely captured a best-case scenario. Fourth, we evaluated transparency on both legal and ethics standards, providing a uniquely comprehensive overview. Lastly, we introduce an innovative strategy to improve the state of transparency for drugs: an annual transparency scorecard that audits and ranks all new medicines and vaccines.
Methods
Data sources
We used data collected from Drugs@FDA, a publicly accessible database containing records of FDA drug regulatory decisions, including drug approvals and medical and scientific reviews of approved drugs; ClinicalTrials.gov, a clinical trial registry and database maintained by the National Library of Medicine (NLM) at the NIH; MEDLINE-indexed journals (accessed through PubMed); information from the NLM to identify certificates of delay (provided by Tse T to Anderson M: personal communication); information from large companies that had new drugs approved by the FDA in 2012; and pharmaceutical company press releases. The databases were accessed several times between October 2013 and April 2014.
Study samples
Drugs
New drugs approved by the FDA in the calendar and fiscal year of 2012 were identified from FDA reports,15 ,16 and included innovative and novel drugs and new molecular entities (NMEs), henceforth referred to simply as ‘drugs’. We restricted the total number of drugs to those that were sponsored by large biotechnology and pharmaceutical companies, defined as the 20 institutions with the highest market capitalisations in 2012.17
Clinical trials
All trials conducted to gain FDA approval in 2012 for each drug
Each drug's 2012 FDA approval package was located in the Drugs@FDA database. We reviewed all pages of a drug's summary review, Medical Review(s), Chemistry Review(s), Pharmacology Review(s), Statistical Review(s), Clinical Pharmacology Biopharmaceutics Review(s), Risk Assessment and Risk Mitigation Review(s), and other reviews to create a list of every clinical trial reviewed by the FDA to approve each drug. Where possible, the basic characteristics of each trial were catalogued, including the organisational identification number, phase, study population, number of research participants, primary end point(s), study start and completion date(s), location, and description of the treatment (eg, dosage and comparators), participants received in the various arms, and whether the trial was controlled and/or interventional. We excluded any trials that were terminated without enrolment of participants, still ongoing or not at least 1 year past their primary completion date by our study cut-off date of 1 February 2014.
FDAAA applicable trials for each drug
We narrowed the ‘all trials’ sample to only those subject to mandatory registration and reporting requirements under FDAAA, that is, generally, ‘controlled clinical investigation(s), other than a phase I clinical investigation, of a drug subject to section 505 of the Federal Food, Drug, and Cosmetic Act or to section 351 of this Act’.18 These trials should have ‘either initiated after 27 September 2007, or initiated on or before that date and were still ongoing as of 26 December 2007’, and meet one of the following conditions:
The trial has one or more sites in the USA,
The trial is conducted under an FDA investigational new drug application (IND), or
The trial involves a drug or biological that is manufactured in the USA or its territories and is exported for research.19
Because of conflicting understandings on the reach of FDAAA, we created two sample pools of FDAAA-subject trials, one for controlled and one for interventional trials. Trials with unknown phases or that were listed as phase I/II in at least two primary data sources were excluded from the FDAAA analysis (n=1).
Main outcome measures
Determining transparency according to the ethics standard that all trial results should be publicly accessible
We ascertained in ClinicalTrials.gov whether each identified trial from the FDA approval packages of each drug (the ‘all trials’ study sample) was registered and reported results. Search terms to locate and match trials included the trial's organisational identification number (org ID), product name, number of trial participants and other trial characteristics captured from the approval packages. Once identified, we abstracted the National Clinical Trial number (NCT number), number of research participants enrolled in the various treatment arms, description of the treatment (eg, dosage and comparators), whether the trial was controlled and/or interventional, primary outcome measurements, trial start date, registration date, primary completion date (date the last participant was examined and data for the primary outcome measure collected) and any links to clinical study reports. Any clinical trial(s) with results received by ClinicalTrials.gov on or before 1 February 2014 was deemed to have results publicly available. This study cut-off date was chosen to provide at least 13 months for trial results disclosure post-FDA approval of a drug.
Second, using search terms that included the branded drug name or active ingredient and trial indication, we determined from PubMed whether individual trials for each drug were published in a MEDLINE-indexed journal, on or before our cut-off date. We then matched at least two of the following characteristics—the NCT number or organisational trial identification number, number of enrolled research participants, descriptions of the treatment (eg, dosage and comparators) and/or primary outcome measurements—in the publication with what was in ClinicalTrials.gov or the Drugs@FDA databases. We also reviewed papers listed on ClinicalTrials.gov for registered trials and used the same matching criteria. Papers summarising and reviewing multiple phase I trial results in a single publication, although rare, were counted.
Determining transparency according to FDAAA legal requirements
We reviewed whether FDAAA applicable trials (for both the ‘controlled’ and ‘interventional’ samples) had timely registration and reporting as defined by FDAAA. Registration (which in our case are trials for approved drugs), in general, should occur within 21 days after enrolling the first participant. Results should be reported, generally, no later than 12 months after the trial's primary completion date, in ClinicalTrials.gov, although results submissions can be delayed by submitting certificates to the NIH (see online supplementary appendices 1 and 2). If a trial met FDAAA requirements for both registration and disclosure of results, it was counted as compliant with legal requirements.
Validation
Data were extracted by at least two research assistants (working independently and blinded to each other's work), with discrepancies resolved through discussion and consensus (see Acknowledgments for a list of research assistants). Our final data sets for each drug were sent to each NDA company sponsor to verify the accuracy and completeness of our extracted information. Data and input received from companies (response rate was 100%) were verified by public data sources.
Statistical analysis
We used descriptive statistics to calculate the median number of clinical trials per drug that were registered and reported results in ClinicalTrials.gov, were published in a MEDLINE-indexed journal, and were publicly accessible. Public accessibility of a trial was defined as being either reported in Clinicaltrials.gov or published in a MEDLINE-indexed journal. We also used descriptive statistics to calculate the median number of clinical trials per drug subject to FDAAA that were in compliance with the statute. All data were collected and analysed in Microsoft Excel V.2013 (Redmond, Washington, USA).
Results
In 2012, the FDA approved 39 novel new medicines, known as NMEs, and 35 novel drugs. Combining these lists, the FDA approved a total of 48 new drug entities, 15 of which were sponsored by 10 large pharmaceutical or biotechnology companies with market capitalisations valued over $19 billion. A total of 342 trials were conducted to gain regulatory approval of the 15 drugs, 24 of which were excluded from our analysis, leaving 318 trials involving 99 599 participants relevant to our study, a median of 17 trials per drug (see table 1).
Transparency Index: ranking of new drugs according to the ethics standard that all trial results should be publicly available to contribute to generalisable knowledge
Transparency evaluated by the ethics standard that all trial results should be publicly accessible
The median proportion, per drug, of publicly registered trials was 57% (IQR 32–83%), of trials reporting results in ClinicalTrials.gov, 20% (IQR 12–28%), and of published trials, 56% (IQR 41–83%; see table 1). A median of 65% (IQR 41–83%) of clinical trial results were publicly available, that is, the results were either reported in ClinicalTrials.gov or published in the medical literature, but with considerable variation (see table 1). Importantly, among trials that reported results in ClinicalTrials.gov, a median of 100% (IQR 86–100%) were also published.
Among the 35% of trials, per drug, with results unavailable in either the medical literature or ClinicalTrials.gov, a median of 91% (IQR 60–100%) were phase I, 0% were phase II (IQR 0–15%), 0% were phase III trials (IQR 0–2%) and 0% were of unknown phase. Among the 15 drugs, 20% had at least one publicly unavailable phase III trial, 27% had at least one undisclosed phase II trial and 47% one of either. In total, 5566 research participants (of the 99 599 total participants) participated in publicly undisclosed trials for these 15 drugs.
Public availability of clinical trial information varied widely by company, and sometimes within companies for those with multiple drugs approved in 2012 (table 1). For example, 3 of the 10 companies (GSK, J&J and Pfizer) publicly disclosed all clinical trial results for at least one of their reviewed drugs, whereas the lowest scoring company, Gilead, disclosed 21% (7 of 34) of the trial results for its HIV medicine Stribild.
Transparency evaluated by FDAAA legal requirements
The legal requirements enshrined in FDAAA (table 2) offer at least two potential interpretations for what constitutes an applicable clinical trial: controlled and interventional trials. Applying the ‘controlled’ definition, a median of 17% (IQR 8–20%) of trials per drug were subject to legal disclosure requirements, hereafter referred to as ‘applicable trials’ .A median of 100% (IQR 93–100%) of these trials met registration requirements, whereas 67% (IQR 0–100%) met reporting requirements. Overall, per drug, a median of 67% (IQR 0–100%) of applicable trials fully complied with the law (see table 2), with considerable variation. Sixty-eight per cent of research participants (67 629 of 99 599) participated in FDAAA-subject trials, with 51% of them (33 405 of 67 629) enrolled in non-compliant trials. Six of the 10 reviewed companies showed 100% compliance with the law for at least one drug. However, an almost equal number, 5 of 10, had at least one drug that was 0% compliant.
Legal Compliance Index: ranking of new drugs according to their compliance with disclosure requirements under the Food and Drug Administration Amendments Acts (FDAAA)
Applying the ‘interventional’ definition, a median of 19% (IQR 15–29%) of trials, per drug, were subject to legal disclosure requirements under FDAAA. A median of 100% (IQR 93–100%) of these trials met registration requirements, whereas 71% (IQR 0–100%) met reporting requirements. Overall, a median, per drug, of 71% (IQR 0–100%) of applicable trials complied with FDAAA (see table 2). Sixty-nine per cent of research participants (68 703 of 99 599) participated in FDAAA-subject trials, with 51% of them (33 786 of 68 703) enrolled in non-compliant trials. Five of the 10 reviewed companies had at least one drug that showed 100% compliance with FDAAA. The same number of companies (5 of 10) had at least one drug that was 0% compliant.
Discussion
Medical practice remains largely an empirical discipline, highly dependent for its advancement on the complete and accurate sharing of information. Nowhere is this truer than in the reporting of clinical trials, in particular those that support the efficacy and safety of new medicines. The purpose of this study was to review all new drugs approved by the FDA in 2012 that were sponsored by large companies, identifying all clinical trials that supported their approval and determining whether the trials were publicly registered and had trial results reported in ClinicalTrials.gov, were published in the medical literature within at least 13 months of FDA approval, and complied with federal disclosure laws. While nearly two-thirds of clinical trials, per drug, were publicly disclosed, there was wide variation among drugs and companies. At first approximation, it may seem difficult to understand failures to comply with federal law, now 8 years old, whose origins track back to 1997, and even more difficult to understand failures to meet the over-riding ethics obligation that human research be designed to contribute to generalisable knowledge.
Transparency by legal standards
There are at least three reasons why compliance with current disclosure laws might be suboptimal. First, legal requirements are perceived to be unclear or ambiguous, as a spectrum of interpretations of FDAAA has emerged. Some companies believe only controlled trials are subject to mandatory disclosure, others interventional trials. Some believe that results are due 1 year after a trial's primary completion date regardless of whether the drug has been approved, while others believe results are not due until 30 days post-FDA approval of a trial's investigated indication. There is also disagreement about the role of certificates of delay. These varying interpretations for FDAAA came to light during our discussions with the ranked companies. Recall that we sent all data to the companies whose products we scored (with a 100% response rate).
Second, mergers, acquisitions, collaborations and licensing agreements may complicate compliance. Two companies in our sample acquired or licensed drugs initially developed by smaller companies, and another used a partner company for some trials, raising questions about whose responsibility it was to ensure trials complied with FDAAA.
Finally, compliance may be affected by a perceived lack of enforcement. FDAAA empowers the FDA to impose a $10 000 a day penalty for non-compliance. To date, this penalty has never been imposed.
Transparency by ethics principles
In contrast to legal requirements, that applied to roughly only one-fifth of all clinical trials supporting a new drug approval, ethics standards enshrined in the Common Rule, Helsinki Declaration and elsewhere, apply to all clinical trials. Ethically, all research involving human participants should be ‘designed to develop or contribute to generalisable knowledge’,20 that is, should be publicly accessible. Surprisingly, adherence to this ethics standard was similar to that to legal standards: the results of approximately two-thirds of the studied trials, per drug, were publicly accessible, either through results reporting in ClinicalTrials.gov or publication. Adherence may be less than complete because companies may not act without an authoritative body promulgating an organising policy. Perhaps the new WHO guidelines calling for all trial results to be publicly disclosed, including phase I trials, may serve this purpose.21 Our proposed pharmaceutical transparency scorecard may also help.
Motivating transparency
Given the wide variation in compliance with both legal and ethical standards across drugs and companies, we propose continuing our clinical trials transparency monitoring, evaluations and scoring of new drugs approved by the FDA, along with their sponsors. These ongoing rankings—developed initially with support from Harvard University, Duke University, Susan G. Komen Foundation and the Raskob Foundation (for a full list of sponsors, see the Acknowledgements section)—will be conducted annually under the auspicious of Bioethics International, with grant support from the Laura and John Arnold Foundation.22–24
This system will help identify best practices, incent better behaviours and standardise the industry's practices and thereby contribute importantly to an enrichment of medical knowledge. Moreover, the scorecard and rankings have the potential to benefit consumers of clinical trial information by helping to assure them of the integrity and completeness of their data. Not least, full transparency of clinical trials would also strengthen the protection of human research participants by avoiding their unknowing recruitment into already failed experiments.
Limitations
Several limitations deserve further consideration. We limited our study to 1 year of FDA approvals: 15 drugs sponsored by 10 large companies. Further measurements are needed of trials for drugs approved in other years, of additional sponsors (such as smaller companies and academic centres), and of the quality of reported information. We are in the process of expanding the rankings to include drugs approved in other years as well as additional trial sponsors. At times, mergers, acquisitions, subsidiaries, partnerships and licensing practices complicated determining the sponsors and responsible parties for trial transparency. Lastly, there is some disagreement on whether the scores and index we presented should include a weighting mechanism to account for the variation in the number of trials conducted per drug to gain FDA regulatory approval.
Conclusion
Nearly two-thirds of clinical trials supporting new drugs approved by the FDA in 2012 were publicly disclosed, perhaps encouraging but below both legal and ethics standards. While several large companies’ drugs were superseding legal requirements, others had low rates of transparency. Implementing a transparency scorecard and ranking system for all newly approved drugs could motivate and increase transparency, thereby supporting existing transparency initiatives, advancing clinical innovation, promoting a trustworthy innovation sector and strengthening protection of human research subjects globally.
Acknowledgments
The authors thank Robert Califf (Duke University), Aaron Kesselheim (Brigham and Women's Hospital and Harvard Medical School), Kevin Schulman (Duke University) and Arti Rai (Duke University) for comments on this paper; Brittany Rush, Jason Dong, Marsha Sukash, Julie Dorais, Jessica Ericson, Rajan Patel, Eidan Jacob, Brooke Whitfield, Priyanka Kanal, Raichel Perper and Nicole Miller for data-gathering assistance; and Malcolm Salter (Harvard Business School), Larry Lessig (Harvard Law School), Arthur Caplan (New York University), Ed Balleisen (Duke University), the directors of Bioethics International, and the fellows of Edmond J. Safra Center for Ethics at Harvard University for their input and feedback on the index and ranking concept. They also thank Janet Woodcock (FDA) and Tony Tse (NIH) for answering FDAAA-related questions and Monique Anderson (Duke University) for providing access to certificate of delay data.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
- Data supplement 1 - Online supplement
Footnotes
Contributors JEM, JSR and DK were responsible for the conception and design of this work. JEM drafted the manuscript, conducted the statistical analysis, and is guarantor. All authors participated in the analysis and interpretation of the data and critically revised the manuscript for important intellectual content.
Funding The conceptualisation and pilot of this study and index (called the Good Pharma Scorecard) were supported by Harvard University's Edmond J. Safra Center for Ethics; Duke University's Kenan Institute for Ethics, Trinity College of Arts and Sciences and Fuqua School of Business; Fordham University's Global Healthcare Innovation Management Center; Susan G. Komen Foundation; Raskob Foundation; and Bioethics International. A conference exploring the rating concept (co-chaired by JEM and Nir Eyal) was held at Harvard Medical School, funded by the Division of Medical Ethics (currently the Harvard Medical School Center for Bioethics); the Petrie-Flom Center for Health Law, Biotechnology, and Bioethics at Harvard Law School; the Harvard Global Health Institute; and the Harvard University Program in Ethics and Health. JSR is supported by the National Institute on Aging (K08 AG032886) and by the American Federation for Aging Research through the Paul B. Beeson Career Development Award Program and receives research support through Yale University from Medtronic, Inc and Johnson & Johnson to develop methods of clinical trial data sharing, from the Centers of Medicare and Medicaid Services (CMS) to develop and maintain performance measures that are used for public reporting, and from the Food and Drug Administration (FDA) to develop methods for postmarket surveillance of medical devices. Bioethics International and the Laura and John Arnold Foundation will support further development and implementation of the ranking system described in this paper.
Competing interests JSR is supported by the National Institute on Aging (K08 AG032886) and by the American Federation for Aging Research through the Paul B. Beeson Career Development Award Program and receives research support through Yale University from Medtronic, Inc and Johnson & Johnson to develop methods of clinical trial data sharing, from the Centers of Medicare and Medicaid Services (CMS) to develop and maintain performance measures that are used for public reporting, and from the Food and Drug Administration (FDA) to develop methods for postmarket surveillance of medical devices.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.