Intended for healthcare professionals

Research Methods & Reporting

Out of sight but not out of mind: how to search for unpublished clinical trial evidence

BMJ 2012; 344 doi: https://doi.org/10.1136/bmj.d8013 (Published 03 January 2012) Cite this as: BMJ 2012;344:d8013
  1. An-Wen Chan, assistant professor and Phelan scientist
  1. 1Women’s College Research Institute, University of Toronto, Toronto, Ontario, Canada
  1. Corrrespondence to: A-W Chan anwen.chan{at}utoronto.ca

A key challenge in conducting systematic reviews is to identify the existence and results of unpublished trials, and unreported methods and outcomes within published trials. An-Wen Chan provides guidance for reviewers on adopting a comprehensive strategy to search beyond the published literature

Summary points

  • The validity of systematic reviews relies on the identification of all relevant evidence

  • Systematic reviewers should search for unpublished information on the methods and results of published and unpublished clinical trials

  • The potential sources of unpublished information on clinical trials have expanded over recent years

  • Recognition of the strengths and limitations of these key information sources can help to identify areas for further emphasis and improvement

Systematic reviews of randomised trials play a key role in guiding patient care and health policy. Their validity depends to a large extent on reviewers’ ability to retrieve relevant information from all existing trials. Unfortunately, about half of clinical trials remain unpublished after receiving ethics approval—particularly those with statistically non-significant findings.1 Even when published, most journal articles do not report all of the outcome data or key methodological information.2 3 The overall result is that the published literature tends to overestimate the efficacy and underestimate the harms of a given intervention, while providing insufficient information for readers to evaluate the risk of bias.

It is thus important that systematic reviewers adopt a comprehensive strategy to search beyond the published literature. The optimal systematic review would have complete information about every trial—the full protocol, final study report, raw dataset, and any journal publications and regulatory submissions.4 The eligibility and risk of bias for each trial could then be evaluated, regardless of its publication status.

There are several potential sources of unpublished information on trial methods and results (table). These sources can help to identify the existence and results of unpublished trials, as well as unreported outcomes within published trials. They can also provide methodological information that facilitates assessment of risk of bias, including the detection of discrepancies between unpublished and published methods.5 6 Systematic reviewers should consider using all potential information sources as part of their search strategy, while keeping in mind the strengths and limitations of each source (table).

Potential sources of unpublished information on trial methods and results

View this table:

Trial registries and results databases

Trial registries serve as a readily accessible online resource for identifying unpublished trials and unreported outcomes. Since 2005, prospective trial registration has gained broad acceptance as an important means of enhancing transparency and tracking the existence of clinical trials at inception. Key stakeholders—including medical journal editors, legislators, and funding agencies—provide enforcement mechanisms that have greatly improved adherence to registration practices.

Basic protocol information on ongoing and completed trials of any intervention type can be retrieved via the World Health Organization’s International Clinical Trials Registry Platform Search Portal (www.who.int/trialsearch/). This searches records from national and international trial registries that meet certain standards, including WHO Primary Registries and ClinicalTrials.gov. Users can search the main registry fields using key words related to the study topic, sponsor, recruitment status, and sites. When the same trial is registered in multiple registries, the WHO Search Portal displays similar records together to facilitate identification of duplicate records. Some registry websites also provide access to the history of changes to the registered information fields.

In addition to basic protocol information, certain registries house study results. Since 2008, ClinicalTrials.gov has had the legislative mandate to record summary results for trials (other than phase I) that involve a drug or device regulated by the US Food and Drug Administration.7 Sponsors are required by law to provide summary baseline and outcome data, which are displayed in a standard format.

Some pharmaceutical companies also maintain their own voluntary trial registers and results databases for drugs that have received regulatory approval. Systematic reviews have previously incorporated unpublished data retrieved from industry registers.8 These public registers provide a synopsis of trial methods and summary results as dictated by company policy. Information is presented in various formats with non-standardised content. For certain companies, there may be information posted for older trials of some marketed interventions. It should be noted that ClinicalStudyResults.org, the results database launched by the International Federation of Pharmaceutical Manufacturers and Associations in 2004, was to be discontinued by the end of 2011 because of overlap with other registries.

Beyond basic protocol information and results, trial registries have the potential to be the repository for full protocols. Legislation in the US allows for the possibility of requiring submission of full protocols to ClinicalTrials.gov for applicable trials.7 Furthermore, certain pharmaceutical companies are recognising the importance of public access to full protocols and have committed to posting them on their register for all published trials.9 These are promising first steps towards facilitating access to protocols for all trials, regardless of publication status.

Despite their importance, trial registries and results databases have several limitations. Firstly, there is no universal mechanism for ensuring adherence to standards for registration or results disclosure, meaning that not all trials will be captured. Journal policy will be ineffective for trials that are not intended for publication, while current legislation does not pertain to procedural, educational, and other unregulated interventions. Secondly, the quality of registered information is highly variable and often uninformative.7 10 11 12 13 Changes to registered information are common,12 meaning that systematic reviewers should review the history of amendments for each registry record. Thirdly, even when a trial is fully registered with complete summary results presented, there is a limited amount of methodological information available that is largely inadequate for assessing the risk of bias.10 This concern would be addressed if full protocols were made available on the registries.9 14 Finally, most trials will not have been registered prior to the introduction of International Committee of Medical Journal Editors policy and WHO standards in 2005.

Regulatory agencies

Regulatory agencies have access to substantially more clinical trial information than the healthcare providers, patients, and researchers who use and evaluate the interventions. Successful attempts to obtain access to regulatory data have previously necessitated litigation and incurred lengthy delays.15 16 17 Over recent years, regulatory agencies have recognised the need to address this untenable situation by increasing public access to information from regulatory submissions.18 19

There are currently two main routes for reviewers to obtain trial data from regulatory agencies—scientific reviews posted in online databases,20 21 and written requests to regulatory agencies.15 Scientific reviews of regulatory submissions contain a narrative summary of the clinical trials that form the basis for approval of regulated drugs. These documents are generally available on searchable internet databases provided by the US Food and Drug Administration and the European Medicines Agency:

Relevant clinical trial summaries are generally labelled as “Statistical review” on Drugs@FDA, and “Scientific Discussion” in EPAR. The Pharmaceuticals and Medical Devices Agency in Japan (http://www.pmda.go.jp/english/service/approved.html) also posts a limited number of reviews with English translations for select drugs and devices.

Limitations of the scientific reviews obtained from regulatory agency websites include the variable presentation format and the lack of text search facility for some scanned documents. In addition, the content is not standardised, information deemed to be commercially sensitive is redacted, and insufficient methodological detail is provided to assess the risk of bias for a trial. Furthermore, many trials are not included in regulatory databases, such as trials of devices and non-regulated interventions. Most trials conducted after regulatory approval would not be captured. For the European Medicines Agency, drugs that are approved by regulators in individual countries but not the central agency will not have public assessment reports available. Drugs@FDA includes information on withdrawn drugs but does not provide scientific reviews for unapproved drugs or drugs approved before 1998.

A second approach has the potential to yield more detailed information from regulatory agencies. Reviewers can make written requests to access the trial protocols and detailed clinical study reports submitted by sponsors. As of December 2010, the European Medicines Agency has committed to accommodating such requests for documents contained in regulatory submissions for drugs, subject to redaction of commercially sensitive information.19 This important advance will be expanded in the future to include proactive public disclosure of documents on the European Medicines Agency website as part of routine practice. The US Food and Drug Administration has previously granted access to clinical trial documents in response to litigation relating to freedom of information requests16 17 and is also exploring ways to increase transparency.18

Limitations of this second approach include potentially lengthy delays in receiving a final decision from regulators, resource-intensive appeals or litigation for denied requests, redaction of potentially important information from documents, and lack of information on interventions other than regulated drugs and devices.

Contacting trialists and sponsors

Systematic reviewers have had variable success in contacting trialists, clinicians, and sponsors for information about unpublished trials.4 22 23 24 25 Efforts to obtain full trial protocols from trialists have been largely disappointing.26 27 On the other hand, surveys soliciting information on the existence and statistical significance of unreported outcomes for published trials have had higher response rates from trialists.28 29 These surveys have also yielded information about the reasons for changing or omitting trial outcomes.

Logistical obstacles include the burden of identifying up to date contact information and sending inquiries and reminders to a potentially large number of individuals who might have knowledge about existing trials. It is also likely that trials for which additional information is provided by investigators or sponsors will differ systematically from trials without such information provided.

Systematic reviewers will need to weigh up the potential yield and costs of contacting investigators and sponsors, which will vary depending on the topic and scope of the review. At a minimum, for each trial identified in the systematic review, it would be reasonable for reviewers to contact investigators to request full protocols as well as information on unreported outcomes, unpublished trials, and other areas of potential bias.

Other sources of information

In some cases trial protocols and results can be obtained from litigation documents. Examples include researchers who had access to internal company documents while serving as expert witnesses in litigation against pharmaceutical companies.30 31 32 In many jurisdictions, these documents are deemed confidential and their use is restricted to the purposes of the particular litigation—unless unsealed through a court order or agreement by the company. Systematic reviewers who are external to the litigation could submit a request to have the documents unsealed by the court to serve the public interest, although this approach has not been widely tested for pharmaceutical data. More extensive experience with public availability and archiving of litigation documents exists for other industries.33

Another potential source of information consists of conference abstracts.34 The Cochrane handbook lists several databases of abstracts that can be useful to search.35 Given the limited amount of information on trial methods and results contained in abstracts, their usefulness lies mainly with identifying the existence of a trial and the types of outcomes measured.

Finally, an internet search of key words can be done to locate full trial protocols in a relatively short amount of time. The median search time in one systematic review was 12 minutes per trial, with protocols being found for five of 42 trials.36 The retrieved documents are often those posted on the websites of specific trials, trial groups, and funders.

Conclusions

Given the dangers of selective data suppression and biased study design or conduct, it is critical that systematic reviewers search beyond the literature for additional information on both published and unpublished trials. The potential sources of information on study methods and results have expanded over recent years, particularly for pharmaceutical trials. These sources can provide complementary trial information that can be collated and compared to identify discrepancies and evaluate the risk of bias.

It is important to recognise the limitations and variable yield of existing information sources. Much work remains to ensure that comprehensive, high quality information is publicly available for all trials, including full protocols, clinical study reports, and raw datasets.4 14 37 There is also a need to develop rigorous methods for reviewing the large amount of unpublished trial information that can potentially be retrieved.4 15 Only with continued advances in access to clinical trial information can the systematic evaluation of health interventions become more accurate, efficient, and reliable for patient care.

Notes

Cite this as: BMJ 2012;344:d8013

Footnotes

  • Editorials doi:10.1136/bmj.d8158
  • Contributors: A-WC was responsible for interpretation of information, drafting the article, and final approval of the version to be published.

  • Competing interests: All authors have completed the Unified Competing Interest form at http://www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.”

  • Provenance and peer review: Commissioned; externally peer reviewed.

References

View Abstract