You are here

  • Home
  • Archive
  • Volume 5, Issue 5
  • Evidence-informed recommendations to reduce dissemination bias in clinical research: conclusions from the OPEN (Overcome failure to Publish nEgative fiNdings) project based on an international consensus meeting

Article Text

Article menu
XML
Evidence based practice
Research
Evidence-informed recommendations to reduce dissemination bias in clinical research: conclusions from the OPEN (Overcome failure to Publish nEgative fiNdings) project based on an international consensus meeting
  1. Joerg J Meerpohl1,
  2. Lisa K Schell1,
  3. Dirk Bassler2,3,
  4. Silvano Gallus4,
  5. Jos Kleijnen5,6,
  6. Michael Kulig7,
  7. Carlo La Vecchia8,
  8. Ana Marušić9,
  9. Philippe Ravaud10,
  10. Andreas Reis11,
  11. Christine Schmucker1,
  12. Daniel Strech12,
  13. Gerard Urrútia13,
  14. Elizabeth Wager14,
  15. Gerd Antes1
  16. the OPEN project consortium
  1. 1German Cochrane Centre, Medical Center—University of Freiburg, Freiburg, Germany
  2. 2Department of Neonatology, University Hospital Zurich, Zurich, Switzerland
  3. 3Center for Pediatric Clinical Studies, University Children's Hospital Tuebingen, Tuebingen, Germany
  4. 4Department of Epidemiology, IRCCS—Istituto di Ricerche Farmacologiche “Mario Negri”, Milan, Italy
  5. 5Kleijnen Systematic Reviews Ltd, York, UK
  6. 6School for Public Health and Primary Care, Maastricht University, Maastricht, The Netherlands
  7. 7Medical Consultancy Department, Gemeinsamer Bundesausschuss (G-BA), Berlin, Germany
  8. 8Department of Clinical Sciences and Community Health, University of Milan, Milan, Italy
  9. 9Department of Research in Biomedicine and Health, University of Split School of Medicine, Split, Croatia
  10. 10INSERM U738 Research Unit, Paris Descartes University, Paris, France
  11. 11Global Health Ethics, Department of Knowledge, Ethics and Research, World Health Organization, Geneva, Switzerland
  12. 12CELLS—Centre for Ethics and Law in the Life Sciences, Institute of History, Ethics and Philosophy of Medicine, Hannover Medical School, Hannover, Germany
  13. 13Centro Cochrane Iberoamericano-Servei d'Epidemiologia Clínica i Salut Pública, Institut d'Investigació Biomèdica Sant Pau, CIBERSP, Barcelona, Spain
  14. 14Sideview, Princes Risborough, UK
  1. Correspondence to Dr Joerg J Meerpohl; meerpohl{at}cochrane.de

Abstract

Background Dissemination bias in clinical research severely impedes informed decision-making not only for healthcare professionals and patients, but also for funders, research ethics committees, regulatory bodies and other stakeholder groups that make health-related decisions. Decisions based on incomplete and biased evidence cannot only harm people, but may also have huge financial implications by wasting resources on ineffective or harmful diagnostic and therapeutic measures, and unnecessary research. Owing to involvement of multiple stakeholders, it remains easy for any single group to assign responsibility for resolving the problem to others.

Objective To develop evidence-informed general and targeted recommendations addressing the various stakeholders involved in knowledge generation and dissemination to help overcome the problem of dissemination bias on the basis of previously collated evidence.

Methods Based on findings from systematic reviews, document analyses and surveys, we developed general and targeted draft recommendations. During a 2-day workshop in summer 2013, these draft recommendations were discussed with external experts and key stakeholders, and refined following a rigorous and transparent methodological approach.

Results Four general, overarching recommendations applicable to all or most stakeholder groups were formulated, addressing (1) awareness raising, (2) implementation of targeted recommendations, (3) trial registration and results posting, and (4) systematic approaches to evidence synthesis. These general recommendations are complemented and specified by 47 targeted recommendations tailored towards funding agencies, pharmaceutical and device companies, research institutions, researchers (systematic reviewers and trialists), research ethics committees, trial registries, journal editors and publishers, regulatory agencies, benefit (health technology) assessment institutions and legislators.

Conclusions Despite various recent examples of dissemination bias and several initiatives to reduce it, the problem of dissemination bias has not been resolved. Tailored recommendations based on a comprehensive approach will hopefully help increase transparency in biomedical research by overcoming the failure to disseminate negative findings.

  • EPIDEMIOLOGY
  • JOURNALISM (see Medical Journalism)

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

https://doi.org/10.1136/bmjopen-2014-006666

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • Comprehensive set of evidence-informed recommendations aimed at reducing dissemination bias based on work conducted as part of European Union FP7 project (Overcome failure to Publish nEgative fiNdings (OPEN) project; http://www.open-project.eu).

    • General and specific recommendations, both targeted at various key stakeholder groups involved in the knowledge generation and translation process.

    • Use of a rigorous and transparent methodology to develop recommendations including at consensus meeting including external experts and stakeholders.

    • Iteration of some recommendations that are not new but have so far not been broadly implemented.

    Introduction

    In order to make informed decisions, healthcare practitioners, consumers, public health professionals, policymakers, and health and research funding bodies rely on evidence from clinical research. The generation of evidence is made possible because patients participate in clinical trials, and accept any research-related risks and burdens. To inform medical decision-making, demonstrate respect to trial participants and maintain public trust in clinical research, it is important that such evidence is made available in an easily accessible and unbiased way.1 However, many research findings are either not published at all (an estimated 50%),2 or only selectively, that is, with a bias towards specific aspects or presenting only partial information. Healthcare professionals and policymakers are therefore frequently unable to make decisions based on the entire relevant research evidence.3 This problem has been called ‘publication bias’,4 or—as more recently suggested—‘dissemination bias’, to reflect the multiple facets of this problem.5 ,6 Dissemination bias has recently received attention due to initiatives such as the AllTrials campaign (http://www.alltrials.net7) and the RIAT proposal,8 the discussion of oseltamivir in the context of avian influenza,9 and announcements from companies, for example, GlaxoSmithKline, to grant access to patient-level data from all company trials.10 However, general awareness of this problem and, even more importantly, the necessity to address and resolve it, has presumably not yet been fully recognised by many stakeholders.

    Therefore, the European Commission called for projects to investigate the extent and impact of dissemination bias, and to develop recommendations to overcome incomplete or selective access to trial results. The European Union has committed funds to two projects, Overcome failure to Publish nEgative fiNdings (OPEN; http://www.open-project.eu)11 and UNCOVER (http://www.ait.ac.at/uncover), supported within the Seventh Framework Program. The two projects used different methodologies but aimed to address similar problems, namely, the non-publication—or in a broader sense the non-dissemination—of the complete data of all clinical trials.

    We present the results of the 24-month OPEN project that ran from November 2011 to October 2013. As an interdisciplinary initiative, it brought together academics and stakeholders from across Europe, with the aim of developing evidence-informed recommendations and strategies for overcoming the failure to publish negative research findings.

    One activity of OPEN was to conduct a series of systematic reviews to assess the occurrence of non-publication of research findings and the resulting dissemination bias. These reviews addressed aspects such as existing terminology to describe problems of publication and related biases,5 available methods to detect and measure dissemination bias,12 and the extent,13 and impact,14 of the problem of non-publication of research findings.

    Another focus of OPEN was to describe current practices by various players involved in knowledge generation and knowledge translation to provide insights about how to avoid or reduce bias due to non-publication of research findings, and to identify ways to change practices that contribute to the problem. This has been addressed by assessing and evaluating the policies and procedures in place for preventing dissemination bias by the main parties involved in approving, funding, conducting, publishing, disseminating and assessing clinical research. Different work packages of OPEN have surveyed representatives of funding agencies, the pharmaceutical industry, research ethics committees, research institutions, researchers,15 trial registers,16 medical journals,17 regulatory agencies and benefit (health technology) assessment agencies such as NICE in the UK (http://www.nice.org.uk) and IQWiG in Germany (http://www.iqwig.de).

    Findings from the OPEN work packages informed a 2-day recommendations workshop in May 2013, attended by the OPEN project partners, researchers from the UNCOVER project and selected key stakeholders from across the world. The workshop aimed at developing and refining a set of general and targeted recommendations designed to specifically consider the roles that the respective stakeholder groups should play in reducing the incomplete dissemination of research findings.

    This article first describes the process used to reach a consensus on the recommendations, followed by a discussion of the general recommendations, which are aimed at the major stakeholders in the knowledge translation process. It then presents 47 concrete recommendations complementing the general recommendations aimed at the specific target groups, namely, funding agencies, pharmaceutical and device companies, research institutions, researchers (systematic reviewers and trialists), research ethics committees, trial registries, journal editors and publishers, regulatory agencies, benefit assessment institutions, and legislators.

    Methods

    Based on the data gathered in each OPEN work package (ie, conducted surveys, qualitative studies, and systematic reviews; see online supplementary figure), each of these groups formulated 4–7 recommendations on how to reduce dissemination bias tailored to their respective key group in the knowledge translation process. These recommendations were compiled, ordered, and harmonised to form draft recommendations used as the basis for discussion at the OPEN workshop, including further international key stakeholders in clinical research. The focus of these recommendations was on non-dissemination of full trials rather than the selective reporting of outcomes2 and/or analyses.18 This workshop was held in Freiburg, Germany, on 23–24 May 2013, in order to reach a consensus and to finalise the recommendations. The participants were the members of the OPEN Consortium (see online supplementary appendix 1), members of the OPEN advisory board (see online supplementary appendix 2), and 11 stakeholders from relevant key groups (see online supplementary appendix 3). Workshop participants discussed the draft recommendations in a structured and transparent manner, and decided on consistent definitions to be used in the phrasing of its recommendations (eg, what is meant by ‘clinical trial’; see box 1).

    Box 1

    Definition of a clinical trial

    Clinical trial:

    A clinical trial is any research study that prospectively assigns human participants or groups of humans to one or more health-related interventions to evaluate the effects on health outcomes. Clinical trials may also be referred to as interventional trials. Interventions include but are not restricted to drugs, cells and other biological products, surgical procedures, radiological procedures, devices, behavioural treatments, process-of-care changes, preventive care, etc. This definition includes phase I–IV trials.39

    The recommendations were also discussed in small groups. To structure the discussion and to facilitate the documentation of the judgements underlying the recommendations, a decision table was used for rating each individual recommendation (figure 1). The decision tables required each group to assess five criteria for each recommendation, and to grade it as a ‘strong recommendation’ or only a ‘recommendation’, based on their judgements. The following five explicit criteria, which are conceptually based on the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach for developing recommendations,19 ,20 were assessed for each recommendation:

    • Confidence in effectiveness—whether the group was confident that if this recommendation were implemented it would effectively reduce dissemination bias.

    • Balance of benefits and downsides—whether the group thought that the benefits of this recommendation would clearly outweigh its potential downsides.

    • Likelihood of opposition—whether the group thought that there would be no or only minor opposition to the implementation of this recommendation by relevant key groups.

    • Resource use—whether the group thought that implementing this recommendation would require only limited resources (both direct financial costs and indirect resources).

    • Implementability and feasibility—whether the group thought that this recommendation could be easily implemented (within a reasonable timeframe) and easily sustained.

    For all five criteria, the groups were asked to provide a ‘yes or no’ answer and an explanation for their judgement. Regarding the classification as either ‘strong recommendation’ or ‘recommendation’, we decided not to use a strict rule (eg, ≥3 saying yes results in strong recommendation, <2 saying yes results in recommendation), but rather apply the following rule of thumb: if a recommendation was rated for each or most of the criteria as ‘yes’, the recommendation was rated as a ‘strong recommendation’. If only one or a few judgements strongly favoured a recommendation, it was classed simply as a ‘recommendation’. The rationale for this flexible approach lies in the fact that the weight each of these criteria bears on the final classification can vary depending on the recommendation at hand and/or on the extent to which a certain criterion speaks for or against a recommendation. In summary, a judgement based on integration of transparent assessments for these criteria seemed more appropriate than following a rigid and inflexible rule.20

    Figure 1
    Figure 1

    Example of framework worksheet used for discussion of recommendations.

    The outcomes of all small group deliberations were then discussed in the open plenary sessions until a consensus on all recommendations was reached. The OPEN Consortium and the other meeting participants then endorsed the full set of recommendations.

    Following our workshop, the list of endorsed recommendations was circulated and refined by members of the OPEN Consortium. As a last step, the final recommendations document was sent to all participants of our recommendations workshop for final technical revisions and approval.

    Results

    In this report, we first present four general overarching recommendations—aimed at major stakeholders in the knowledge generation and translation process—followed by the main results from the OPEN project, that is, 47 targeted recommendations addressing 11 key stakeholder groups.

    General recommendations

    Each general recommendation is followed by an explanation of the rationale and reasons why it was rated as ‘strong’ or not based on our decision table (see online supplementary file 1 for detailed decision tables).

    1. All stakeholders should raise awareness about dissemination bias and measures to reduce it (strong recommendation).

    Awareness of the problem of dissemination bias is a prerequisite for change. However, some of the responses to our surveys indicated that many people still deem this problem not relevant.17 It needs to be emphasised among all stakeholder groups that dissemination bias may have serious consequences on the health of people and the trust in research.

    We decided to make this a strong recommendation because we think that the benefits of this recommendation clearly outweigh its potential drawbacks and that only limited resources are necessary for its implementation (as has recently been shown by the AllTrials campaign7). Potential drawbacks of this strong recommendation are that it might not be very effective in the short term and that it might face some opposition. But the Consortium deems the recommendation to be highly effective in the long run, and recognises that any opposition is likely to be due to passivity on the part of the key stakeholder groups and the limited resources available for actively engaging in raising awareness, and emphasising its relevance. Another important aspect that was identified is the need to extend awareness-raising activities to patients and their representatives. All patients participating in clinical trials should be made aware of the fact that their participation only contributes to scientific progress if the results are published, and that this is often not the case.

    • 2. All stakeholders should disseminate and facilitate the implementation of targeted OPEN recommendations as outlined in table 1 (strong recommendation).

    In order for our recommendations to have an impact, they need to be disseminated to as many representatives of the key stakeholder groups as possible. Another prerequisite to change is that the stakeholders make a collective effort. The problem of dissemination bias needs to be addressed through concerted activities that include several or all stakeholders in order to prevent responsibilities from being shifted from one stakeholder to the other (our research showed that many groups believe it to be ‘somebody else's problem’).

    View this table:
    Table 1

    Targeted recommendations by key stakeholder group

    This recommendation was rated a strong recommendation. Its drawbacks are the likelihood of some opposition, and the need for substantial resources for comprehensive dissemination and implementation. While the dissemination of the recommendations alone will not effectively reduce dissemination bias, their full or even partial implementation should help reduce dissemination bias. In order to be fully effective, tailored dissemination and implementation strategies will be necessary.

    • 3. All stakeholders should promote trial registration and posting of results, and support initiatives that facilitate searches across multiple trial registries (strong recommendation).

    Clinical trial registers are important sources for identifying completed and ongoing trials. Efforts to promote prospective registration of trials are, therefore, crucial. The usefulness of registers, however, depends on the completeness and comprehensiveness of the included information.21 Accordingly, adequate quality control measures need to be implemented. It also became evident that metasearching, the possibility of searching across different registries through a single interface, is key to their usefulness. This is highly relevant considering existing language barriers and a lack of awareness of the existence of national registries among users, which might result in relevant registers not being included in their searches. A platform offering a metasearch for trials across multiple registries would help in identifying trials that would otherwise have been missed.

    This recommendation was considered to have long-term, rather than immediate, impact. Together with the favourable balance of benefits and drawbacks, the knowledge of how limited resources could hinder good implementation led us to make this a strong recommendation. As a result of the US FDA Amendments Act, Section 801, passed in 2007, ClinicalTrials.gov has required (since September 2008) the posting of results of aggregate data from applicable trials by researchers within 12 months of completion.22 Once more registries adopt this approach and once platforms allowing metasearching are available, clinical trial registers will become essential tools to reduce dissemination bias.

    • 4. All stakeholders should support activities to systematically, with rigorous methodology, synthesise information from studies (strong recommendation).

    The work of identifying all studies that address a particular question undertaken by authors of systematic reviews helps detect dissemination bias. These efforts include systematic literature searches across electronic databases, the search of trial registries23 and grey literature as well as contacting experts in the field. Also helpful in detecting dissemination bias are, for example, the exploration of selective reporting of outcomes as part of the detailed risk of bias evaluation within systematic reviews,24 and the graphical (eg, funnel plots) and statistical assessment of potentially selective publication of whole studies (eg, Egger's test).25 ,26 Finally, users of systematic reviews will be made aware of the potential impact of dissemination bias on the interpretation of the findings (if these are clearly described by the review authors).

    It was noted during the discussion that the availability of systematic reviews in languages other than English needs to be increased to achieve wider dissemination and use of these crucial resources for evidence-based decision-making.27 Broad acceptance and use of systematic reviews will also help to increase awareness about the problem of dissemination bias, as suggested in recommendation #1.

    Specific recommendations targeted at key stakeholder groups

    The 47 specific recommendations targeted at funding agencies, pharmaceutical and device industry, research institutions, researchers (systematic reviewers and trialists), research ethics committees, trial registries, journal editors and publishers, regulatory agencies, benefit assessment institutions and legislators are presented in table 1.

    While they focus on broader concepts such as transparency (trial registration, access to study protocols and results data), education and training, the tackling of language barriers, and regulatory and legislative activities to counteract selective dissemination, they do provide clear guidance on activities that could and should be undertaken by the respective key stakeholder groups. It is obvious that some of these recommendations are also relevant for other stakeholder groups; however, we decided to list them primarily among those key groups where the largest impact can be expected. More detailed rationales as well as related assessments based on our decision tables for these targeted recommendations will be discussed more extensively in forthcoming articles and will be made available through the OPEN website (http://www.open-project.eu).

    Discussion

    The OPEN project included a broad range of key stakeholders across all relevant areas of the knowledge generation and knowledge translation process. The research that was undertaken within the separate work packages not only shed light on current measures in place to limit dissemination bias, but also gathered the views of key stakeholders on the possible barriers to transparency in medical research. Thus, based on the evidence collated throughout the first phase of OPEN, a draft set of preliminary recommendations for each stakeholder group was developed. These were then discussed with the OPEN Consortium, the advisory board of OPEN and with selected external key stakeholders to capture the different perspectives.

    Instead of solely describing aspects of the general problem of dissemination bias, in addition to the general recommendations, we developed several targeted and concrete recommendations that specifically address each key stakeholder group and thus, as a whole, offer a holistic approach to address the problem. Further, due to the discussions held on topics such as possible opposition, resources required and implementability, we were able to go beyond generic recommendations. The detailed rationales and explanations for these recommendations will be made available soon in separate articles and through the OPEN website (http://www.open-project.eu). By means of this dissemination strategy, we hope to expedite the widespread dissemination and uptake of OPEN recommendations, by the relevant key groups and institutions, and to thus initiate the implementation of more adequate mechanisms to reduce and prevent dissemination bias in the long run.

    Although we developed separate sets of recommendations for industry and public funding agencies, it is important to highlight that we agreed that the same ethical considerations apply to all research funders. After all, the interests of patients and the public in general should always override commercial or other interests.28 During workshop discussions it was noted that the tendency of for-profit companies and also (to some extent) other stakeholder groups such as medical licensing bodies, research institutions or policymakers and legislators, to place the protection of commercial interest over the public interest in access to data from clinical trials, does not seem to be proportional.29 ,30 Despite the fact that companies are investing large sums of money to develop new treatments, the group agreed that intellectual and commercial property rights are overall less important than basic human rights with regard to information and health.

    As already raised, it is obvious that some of our recommendations are relevant for more than one stakeholder group. The recommendations for benefit assessment institutions, for example, are also applicable for regulators; however, they are specifically for benefit assessment agencies since work conducted within OPEN revealed that cooperation between agencies, and the transparency of methodology used, could particularly be improved within this key group. Similarly, recommendations with respect to data sharing, for example, for research ethics committees and legislators, are again also relevant for other key groups. It needs to be kept in mind, though, that data sharing was not the focus of the OPEN project.

    We hope that the results of our project will complement and support ongoing and future activities aiming for more transparency in clinical trial results dissemination. As outlined in the OPEN paper on our conceptual approach to the problem of selective dissemination and resulting dissemination bias (Bassler et al, submitted for publication), the OPEN project is unique in that it aimed to determine which stakeholder groups should be held responsible, and for identifying what they can do to reduce and eliminate this multifaceted and multidimensional problem. In addition, all stakeholders who are willing to address the problem can now be provided with clear and concrete recommendations for action.

    While the OPEN project aims mainly at supporting changes that will result in greater transparency in the knowledge generation and knowledge translation process in the future, it is clear that transparency and free access to data also need to be established for past trials of current treatments. As recently raised by Doshi et al8 as well as by the AllTrials campaign (http://www.alltrials.net), the complete and unbiased reporting of findings from past trials is equally important for securing access to the full evidence. The broader issue of waste in research, first introduced by Chalmers and Glasziou,31 has recently been explored in a series of articles including—among others—waste due to inaccessible research,32 and waste due to incomplete or unusable research reports.33

    The recommendations from the OPEN project coincide with a number of important decisions at the European Medicines Agency (EMA) and the European Parliament, namely, the finalisation and implementation of the EMA's data sharing policy. EMA and the European Parliament, which approved the new European clinical trials regulation on 2 April 2014, have received praise for their leadership on facilitating access to clinical study reports.34 However, the recently released draft terms for use of these reports included several important restrictions, such as strict confidentiality, access to registered users and on-screen viewing only.35 ,36 Certainly, this would impede the widespread and most effective use of these important data sources. Criticism was voiced by academia, to which EMA responded with a press release announcing more user-friendly amendments.37 As of now, it is unclear what the final policy will look like.

    We hope that the findings from the OPEN project will contribute to these decisions and implementation processes, emphasising once more the needs of academia and the public for unrestricted access to an unbiased body of evidence of complete results of all clinical trials.

    Acknowledgments

    The authors would like to thank all participants of the OPEN workshop (see online supplementary appendix 3) for the critical discussion and great contributions to the final OPEN recommendations. Special thanks goes to the OPEN Advisory Board (see online supplementary appendix 2) for their input throughout the project.

    References

    1. ↵
      1. Strech D
      . Normative arguments and new solutions for the unbiased registration and publication of clinical trials. J Clin Epidemiol 2012;65:276–81. doi:10.1016/j.jclinepi.2011.07.002
      OpenUrlCrossRefPubMed
    2. ↵
      1. Dwan K,
      2. Gamble C,
      3. Williamson PR, et al
      . Systematic review of the empirical evidence of study publication bias and outcome reporting bias—an updated review. PLoS ONE 2013;8:e66844. doi:10.1371/journal.pone.0066844
      OpenUrlCrossRefPubMed
    3. ↵
      1. Antes G,
      2. Chalmers I
      . Under-reporting of clinical trials is unethical. Lancet 2003;361:978–9. doi:10.1016/S0140-6736(03)12838-3
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      1. Dickersin K
      . The existence of publication bias and risk factors for its occurrence. JAMA 1990;263:1385–9. doi:10.1001/jama.1990.03440100097014
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. Muller KF,
      2. Briel M,
      3. D'Amario A, et al
      . Defining publication bias: protocol for a systematic review of highly cited articles and proposal for a new framework. Syst Rev 2013;2:34. doi:10.1186/2046-4053-2-34
      OpenUrlPubMed
    6. ↵
      1. Song F,
      2. Parekh S,
      3. Hooper L, et al
      . Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess 2010;14:iii, ix-xi, 1–193. doi:10.3310/hta14080
      OpenUrlPubMedWeb of Science
    7. ↵
      1. Brown T
      . It's time for all trials registered and reported. Cochrane Database Syst Rev 2013;4:ED000057.
      OpenUrlPubMed
    8. ↵
      1. Doshi P,
      2. Dickersin K,
      3. Healy D, et al
      . Restoring invisible and abandoned trials: a call for people to publish the findings. BMJ 2013;346:f2865. doi:10.1136/bmj.f2865
      OpenUrlFREE Full Text
    9. ↵
      1. Groves T
      . What does oseltamivir do, and how will we know? BMJ 2013;347:f4687. doi:10.1136/bmj.f4687
      OpenUrlFREE Full Text
    10. ↵
      1. Nisen P,
      2. Rockhold F
      . Access to patient-level data from GlaxoSmithKline clinical trials. N Engl J Med 2013;369:475–8. doi:10.1056/NEJMsr1302541
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. Littmann J,
      2. Antes G,
      3. Strech D, et al
      . To overcome failure to publish negative findings: The OPEN project. Maturitas 2013;76:111–12.
    12. ↵
      1. Mueller KF,
      2. Meerpohl JJ,
      3. Briel M, et al
      . Detecting, quantifying and adjusting for publication bias in meta-analyses: protocol of a systematic review on methods. Syst Rev 2013;2:60. doi:10.1186/2046-4053-2-60
      OpenUrlPubMed
    13. ↵
      1. Schmucker C,
      2. Bluemle A,
      3. Briel M, et al
      . A protocol for a systematic review on the impact of unpublished studies and studies published in the gray literature in meta-analyses. Syst Rev 2013;2:24. doi:10.1186/2046-4053-2-24
      OpenUrlPubMed
    14. ↵
      1. Portalupi S,
      2. von Elm E,
      3. Schmucker C, et al
      . Protocol for a systematic review on the extent of non-publication of research studies and associated study characteristics. Syst Rev 2013;2:2. doi:10.1186/2046-4053-2-2
      OpenUrlPubMed
    15. ↵
      1. Malički M,
      2. Marušić A
      . Is there a solution to publication bias? Researchers call for changes in dissemination of clinical research results. J Clin Epidemiol 2014;67:1103–10. doi:10.1016/j.jclinepi.2014.06.002
      OpenUrlPubMed
    16. ↵
      1. Viergever RF,
      2. Karam G,
      3. Reis A, et al
      . The quality of registration of clinical trials: still a problem. PLoS ONE 2014;9:e84727. doi:10.1371/journal.pone.0084727
      OpenUrlPubMed
    17. ↵
      1. Wager E,
      2. Williams P
      ; Project Overcome failure to Publish nEgative fiNdings C. “Hardly worth the effort”? Medical journals’ policies and their editors’ and publishers’ views on trial registration and publication bias: quantitative and qualitative study. BMJ 2013;347:f5248. doi:10.1136/bmj.f5248
      OpenUrlAbstract/FREE Full Text
    18. ↵
      1. Dwan K,
      2. Altman DG,
      3. Clarke M, et al
      . Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials. PLoS Med 2014;11:e1001666. doi:10.1371/journal.pmed.1001666
      OpenUrlCrossRefPubMed
    19. ↵
      1. Andrews J,
      2. Guyatt G,
      3. Oxman AD, et al
      . GRADE guidelines: 14. Going from evidence to recommendations: the significance and presentation of recommendations. J Clin Epidemiol 2013;66:719–25. doi:10.1016/j.jclinepi.2012.03.013
      OpenUrlCrossRefPubMed
    20. ↵
      1. Andrews JC,
      2. Schunemann HJ,
      3. Oxman AD, et al
      . GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation's direction and strength. J Clin Epidemiol 2013;66:726–35. doi:10.1016/j.jclinepi.2013.02.003
      OpenUrlCrossRefPubMed
    21. ↵
      1. Zarin DA,
      2. Tse T,
      3. Williams RJ, et al
      . The ClinicalTrials.gov results database––update and key issues. N Engl J Med 2011;364:852–60. doi:10.1056/NEJMsa1012065
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. Hirsch L
      . Trial registration and results disclosure: impact of US legislation on sponsors, investigators, and medical journal editors. Curr Med Res Opin 2008;24:1683–9. doi:10.1185/03007990802114849
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      1. Jones C,
      2. Keil L,
      3. Weaver M, et al
      . Clinical trials registries are under-utilized in the conduct of systematic reviews: a cross-sectional analysis. Syst Rev 2014;3:126. doi:10.1186/2046-4053-3-126
      OpenUrlPubMed
    24. ↵
      1. Higgins JPT,
      2. Altman DG,
      3. Sterne JAC
      . Chapter 8: assessing risk of bias in included studies. In: Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions version 510. (updated March 2011), Wiley-Blackwell, 2011:187–241.
    25. ↵
      1. Higgins JP,
      2. Green S
      . Cochrane handbook for systematic reviews of interventions; version 5.1.0. Chichester, UK: John Wiley & Sons, Ltd, 2011. http://www.cochrane-handbook.org
    26. ↵
      1. Sterne JA,
      2. Sutton AJ,
      3. Ioannidis JP, et al
      . Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ 2011;343:d4002. doi:10.1136/bmj.d4002
      OpenUrlFREE Full Text
    27. ↵
      1. Ofori-Adjei D,
      2. Antes G,
      3. Tharyan P, et al
      . Have online international medical journals made local journals obsolete? PLoS Med 2006;3:e359–0. doi:10.1371/journal.pmed.0030359
      OpenUrlCrossRefPubMed
    28. ↵
      1. Gotzsche PC
      . Why we need easy access to all data from all clinical trials and how to accomplish it. Trials 2011;12:249. doi:10.1186/1745-6215-12-249
      OpenUrlCrossRefPubMed
    29. ↵
      1. Strech D,
      2. Littmann J
      . Lack of proportionality. Seven specifications of public interest that override post-approval commercial interests on limited access to clinical data. Trials 2012;13:100. doi:10.1186/1745-6215-13-100
      OpenUrlPubMed
    30. ↵
      1. Gotzsche PC,
      2. Jorgensen AW
      . Opening up data at the European Medicines Agency. BMJ 2011;342:d2686. doi:10.1136/bmj.d2686
      OpenUrlFREE Full Text
    31. ↵
      1. Chalmers I,
      2. Glasziou P
      . Avoidable waste in the production and reporting of research evidence. Lancet 2009;374:86–9. doi:10.1016/S0140-6736(09)60329-9
      OpenUrlCrossRefPubMedWeb of Science
    32. ↵
      1. Chan AW,
      2. Song F,
      3. Vickers A, et al
      . Increasing value and reducing waste: addressing inaccessible research. Lancet 2014;383:257–66. doi:10.1016/S0140-6736(13)62296-5
      OpenUrlCrossRefPubMedWeb of Science
    33. ↵
      1. Glasziou P,
      2. Altman DG,
      3. Bossuyt P, et al
      . Reducing waste from incomplete or unusable reports of biomedical research. Lancet 2014;383:267–76. doi:10.1016/S0140-6736(13)62228-X
      OpenUrlCrossRefPubMedWeb of Science
    34. ↵
      Regulation of the European Parliament and of the Council on clinical trials on medicinal products for human use, and repealing Directive 2001/20/EC. 2014. http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32014R0536&from=EN (accessed 5 Jun 2014).
    35. ↵
      1. Torjesen I
      . European drug agency backtracks on plan to give researchers access to clinical trial reports. BMJ 2014;348:g3432. doi:10.1136/bmj.g3432
      OpenUrlFREE Full Text
    36. ↵
      1. Goldacre B,
      2. Godlee F,
      3. Heneghan C, et al
      . Open letter: European Medicines Agency should remove barriers to access clinical trial data. BMJ 2014;348:g3768. doi:10.1136/bmj.g3768
      OpenUrlFREE Full Text
    37. ↵
      EMA. European Medicines Agency agrees policy on publication of clinical trial data with more user-friendly amendments. 2014. http://www.ema.europa.eu/docs/en_GB/document_library/Press_release/2014/06/WC500168342.pdf (accessed 4 Aug 2014).
      1. Eden J,
      2. Levit L,
      3. Berg AO, et al
      . Finding what works in health care: standards for systematic reviews. National Academies Press, 2011.
    39. ↵
      World Health Organization. International Clinical Trials Registry Platform (ICTRP). Secondary International Clinical Trials Registry Platform (ICTRP). http://www.who.int/ictrp/en/

    Supplementary materials

    Footnotes

    • Twitter Follow Joerg Meerpohl at @meerpohl and Elizabeth Wager at @SideviewLiz

    • Contributors JJM and LKS drafted this manuscript based on the discussions and results of the OPEN recommendations workshop. All named authors were involved in the work packages that led to the development of the draft recommendations. All actively participated in the OPEN workshop, and repeatedly commented and revised the draft manuscript; they are listed in alphabetical order. All authors approved the final manuscript. People listed as part of the OPEN project consortium contributed to the various work packages of the OPEN project, participated in the workshop on the basis of which the draft recommendations were developed. All members of the OPEN consortium read, commented and approved the final manuscript.

    • Funding The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement number 285453.

    • Competing interests All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/coi_disclosure.pdf and declare: the OPEN project has received funding from the European Union under grant agreement number 285453; Davina Ghersi reports being the Chair of the WHO Advisory Panel on the International Clinical Trials Registry Platform; JK reports grants from Amgen, GSK, Sanofi, Biomarin, MSD, Roche and Bayer outside the submitted work; JJM reports being a member of the WHO Advisory Panel on the International Clinical Trials Registry Platform; AR is a staff member of the WHO - the author alone is responsible for the views expressed in this article and they do not necessarily represent the decisions, policy or views of the WHO; EW reports training and consultancy activities for various companies and publishers, and training activities for various universities; Robert Wolff reports grants from Amgen, GSK, Sanofi, Biomarin, MSD, Roche and Bayer outside the submitted work.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement No additional data are available.