You are here

Article Text

Article menu

Download PDFPDF

XML
Medical publishing and peer review
Research
Time to publication for publicly funded clinical trials in Australia: an observational study
  1. Linn Beate Strand1,2,
  2. Philip Clarke3,
  3. Nicholas Graves1,
  4. Adrian G Barnett1
  1. 1Institute of Health and Biomedical Innovation, School of Public Health and Social Work, Queensland University of Technology, Brisbane, Queensland, Australia
  2. 2Department of Public Health and General Practice, Norwegian University of Science and Technology, Trondheim, Norway
  3. 3Centre for Health Policy, School of Population and Global Health, The University of Melbourne, Melbourne, Queensland, Australia
  1. Correspondence to Dr Linn Beate Strand; linn.b.strand{at}ntnu.no

Abstract

Objective To examine the length of time between receiving funding and publishing the protocol and main paper for randomised controlled trials.

Design An observational study using survival analysis.

Setting Publicly funded health and medical research in Australia.

Participants Randomised controlled trials funded by the National Health and Medical Research Council of Australia between 2008 and 2010.

Main outcome measures Time from funding to the protocol paper and main results paper. Multiple variable survival models examining whether study characteristics predicted publication times.

Results We found 77 studies with a total funding of $A59 million. The median time to publication of the protocol paper was 6.4 years after funding (95% CI 4.1 to 8.1). The proportion with a published protocol paper 8 years after funding was 0.61 (95% CI 0.48 to 0.74). The median time to publication of the main results paper was 7.1 years after funding (95% CI 6.3 to 7.6). The proportion with a published main results paper 8 years after funding was 0.72 (95% CI 0.56 to 0.87). The HRs for how study characteristics might influence timing were generally close to one with narrow CIs, the notable exception was that a longer study length lengthened the time to the main paper (HR=0.62 per extra study year, 95% CI 0.43 to 0.89).

Conclusions Despite the widespread registration of clinical trials, there remain serious concerns of trial results not being published or being published with a long delay. We have found that these same concerns apply to protocol papers, which should be publishable soon after funding. Funding agencies could set a target of publishing the protocol paper within 18 months of funding.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

https://doi.org/10.1136/bmjopen-2016-012212

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • We had a relatively small sample size of 77 studies.

    • The protocol paper was examined as a time-dependent intermediate event when examining the time taken to publish the main paper.

    • We did not contact authors to confirm publication dates.

    Introduction

    Randomised controlled trials (RCTs) represent the gold standard when evaluating healthcare interventions, mainly because they minimise selection bias.1 Policymakers often depend on the published results from RCTs and systematic reviews based on RCTs to learn about the efficacy of interventions and to apply the findings in health policy.2 Previous research has found 30–50% of clinical trial results do not get published, even years after completion of the project.3–7 This missing research is a large part of the overall ‘waste in research’, with the estimate that 85% of current research investment is wasted.8

    National Health and Medical Research Council (NHMRC) Project Grants are the biggest source of funding in Australia for clinical trials. Project Grants aim to support individual researchers as individuals or small teams for a defined project from 1 to 5 years. Applications are extensive, including sections on past funding, budget and progress reports, but the key sections are the researchers' track records and the nine-page detailed background and research plan. Applications for funding large clinical trials are generally peer reviewed by a separate panel of experts who only consider clinical trials.

    The NHMRC have supported policies to improve clinical trial reporting. In 2005, an NHMRC grant helped establish the Australian and New Zealand Clinical Trial Registry (ANZCTR). In 2007, the NHMRC published a code for the responsible conduct of research, which stated that researchers must register clinical trials and have a responsibility to their colleagues and the wider community to disseminate a full account of their research as broadly as possible.9 In 2014, the NHMRC supported the Alltrials campaign, which calls for all past and present clinical trials to be registered and their results reported. To date, there is no NHMRC policy that mentions publishing research protocols.

    The aim of this paper was to examine the time delay between funding and publication for government-funded RCTs in Australia. As well as the time to the publication of the main paper, we also examined the time to a protocol paper. The main purpose of the protocol is to predetermine the methods and primary outcomes, so that issues such as outcome switching are reduced or at least made explicit.10 Publishing a protocol paper also allows others to see what research is ongoing and so helps avoid duplicated research being funded. Ideally, protocols would be published quickly, preferably soon after the research is funded. We were also interested in whether publication of a protocol paper was associated with time to publication of the main paper and whether there was an association between the study characteristics (ie, funding amount, number of investigators and number of participants) and time to publication.

    Methods

    We examined studies funded by the NHMRC of Australia. We combined data on the funding amount, estimated number of research participants, number of investigators and publication times in peer-reviewed journals of the protocol paper and main results paper. We examined the time to publication of the protocol paper and the main paper, and whether characteristics of the study predicted these times.

    Data collection

    The NHMRC is Australia's main funding body for medical research. The NHMRC administers funding for health and medical research on behalf of the Australian Government, and provides funding for all areas of research relevant to human health and medicine. This study examined Project Grants which are the NHMRC's main scheme for individuals and teams of researchers undertaking RCTs in any field relevant to human health. Applicants can apply for grants of up to 5 years in duration.11

    Every RCT should be registered. The three most relevant trial registries for this study are the: ANZCTR, International Standard Randomised Controlled Trial Number (ISRCTN) and ClinicalTrials.gov. They represent a comprehensive source for information about ongoing and completed publicly and privately funded trials within and outside Australia. All three registries use web-based systems to facilitate registration of clinical trials.12–14 The registries include compulsory and optional data items, and trials will not be registered without completion of all compulsory data items and approval by a human participant ethics review board. Additional information about the registries is available from their websites.12–14

    Search strategy

    We aimed to find all RCTs funded by the NHMRC between 2008 and 2010. We used these years to strike a balance between examining recent data and allowing a reasonable time for trial completion and publication. We identified 77 RCTs by searching for the words ‘RCT’, ‘randomised’ and ‘trial’ in NHMRC documents available on the NHMRC website (http://www.nhmrc.gov.au/grants-funding/outcomes-funding-rounds). From the same data we also extracted the funding amount ($A), number of investigators and planned study duration. To find the number of participants and trial registration number, we first searched for the project name and principal investigator within ANZCTR. When a study could not be located in ANZCTR, we searched for it within ISRCTN and ClinicalTrials.gov. Some studies may have been registered in multiple registries, but we did not record this. The data were entered into an Excel spreadsheet.

    Two investigators (LBS and AGB) independently determined the publication status of each trial in January 2016. To locate a protocol paper or main results paper, we searched for the unique NHMRC project number and the unique trial registration number in PubMed and in Google Scholar. If we were unable to locate the papers by this method, we searched for the name of the principal investigator and the name of the project. We matched the articles that were identified to the corresponding trial using information about location, enrolment, study start and completion dates, and outcome measures described in the clinical trial registers. We also searched all available NHMRC grant summaries from 2004 to 2012 (the last year available from the NHMRC). If we found a protocol paper, we searched all the papers that cited the protocol paper to look for the main paper. Similarly, if we found the main paper, we searched the text for mention of the protocol paper. We only included protocols published in peer-reviewed journals. The data collection process is shown in figure 1.

    Figure 1
    Figure 1

    Flow chart showing the data collection process. ANZCTR, Australian and New Zealand Clinical Trial Registry NHMRC, National Health and Medical Research Council; RCT, randomised controlled trial.

    Publication status and time to publication

    For all published papers, we calculated time to publication from the date of the funding announcement. For papers published online ahead of print, we used the date the publication was available electronically. For trials where we could not find a protocol paper or a main results paper, we used a censored follow-up time as the last day we performed our search (31 January 2016).

    We were interested in whether some trials were more likely to be completed and wanted to examine whether time to publication of the protocol paper and main results paper was associated with funding amount, number of investigators or participant numbers. We were also interested in whether publishing a protocol paper influenced the time it took for the main results paper to be published.

    Statistical analysis

    Kaplan-Meier survival plots were used to show the time to publication of protocol paper and main results paper. The y-axes in the plots were reversed to show the increasing proportion of published papers over time. We calculated the median time to publication and 95% CIs, and the mean survival and 95% CI at the maximum follow-up time of 8 years.

    We used Cox proportional hazard models with time since funding date as the time scale, and presented results as HRs with 95% CIs. The predictor variables for the publication of a protocol paper were funding amount (per $A100K), number of investigators, project duration and participant numbers (per 100). These variables were chosen as indicators of complexity that may make a trial more difficult to complete.

    When examining the time to publication of the main results paper, we added the protocol paper as a time-dependent intermediate variable (figure 2).15 The alternative approach of using a binary variable for protocol published (yes/no) ignores time and assumes the protocol was published on the funding date. This lengthens the time between the protocol and main paper and would bias the HR downwards.16 All analyses were performed using R V.3.1.0.

    Figure 2
    Figure 2

    The three key events over time. Studies may go straight from funding to main paper publication or first publish a protocol paper, which is therefore a time-dependent variable. The illustration shows how ignoring the timing of the protocol lengthens the time between the protocol paper and the main paper which would bias the HR downwards. The thicker part of the arrow from protocol to main paper in the right panel is the arrow length in the left panel.

    Missing data

    We were unable to find four studies in any of the trial registers, and so we did not have information on the number of participants. We therefore randomly imputed these missing numbers from the available participant numbers. We created 10 imputed data sets and combined the results using the ‘mitools’ package in R.17

    Results

    We found 77 studies with a total funding of ∼$A59 million (see online supplementary appendix for list of studies). Forty-three studies published a protocol paper and 39 studies published a main results paper (figure 3). Thirty million dollars and 50% of the funding went to studies that have not yet published the main results paper during the mean follow-up time of 6.2 years.

    Figure 3
    Figure 3

    Flow chart showing publication status (protocol and main paper) of the 77 funded studies.

    Characteristics of the RCTs by publication status are in table 1.

    View this table:
    Table 1

    Characteristics of the randomised controlled trials funded by the National Health and Medical Research Council (NHMRC) Project Grants 2008–2010 by publication status of main results paper

    Time to publication

    From the funding date, the median time to publication of the protocol paper was 6.4 years (figure 4), with a 95% CI from 4.1 to 8.1 years. The estimated proportion with a published protocol paper 8 years after funding was 0.61 (95% CI 0.48 to 0.74). One protocol was published before funding.

    Figure 4
    Figure 4

    Kaplan-Meier survival curve (and 95% CIs as dashed lines) for time from funding to protocol paper published (n=77). Horizontal lines on the survival curve indicate censoring.

    The median time to publication of the main results paper was 7.1 years (figure 5), with a 95% CI 6.3 to 7.6. The estimated proportion with a published main results paper 8 years after funding is 0.72 (95% CI 0.56 to 0.87). As expected, shorter studies were generally published faster.

    Figure 5
    Figure 5

    Kaplan-Meier survival curves (and 95% CIs as dashed lines) for time from funding to main paper published for all papers combined (left) and split into relatively long and short studies by median study length (right; n=77). Horizontal lines on the survival curves indicate censoring.

    Multivariable analysis

    We found no evidence for an association between funding amount, number of investigators or participant numbers with time to publication of protocol paper or main results (table 2). Publication of a protocol paper was not associated with time to publication of the main results paper. Project duration was not associated with time to protocol paper, but not surprisingly longer projects took longer to publish their main results paper, with a HR of 0.62 per extra year (95% CI 0.43 to 0.89).

    View this table:
    Table 2

    HRs and 95% CIs for time to publication of protocol paper and main results paper dependent on study characteristics

    Discussion

    The median time to publishing a protocol paper from NHMRC-funded research was 6.4 years after funding. The median time to publication of the main results paper was 7.2 years, well beyond the duration of funding for NHMRC grant schemes. It is hard to understand the reasons for these delays, because the study design should largely be finalised at the time the researchers receive funding, and a key criteria the NHMRC uses for funding research is the feasibility of the project and the investigators' track record.18 Some delays may be due to protracted peer review, which is no fault of the investigators.19 Other delays outside the investigators' control are sponsor termination and contractual issues.20

    Publishing a protocol

    Publishing a protocol early increases transparency and gives the research community and the public a better idea of ongoing studies. Protocols are often included in systematic reviews or assist in development of prospective meta-analyses, and can be particularly important for research questions with little current evidence. While some journals (eg, BMJ Open) will not accept protocol papers if the data collection has concluded, there are few guidelines on the timing of the publication of protocol paper in relation to funding or initiation of the study. While there is an established guideline for the reporting of a protocol,21 these only cover the content and not the timing of protocol papers. The guidelines could be extended to include a recommended target date for publication after funding, or after ethics approval for unfunded studies. A period of 18 months after funding seems a sensible target. This time allows researchers to transfer the information from the grant application into the protocol format and allows time for peer review. Delays due to peer review could be avoided by adding the protocol to a clinical trial registry. Funding agencies such as the NHMRC could mandate a target and withhold further funding if the protocol is not published or at least submitted to a journal to provide a clear incentive to publish the protocol. There is a precedent for this, as the UK National Institute for Health Research include in their funding contract that a publicly available final report must be submitted, and this agency has a 98% publication rate.22

    Current editorial processes at many journals mean it is now much harder to get an RCT published without a protocol, and so the problem we identified may be a historic concern. However, the median time to protocol paper was 7.3 years for studies funded in 2008, 3.9 years for studies funded in 2009 and 6.0 years for studies funded in 2010. So our data do not show clear signs of improvement between 2008 and 2010.

    Predictors of publication timing

    Our sample size was relatively small, but the HRs for how study characteristics might influence timing were generally close to one with narrow CIs (table 2), which suggests that the issues with timely publication are common to large and small studies. The two exceptions were study length and protocol paper. The mean HR for the protocol paper is <1, suggesting that publishing a protocol slows the time to the main paper. This could be because not having a protocol paper allows authors to focus on statistically significant results that may not have been the primary outcome, and there is a bias towards journals accepting papers with statistically significant results,23 and it generally takes longer to publish negative results.24 However, the CIs around the HR for the protocol were wide, and a larger study is needed to examine this potential association.

    Previous studies

    Our results are similar to related studies. A large study of US RCTs found that only 36% had published their results within 2 years of study completion7 and another US study of clinical trials found that only 46% were published within 30 months of trial completion.4 These results and ours highlight the serious problem of missing research and the timeliness of completing studies. Registering trials has been an important step in avoiding the waste of missing research, but it has not solved the problem as there are many registered randomised trials that remain unpublished.

    Ours is the first study to use Australian data and the first to additionally examine the protocol paper.

    Limitations

    There are a number of limitations. Our sample size was relatively small compared with related studies. We did not email authors to confirm publication status, but a related study found that emailing authors did not provide much additional information.4 Our search to identify RCTs would have missed studies that did not include the three phrases we searched for in their title.

    Conclusions

    In a time when there is very high competition for scarce research funds, we found that both the protocol paper and the results of clinical trials are not being published in a timely fashion. Funding bodies need to focus more on the outputs from research and timely publication of a protocol paper should be the first milestone to assess whether trials will produce results which can be added to the body of evidence used for medical decision-making.

    References

    1. ↵
      1. Odgaard-Jensen J,
      2. Vist GE,
      3. Timmer A, et al
      . Randomisation to protect against selection bias in healthcare trials. Cochrane Database Syst Rev 2011;(4):MR000012. doi:10.1002/14651858.MR000012.pub3
    2. ↵
      1. Whitty CMJ
      . What makes an academic paper useful for health policy? BMC Med 2015;13:1–5. doi:10.1186/s12916-015-0544-8
      OpenUrlCrossRefPubMed
    3. ↵
      1. Ross JS,
      2. Mulvey GK,
      3. Hines EM, et al
      . Trial publication after registration in ClinicalTrials.Gov: a cross-sectional analysis. PLoS Med 2009;6:e1000144. doi:10.1371/journal.pmed.1000144
      OpenUrlCrossRefPubMed
    4. ↵
      1. Ross JS,
      2. Tse T,
      3. Zarin DA, et al
      . Publication of NIH funded trials registered in ClinicalTrials.gov: cross sectional analysis. BMJ 2012;344:d7292. doi:10.1136/bmj.d7292
      OpenUrlCrossRefPubMed
    5. ↵
      1. Chinnery F,
      2. Young A,
      3. Goodman J, et al
      . Time to publication for NIHR HTA programme-funded research: a cohort study. BMJ Open 2013;3:e004121. doi:10.1136/bmjopen-2013-004121
      OpenUrlAbstract/FREE Full Text
    6. ↵
      1. Doshi P
      . Data too important to share: do those who control the data control the message? BMJ 2016;352:i1027. doi:10.1136/bmj.i1027
      OpenUrlFREE Full Text
    7. ↵
      1. Chen R,
      2. Desai NR,
      3. Ross JS, et al
      . Publication and reporting of clinical trial results: cross sectional analysis across academic medical centers. BMJ 2016;352:i637. doi:10.1136/bmj.i637
      OpenUrlAbstract/FREE Full Text
    8. ↵
      1. Chalmers I,
      2. Glasziou P
      . Avoidable waste in the production and reporting of research evidence. Lancet 2009;374:86–9. doi:10.1016/S0140-6736(09)60329-9
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      National Health and Medical Research Council, Australian Research Council, Universities Australia. Australian Code for the Responsible Conduct of Research. 2007. https://www.nhmrc.gov.au/guidelines-publications/r39
    10. ↵
      1. Goldacre B
      . How to get all trials reported: audit, better data, and individual accountability. PLoS Med 2015;12:e1001821. doi:10.1371/journal.pmed.1001821
      OpenUrlCrossRefPubMed
    11. ↵
      National Health and Medical Research Council. Project grants. 2015. https://www.nhmrc.gov.au/grants-funding/apply-funding/project-grants
    12. ↵
      Australia and New Zealand Clinical Trials Registry. About us. 2016. http://www.anzctr.org.au/Support/AboutUs.aspx
    13. ↵
      ClinicalTrials.gov. ClinicalTrials.gov: A service of the U.S. National Institutes for Health. 2016. https://clinicaltrials.gov
    14. ↵
      ISRCTN. ISRCTN registry. 2016. http://www.controlled-trials.com/page/about
    15. ↵
      1. Beyersmann J,
      2. Allignol A,
      3. Schumacher M
      . Competing risks and multistate models with R. New York: Springer, 2012.
    16. ↵
      1. Beyersmann J,
      2. Gastmeier P,
      3. Wolkewitz M, et al
      . An easy mathematical proof showed that time-dependent bias inevitably leads to biased effect estimation. J Clin Epidemiol 2008;61: 1216–21. doi:10.1016/j.jclinepi.2008.02.008
      OpenUrlCrossRefPubMedWeb of Science
    17. ↵
      1. Lumley T
      . mitools: Tools for multiple imputation of missing data. R package version 2.0.1. 2010. https://cran.r-project.org/web/packages/mitools/index.html
    18. ↵
      National Health and Medical Research Council. NHMRC Project Grants peer review guidelines for funding commencing in 2014. 2013. http://www.nhmrc.gov.au/grants-funding/apply-funding/project-grants
    19. ↵
      1. Mooney LA,
      2. Fay L
      . Cross-sectional study of Pfizer-sponsored clinical trials: assessment of time to publication and publication history. BMJ Open 2016;6:e012362. doi:10.1136/bmjopen-2016-012362
      OpenUrlAbstract/FREE Full Text
    20. ↵
      1. Tice DG,
      2. Carroll KA,
      3. Bhatt KH, et al
      . Characteristics and causes for non-accrued clinical research (NACR) at an academic medical institution. J Clin Med Res 2013;5:185–93. doi:10.4021/jocmr1320w
      OpenUrl
    21. ↵
      1. Chan AW,
      2. Tetzlaff JM,
      3. Altman DG, et al
      . SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med 2013;158:200–7. doi:10.7326/0003-4819-158-3-201302050-00583
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. Turner S,
      2. Wright D,
      3. Maeso R, et al
      . Publication rate for funded studies from a major UK health research funder: a cohort study. BMJ Open 2013;3:pii: e002521. doi:10.1136/bmjopen-2012-002521
      OpenUrl
    23. ↵
      1. Emerson GB,
      2. Warme WJ,
      3. Wolf FM, et al
      . Testing for the presence of positive-outcome bias in peer review: a randomized controlled trial. Arch Intern Med 2010;170:1934–9. doi:10.1001/archinternmed.2010.406
      OpenUrlCrossRefPubMedWeb of Science
    24. ↵
      1. Ioannidis JA
      . Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 1998;279:281–6.
      OpenUrlCrossRefPubMedWeb of Science

    Footnotes

    • Twitter Follow Adrian Barnett @aidybarnett

    • Contributors PC, NG and AGB came up with the idea for the study while all authors contributed to the overall conception and design. AGB and LBS independently determined the publication status of each trial and carried out the statistical analysis. LBS and PC wrote the first draft and all authors contributed to the interpretation of results and helped with the manuscript writing. All authors had full access to all of the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis.

    • Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement The data are available on figshare: https://figshare.com/articles/Time_to_publication_data/4054878.