Article Text
PDF
Abstract
Objectives To determine the relationship between manufacturer-related financial ties among investigators of published drug trials and rates of discrepant registered and published primary trial outcomes.
Design Cross-sectional study.
Setting Human subjects drug trials published in ‘core clinical’ MEDLINE journals in 2013.
Primary and secondary outcome measures The primary study endpoint was the presence of a prospectively registered, clearly defined primary outcome that matched the published primary outcome for each included trial. Secondary outcomes included assessments of registration timing and quality, and the impact of outcome discrepancies between registration and publication on the statistical significance of the included trials.
Results Of 192 included trials, 134 (70%) were positive and 58 (30%) were negative. Financial ties were present between first or last authors and drug manufacturers for 130 trials (68%), of which 78% were positive, versus 53% of trials with no financial ties that were positive. Clearly defined, prospectively registered outcomes that matched the published outcomes were present in just 76 of the 192 trials (40%). After adjusting for study start date and sample size, the observed relationship between investigator financial ties and the presence of a match between prospectively registered and published primary outcomes was of borderline statistical significance (OR 2.12, 95% CI 0.998 to 4.50). Studies with financial ties present were more likely than studies without ties to have been prospectively registered (78%vs48%, P<0.001) and were more likely to have prospectively registered a clearly defined primary outcome(62%vs35%, P<0.001).
Conclusions Less than half of the trials in this cohort were prospectively registered with a clear primary outcome that was consistent with the primary outcome reported in the published manuscript. The presence of investigator financial ties was associated with higher quality registration practices, though this association diminished after adjusting for factors that impact registration quality.
- conflict of interest
- clinicaltrials.gov
- trial registration
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
The relationship between investigator–manufacturer financial ties and registration quality has not been previously assessed.
Multiple reviewers independently assessed study endpoints.
Trial registration patterns have evolved over time, and these results may not reflect current registration and publication practices.
Introduction
Randomised controlled trials (RCTs) are a critical means of assessing the efficacy of medical interventions. A core principal in the ethical conduct of RCTs is the presence of clinical equipoise, which mandates the existence of genuine uncertainty within the medical community with respect to the best choice between the various treatment options that a trial is investigating.1 A recent cross-sectional study investigating the relationship between the financial ties of principal investigators and RCT outcomes found that trials with financial ties between principal investigators and drug manufacturers were substantially more likely to report positive results than trials without financial ties.2
There are several plausible explanations for the higher proportion of positive results among trials in which the investigators have financial ties to drug manufacturers. First, these investigators may make design decisions such as selection of a comparator or enrolment of a specific population that favour the intervention.3–5 Second, investigators with financial ties may be less likely to publish the results of unfavourable trials, leading to publication bias.6 7 Third, investigators with financial ties may be more likely than others to change primary study outcomes after trial completion in order to highlight those outcomes that are statistically significant or otherwise favourable.8 In evaluating this third possibility, clinical trial registries may allow for the assessment of outcome switching by facilitating comparisons between prospectively registered primary outcomes and published primary outcomes.9 10
The goal of this investigation was to determine the relationship between the presence of manufacturer-related financial ties among principal investigators of published drug trials and rates of discrepant registered and published primary trial outcomes.
Methods
Sample identification
We analysed a group of clinical trials that had been identified for a recent cross-sectional study assessing the association between the financial ties of principal study investigators and study outcomes. Trials were randomly selected for inclusion in the original study from among English-language, human subjects drug trials that had been published between 1 January 2013 and 31 December 2013 in ‘core clinical’ MEDLINE journals.2 The original study included manuscripts describing 195 trials. We excluded three of these manuscripts because they reported the results of secondary or follow-up analyses, leaving 192 included trials.
Investigator financial ties
We defined principal investigators as the first and last author, along with any other authors specified as sharing first or last author responsibilities for each included manuscript. A comprehensive search was performed to identify financial ties between these investigators and manufacturers of the study drug. A financial tie was defined as direct compensation to the investigator in the form of advisor/consultancy payments, honorariums, speaker’s fees, stock ownership, travel/meal reimbursement, a patent for which the investigator was the inventor or an employer–employee relationship. The search for financial ties has been previously described in detail and included a search of the trial publication, MEDLINE (to allow review of other publications by the investigator), Google, ProPublica’s Dollars for Doctors database and the US Patent Office.2 Financial ties were only included if they occurred within 2 years before article publication. All financial ties were independently verified by at least two reviewers, and discrepancies were resolved by consensus. If any of the principal investigators of a manuscript was found to have a financial tie to a manufacturer of the study drug, the manuscript was considered to have a financial tie.
Registry search
For each trial, one of two reviewers initially inspected the published manuscript for evidence of trial registration. If no registration information was provided, the reviewers searched ClinicalTrials.gov by keyword to identify registry entries corresponding to the included trials. When this initial search failed to identify a trial registration entry matching the published trial report, a third reviewer (CJ) with expertise in trial registration searched ClinicalTrials.gov, the International Standard Registered Clinical/soCial sTudy Number database, the WHO International Clinical Trials Registry Platform search portal and any national registries corresponding to the principal investigators’ countries of origin (eg, Australian New Zealand Clinical Trials Registry and European Union Clinical Trials Register). The final registry search occurred in February 2017. Searches were performed by keyword, title, principal investigator, and funding source. In addition to these characteristics, potential matches between published trials and registry entries were assessed based on study interventions, planned sample sizes, enrolment dates and trial locations. We considered trials unregistered if neither independent search identified a corresponding registry entry.
Assessment of outcomes from the clinical trials
Standardised data collection forms were used to record information from published manuscripts and registry entries corresponding to each included trial. We recorded the primary and secondary outcome(s) reported within each published manuscript. If no primary outcome was explicitly defined within the manuscript or abstract, we considered the outcome used for the power calculation to be the primary published outcome. If no primary outcome was defined and there was no power calculation, we considered the published primary outcome to be undefined.
Trial outcomes were classified as positive if the study hypothesis was supported for the primary efficacy outcome and negative if it was not. Superiority trials were considered positive if the drug in the intervention arm was statistically superior to the control (P<0.05) and non-inferiority trials were considered positive if the intervention arm was not significantly worse than the control. Trials with multiple published primary outcomes were positive if at least one efficacy outcome was either positive (superiority studies) or not significantly different from the control (non-inferiority studies).
For each registered trial, we recorded the date of initial trial registration, the date on which a primary outcome was first registered, the registered primary outcome(s) and registered secondary outcomes. If changes to the registered primary outcome were made, we recorded the primary outcome that was listed at the time trial enrolment began. Two reviewers (CJ and BSM) then independently determined whether each registered outcome was clearly defined. In order to be considered a clearly defined primary outcome, the registered outcome needed to provide sufficient detail to allow a reader to reasonably design a study measuring the same outcome measure. In most cases, in order to meet this standard, the registered outcome needed to describe a specifically defined variable of interest and a specific time-point for assessment. Discrepancies were resolved by consensus.
We considered trials that had a clearly defined primary outcome registered within 90 days of the start of enrolment to be prospectively registered; trials registered more than 90 days after enrolment began were classified as being retrospectively registered.
Primary outcome
For those trials with prospectively registered, clearly defined primary outcomes, we compared the registered and published primary outcomes to assess for primary outcome switching. Two reviewers (CJ and BSM) who were blinded to the presence or absence of principal investigator financial ties independently assessed outcome consistency. Discrepancies were resolved by consensus following a review of the full manuscript and the full registry entry. Registered and published outcomes were considered consistent with one another if every registered primary outcome for a specific study was reported as a primary outcome in the published manuscript and if every primary outcome described in the manuscript was registered clearly and prospectively. We then categorised primary outcome discrepancies according to the classification used by Mathieu et al.11
The primary outcome for our study was the presence of a prospectively registered, clearly defined primary outcome that was consistent with the published outcome. We report the proportion of all included trials meeting this primary outcome measure, and also compare the primary outcome results among trials having investigator financial ties to those trials without investigator financial ties.
Secondary outcomes
Secondary outcomes for the present study were study registration, prospective study registration and prospective registration with a clearly defined primary outcome. When possible, we also assessed the impact of any outcome discrepancies on each trial’s statistical significance. An outcome discrepancy was defined as favouring statistical significance if the discrepancy resulted in publication of a statistically significant published outcome or if it resulted in demotion of a non-significant registered primary outcome. We considered a trial’s registration to be high quality if it was prospectively registered with matching registered and published primary outcomes.
Statistical methods
We compared registration outcomes between trials having principal investigators with financial ties and those without financial ties using χ2 testing and Fisher’s exact test. Registration outcomes were also compared between trials with and without industry funding using χ2 testing. Logistic regression was used to assess the relationship between the presence of financial ties and a clearly defined registered primary outcome that matched the published primary outcome while controlling for the study start date and sample size, as these factors are both associated with registration rate and quality.9 12 13 Analyses were performed using PASW Statistics V.18.0 (IBM Corp, Armonk, New York, USA). Missing data were excluded in pair-wise fashion.
Results
A total of 192 trials met inclusion criteria and were included in the analysis. Most were phase III trials (53%), and 70% were funded by industry (table 1). The study cohort included trials across a broad range of sample sizes, including 21% with fewer than 100 participants and 27% with at least 500 participants. Principal investigators had financial ties for 130 of the 192 included trials (68%). These ties were most common among trials that were large (88% of trials with ≥500 participants) and among trials sponsored by industry (84%). In total, 134 of the 192 included trials (70%) were positive and 58 (30%) were negative. Trials with financial ties were more likely to be positive than trials with no financial ties (78% vs 53%, P=0.001).
Baseline characteristics of included trials
Almost all trials were registered (97%), and the majority of trials were registered with ClinicalTrials.gov (84%). Sixty-eight per cent (131/192) of the included trials were prospectively registered, with a median time from study start to registration of 15 days (IQR for registration timing was from 26 days before enrolment to 116 days after enrolment; range 906 days before enrolment to 5225 days after enrolment). Fifty-five trials (29%) were retrospectively registered, and six trials (3%) were unregistered (table 2). Among the prospectively registered trials, 103 (79%) had clearly defined primary outcomes, and 28 (21%) had unclear outcomes.
Registration characteristics of included trials
In analysing our primary outcome, clearly defined, prospectively registered outcomes that matched the published outcomes were present in just 76 of the 192 included trials (40%). In the unadjusted primary outcome analysis, trials with investigator financial ties were more likely than those without financial ties to be prospectively registered with clear primary outcomes that matched the published primary outcomes (48% vs 23%, P<0.001). More recent trials were more likely than earlier trials to follow high-quality registration practices (ie, prospectively registered with a matching published outcome) and the proportion of trials with financial ties was also greater among more recently initiated trials. When controlling for study start date and sample size, the presence of investigator financial ties showed a borderline statistically significant association with registration quality (OR 2.12, 95% CI 0.998 to 4.51) (table 3). Within this model, study start prior to 2006 (OR 0.05, 95% CI 0.01 to 0.24 relative to start after 2009) and study start in 2006/2007 (OR 0.42, 95% CI 0.18-.996) were both associated with lower rates of high-quality trial registration. As compared with trials with no financial ties, those with the presence of financial ties were also more likely to be prospectively registered (P<0.01) and more likely to have clearly registered primary outcomes (P<0.01).
Odds of a prospectively registered outcome consistent with the published outcome on unadjusted analysis and after adjusting for study characteristics
Among the 28 trials with an unclear registered primary outcome, positive results in the published manuscript were reported for 21 (75%), and 73 of the 103 (71%) trials with clearly registered primary outcomes reported positive results. Among the 27 trials with unmatched registered and published primary outcomes, 22 (81%) reported positive results, as compared with 51/76 (67%) of those with matching outcomes.
Trials with industry funding were more likely than those without to be registered with a clear primary outcome that was consistent with the published outcome (46% vs 24%, P<0.01), and trials with either investigator financial ties or industry funding were also more likely than those without any industry ties to be registered with a clear, consistent primary outcome (44% vs 22%, P<0.01).
Discrepancies between registered and published primary outcomes were present in 27 trials. The most commonly observed discrepancies were the addition of a new primary outcome in the published manuscript (n=12) and reporting a registered primary outcome as a secondary outcome in the manuscript (n=11). Among trials with financial ties present, 12 of 19 (63%) discrepancies favoured the publication of a statistically significant result, 3 (16%) did not and 4 (21%) could not be classified. Among trials without financial ties, 2 of 8 (25%) discrepancies favoured statistical significance, 5 (63%) did not and 1 (13%) could not be classified.
Rates of retrospective registration, unclear outcome registration and inconsistencies between registered and published primary outcomes were all similar between positive and negative trials (table 4). The impact of primary outcome inconsistencies on the statistical significance of the published outcomes is described in table 5.
Presence of investigator financial ties and evaluation of registration quality among positive and negative trials
Statistical significance of primary outcome changes among prospectively registered trials with inconsistent registered and published outcomes
Discussion
We assessed the consistency between prospectively registered and published primary outcomes among 192 trials that were characterised according to the presence or absence of financial ties among the principal study investigators. We found that approximately half of the trials with financial ties were registered appropriately, with a clear, prospectively defined primary outcome that matched the published outcome, and fewer than one quarter of trials without financial ties met this standard. However, after controlling for other relevant trial characteristics, these differences reached only borderline significance. Retrospective registration, registration with unclear primary outcomes and inconsistencies between registered and published outcomes were common within this cohort of clinical trials regardless of the presence of industry-related financial ties among principal investigators.
This study represents an attempt to understand why trials in which principal investigators have financial interests at issue are more likely to produce positive outcomes as compared with trials without direct investigator financial interests at stake. This pattern is troublesome as it suggests that clinical equipoise, and therefore the ethical justification to randomise participants to different treatment groups, may be violated.14
Several primary mechanisms exist that might result in higher rates of positive trial results among those with investigator financial ties. First, it is possible that investigators with financial ties make study design decisions that increase the chances of producing a trial with positive results. These decisions potentially include choosing comparators or selecting dosages that favour the intervention, selecting outcomes that favour the intervention or choosing data analysis techniques that are more likely to produce favourable results.3–5 15 16 Our study design was unable to address this possibility. However, given the existing evidence showing that study design decisions sometimes introduce bias into industry-sponsored trials, it is likely that similar problems may affect trials with investigator financial ties. Second, it is possible that investigators with financial interests are more likely than others to change primary published outcomes in order to favour statistically significant results. We observed some evidence of outcome switching favouring positive results among trials in which financial ties are present, but this does not appear to be more common than among trials in which investigators do not have financial ties.
Another possibility is that trials with financial ties may be at higher risk of non-publication due to publication bias.6 7 In other words, investigators with financial ties may be less likely to publish trials with results that they perceive as being unfavourable. Previous studies have shown that registered trials sponsored by industry are less likely to be published than those without industry sponsorship.17–19 However, the relationship between investigator financial ties and publication has not been thoroughly assessed. While our data do not directly address the relationship between investigator financial ties and non-publication, we did not observe increased rates of retrospective registration or failure to register among trials with investigator financial ties, which could create the opportunity for bias. Importantly, the absence of this pattern does not rule out publication bias as an explanation for the high observed rate of positive published trials. Additionally, almost 90% of the trials in this sample with investigator financial ties received industry funding, which is associated with high rates of favourable reported study outcomes.16 Industry-funded trials are often performed for regulatory purposes; as such, they often differ from non-funded trials with respect to the population studied, study design and trial implementation. Therefore, the presence of an investigator financial tie may serve as a marker for these other fundamental trial differences.
This study has several limitations that are important to consider when interpreting these results. The majority of the included trials were started between 2006 and 2010. Trial registration patterns have evolved over time, and observations based on this cohort may not reflect current practice.12 The process of assessing registered and published outcomes necessarily involves making somewhat subjective judgements regarding the consistency of these outcomes. We addressed this issue by using multiple independent raters, and in most cases the observed outcome inconsistencies were not subtle. Importantly, our results are consistent with findings from other similar studies assessing registration quality.10 11 20–24 A further study limitation is that we considered the presence or absence of financial ties among principal investigators, which we defined as the first and last study authors. It is possible that for some trials other members of the research team did have financial ties and exerted significant influence over the conduct and reporting of these trials. However, in most cases, the first and last study authors are primarily responsible for study conduct and reporting, and inclusion of other authors in this analysis would have potentially diluted any observed relationship between financial ties and study outcomes or registration practices. Finally, we defined retrospective registration as registration that occurred more than 90 days after the start of trial enrolment. While the International Committee of Medical Journal Editors (ICMJE) requires that registration occur prior to beginning enrolment, previous studies have used this 90-day definition.25 In this case, we considered it unlikely that registration within 90 days after the start of enrolment would have been performed in response to an analysis of the trial data and that registration within this timeframe is likely more reflective of poor familiarity with registration requirements rather than an attempt to impact the trial outcome by changing outcome definitions.
Conclusions
We observe that regardless of the presence of financial ties, less than half of the trials in this cohort were prospectively registered with a clear primary outcome, which was consistent with the primary outcome reported in the published manuscript. Registration practices within this cohort were consistently poor, and after adjusting for relevant factors, registration quality did not differ substantially between trials with and without investigator financial ties. These results do not support our hypothesis that post hoc outcome switching explains the high rate of positive results among trials with investigators that have financial ties to industry. If the findings of RCTS are to appropriately inform patient care, the primary findings reported must reflect the study’s prespecified outcome measures. Trial investigators, sponsors, peer reviewers and editors all have a responsibility to ensure that published manuscripts consistently report the prespecified primary outcomes.
References
Footnotes
Contributors CJ conceived the study. CJ, TFP-M and SK all contributed to the study design. CJ, BSM, AA, RA, AW and SS performed data collection. SK, DK and CJ verified the data. All authors analysed and interpreted the data. CJ drafted the manuscript and all authors critically revised the manuscript and read and approved the final manuscript.
Funding SK is funded by grants from the National Institutes of Health (RO1 HL116522-01A1 and RO1 HL114563-01A1) and Veterans Affairs Health Services Research and Development Service (1IP1HX001994). DK is funded by a Cancer Center Support Grant from the National Cancer Institute to Memorial Sloan Kettering Cancer Center (award number P30 CA008748).
Competing interests We have read and understood BMJ policy on declaration of interests and declare the following interests: CJ is an investigator on unrelated studies sponsored by AstraZeneca, Roche Diagnostics, Inc, Hologic, Inc, and Janssen, for which his department received research grants.
Patient consent Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement The full dataset is available from the corresponding author on reasonable request.