Background Conflicts of interest may bias the findings of systematic reviews. The objective of this methodological survey was to assess the frequency and different types of conflicts of interest that authors of Cochrane and non-Cochrane systematic reviews report.
Methods We searched for systematic reviews using the Cochrane Database of Systematic Reviews and Ovid MEDLINE (limited to the 119 Core Clinical Journals and the year 2015). We defined a conflict of interest disclosure as the reporting of whether a conflict of interest exists or not, and used a framework to classify conflicts of interest into individual (financial, professional and intellectual) and institutional (financial and advocatory) conflicts of interest. We conducted descriptive and regression analyses.
Results Of the 200 systematic reviews, 194 (97%) reported authors' conflicts of interest disclosures, typically in the main document, and in a few cases either online (2%) or on request (5%). Of the 194 Cochrane and non-Cochrane reviews, 49% and 33%, respectively, had at least one author reporting any type of conflict of interest (p=0.023). Institutional conflicts of interest were less frequently reported than individual conflicts of interest, and Cochrane reviews were more likely to report individual intellectual conflicts of interest compared with non-Cochrane reviews (19% and 5%, respectively, p=0.004). Regression analyses showed a positive association between reporting of conflicts of interest (at least one type of conflict of interest, individual financial conflict of interest, institutional financial conflict of interest) and journal impact factor and between reporting individual financial conflicts of interest and pharmacological versus non-pharmacological intervention.
Conclusions Although close to half of the published systematic reviews report that authors (typically many) have conflicts of interest, more than half report that they do not. Authors reported individual conflicts of interest more frequently than institutional and non-financial conflicts of interest.
- conflict of interest
- systematic review
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
Strengths and limitations of this study
First methodological survey of a large and representative sample of systematic reviews evaluating the frequency and different types of conflicts of interest that authors of systematic reviews report.
Use of a comprehensive conflicts of interest framework that allowed assessment of the different types of conflicts of interest, including non-financial conflict of interest.
Use of systematic and transparent methods, for example, duplicate and independent processes in screening and data collection.
Includes systematic reviews limited to the clinical field, that is, our results may not apply to other fields such as health policy.
According to the Institute of Medicine, a conflict of interest (COI) is ‘a financial or intellectual relationship that may impact an individual's ability to approach a scientific question with an open mind’.1 The field of healthcare research has recognised, studied and considered financial relationships when setting COI disclosures and management policies.2–4 Non-financial COIs, such as intellectual, professional and institutional, are increasingly gaining attention.5–7
In an attempt to reduce bias associated with COI, institutions, journals and guideline development groups are increasingly requiring authors to disclose COI.8–12 The World Association of Medical Editors policy statement on COI invites journals to provide authors with COI definitions and with instructions on what and how the different types of COI must be disclosed.13
Generally, authors conduct systematic reviews to ensure that clinical decision-making is based on the best available empirical evidence.14–16 Such reviews also help in defining research gaps and directing funding.17 However, a number of studies have found that systematic review findings may be biased by COI.18–21 This in turn may lead to suboptimal decisions made by patients, clinicians and policymakers.22 Given that COIs have the potential to bias the findings of systematic reviews,23 understanding their nature and frequency among authors of systematic reviews is important. Therefore, we assessed the frequency and different types of conflicts of interest that authors of Cochrane and non-Cochrane systematic reviews report.
Design overview and definitions
This study consists of a methodological survey using systematic methods for study selection and data abstraction. The study involved no human participants and required no ethical approval.
We defined Cochrane systematic reviews as reviews published in the Cochrane Database of Systematic Reviews. We defined non-Cochrane systematic reviews as systematic reviews published in the Core Clinical Journals.
We defined a COI disclosure as the reporting of whether a COI exists or not (ie, includes a statement of the absence of COI). We developed a framework of the different types of COI based on our review of the literature and of the International Committee of Medical Journal Editors (ICMJE) COI disclosure form24 (figure 1 and online supplementary appendix 1 present more details). We developed clear and specific definitions and instructions. The framework included the following types of COI:
Individual financial COI,
Individual professional COI,
Individual intellectual COI,
Institutional financial COI,
Institutional advocatory COI,
‘Other types’ of COI.
We coined the word ‘loogly’ to label any additional statement in the COI disclosure that attempts to downplay a disclosed relationship by suggesting that it is unrelated to COI, for example, ‘this relationship did not influence his prescription of the drug’.
We included articles meeting the following criteria:
Described as a systematic review or a meta-analysis;
Included a search strategy of at least one database;
Planned to include randomised clinical trials or controlled clinical trials comparing at least two alternative therapeutic interventions in humans;
Was published in English in 2015.
We excluded methodological systematic reviews, systematic reviews on causal associations or on diagnostic accuracy, conference abstracts and research letters.
We searched for Cochrane systematic reviews using the Cochrane Database of Systematic Reviews in June 2015. We searched for non-Cochrane reviews using Ovid MEDLINE (In-Process and Other Non-Indexed Citations and Ovid MEDLINE). We limited our search to the 119 Core Clinical Journals (Abridged Index Medicus (AIM)) and to the year 2015. Also, we applied the systematic review filter designed by the Health Information Research Unit of McMaster University, Hamilton, Ontario, Canada. Online supplementary appendix 2 presents the search strategy for each database.
We drew a random sample from the set of citations captured by the literature search to undergo the selection process. We performed the random selection of citations by using an online sequence generator (http://www.random.org/sequences). Two review authors screened in duplicate and independently the titles and abstracts of citations against the inclusion criteria. We acquired the full text of the citations judged as potentially eligible by at least one of the two reviewers.
The same team of review authors screened the full texts in duplicate and independently. The authors resolved disagreements by discussion, involving a third review author when required. We recorded reasons for exclusion, and summarised the search and the results of the selection process using a PRISMA flow diagram.25 The review authors working on study selection completed calibration exercises.
Data extraction process
We collected and managed study data using Research Electronic Data Capture (REDCap) tools hosted at the American University of Beirut.26 REDCap is a secure, web-based application designed to support data capture for research studies.
We developed and pilot-tested a standardised data extraction form with detailed instructions. Seven data extractors completed calibration exercises. They were organised into six teams of two reviewers who extracted data in duplicate and independently. They compared their results and resolved disagreements through discussion, involving a third review author when required.
We extracted information about the following characteristics of the systematic review:
Type of review (Cochrane vs non-Cochrane);
Number of systematic review authors;
Number of included trials;
Number of participants per arm reported by the included trials;
Type of intervention: pharmacological, surgical/invasive procedure, lifestyle intervention, screening/diagnostic intervention, psychotherapeutic intervention, rehabilitation, other;
Type of control: active control (similar to above types of interventions) versus non-active control (no treatment, placebo, sham procedure, waiting list or standard of care);
Risk of bias assessment tool: Cochrane risk of bias tool, a revised Cochrane risk of bias tool, Jadad's scale, other tool, no tool;
Quality of evidence assessment tool: GRADE, other system or no system.
In addition, we extracted information about the characteristics of the reported funding of the systematic review:
Whether authors reported funding, reported no funding or did not report any information on funding;
Reported source(s) of funding (see online supplementary appendix 3 for more details on categories and definitions of the sources of funding);
Whether the role of the funder was reported for: protocol/design of the study, data collection, data analysis and interpretation, preparation/review/approval of the manuscript, decision to submit the manuscript, management, team assembly, other, not involved.
Finally, we extracted information regarding the characteristics of the reported COI disclosures (per categories defined above and in online supplementary appendix 1) of the systematic review authors:
Form in which COI disclosures were provided (eg, a narrative statement, an online document);
Number of authors per paper who report any type of COI;
Number of authors per paper who report each specific type of COI and, when applicable, the different subtypes of COI;
Whether the paper reports relevant characteristics of the COI (eg, source, monetary value, duration);
Number of authors who have the same disclosures or discrepant disclosures reported in the online documents and in the main documents;
Number of authors who have the same disclosures or discrepant disclosures reported in the provided documents and in the main documents;
COI disclosures described as available on request;
Number of authors per paper providing a ‘loogly’ statement.
We assessed agreement between reviewers for inclusion of systematic reviews at the full-text screening stage using chance-corrected agreement (κ statistic).
For the included systematic reviews, we conducted descriptive analyses of the general characteristics of the systematic reviews, the characteristics of the reported funding of the systematic review, and the characteristics of the reported COI disclosures. We analysed the data combined as well as stratified by type of review (Cochrane vs non-Cochrane).
We tested the continuous variables for normality of distribution using the Kolmogorov-Smirnov test. We presented data of non-normally distributed variables as medians and quartiles. The distributions of COI-related variables were highly skewed, showing many papers with no authors reporting a COI and some papers with many authors reporting a COI. Therefore, we opted to present them for each type of COI in a tabular format as follows:
The percentages of papers with at least one author reporting COI (denominator excluding papers that did not provide a COI statement);
For papers with at least one author reporting the type of COI (denominator excluding papers that did not provide a COI statement and papers that reported absence of COI for all authors): the median and IQR of the percentages of authors per paper reporting COI.
We tested whether the differences between the Cochrane and non-Cochrane reviews for the percentage of papers with at least one author reporting COI were statistically significant.
We present results for categorical variables as frequencies and percentages, and analysed them using the χ2 test or Fisher's exact test if the expected event number was <5. We considered a p<0.05 as statistically significant. We performed all calculations using SPSS, V.21.0 for Windows (SPSS, Chicago, Illinois, USA).
For each type of COI, we conducted a multiple logistic regression analysis using data from papers that reported authors' COI disclosures, ‘having at least one author reporting the COI type’ as the dependent variable. The independent variables were the characteristics of the systematic review and variables related to Journal COI policy.27 The latter included the journal impact factor, the journal requirement for COI disclosure form and the journal requirement for disclosure of at least one non-financial COI. We tested the association of each candidate independent variable with the dependent variable and applied a univariable cut-off p value of 0.2 to include the variables in the model.
Out of 1438 citations identified by the search strategy, we included a total of 200 systematic reviews: 100 Cochrane and 100 non-Cochrane systematic reviews (figure 2). The agreement at the full-text screening stage was moderate (κ=0.69).
General characteristics of the systematic review
Table 1 presents the characteristics of the included systematic review papers, stratified by type of review (Cochrane vs non-Cochrane), with a p value for the test of difference between the two types. Cochrane reviews included fewer review authors and trial participants, and were more likely to use Cochrane risk of bias tool and the GRADE system to evaluate the quality of evidence by outcome.
Characteristics of the reported systematic review funding
Table 2 presents the characteristics of the reported review funding stratified by type of review. Twenty-two per cent of reviews did not report their funding source, with non-Cochrane reviews reporting it less frequently. When reported, the top funding source was governmental. Cochrane reviews more frequently reported internal sources of funding and were less likely to report the role of the funder. Only non-Cochrane reviews reported private for profit sources of funding.
Characteristics of the reported COI disclosures
Table 3 presents the characteristics of the reported COI disclosures stratified by the type of systematic review. Overall, 100% and 94% of Cochrane and non-Cochrane reviews, respectively, reported authors' COI disclosures (difference statistically significant). Nine non-Cochrane reviews stated that COI disclosures are available on request; at our request, the corresponding authors for three of those reviews made those disclosures available, one in the form of a narrative statement and two as completed ICMJE forms.
Three non-Cochrane reviews made their COI disclosures available both as narrative statements in the main document and in accessible online ICMJE forms. Two of those reviews had more information on COI in the online document. Out of the three reviews whose disclosures were made available at our request, one review had more information on COI in the provided disclosures compared with disclosures published in the main document.
Different types of COI: Figure 3 displays the distributions of the percentages of authors per paper reporting the different types of COI. Table 4 presents the same distributions in the tabular format described in the Methods section. Of the 194 systematic reviews that provided COI disclosures, 41% had at least one author reporting any type of COI. Institutional COIs were generally less frequently reported than individual COIs. The least reported type of COI was individual professional (3%). The statistically significant differences for the stratified analysis showed that there were more Cochrane reviews, relative to non-Cochrane reviews, with at least one author reporting the following COIs: at least one type, individual intellectual and ‘other types’, in addition to providing with their COI disclosures an additional ‘loogly’ statement, claiming that a relationship is unrelated or non-influential.
Individual financial COI: Figure 4 displays the distributions of the percentages of authors per paper reporting the different subtypes of individual financial COI. Table 5 presents the same distributions in the tabular format described in the Methods section. The most frequently reported subtype was ‘personal fees’ (18%). There were no statistically significant differences for the stratified analysis (Cochrane reviews vs non-Cochrane reviews). Table 6 presents the characteristics of the individual financial COI reported in 60 systematic review papers. Source of COI was the most commonly reported characteristic (87%), particularly by non-Cochrane reviews. Monetary value of the COI was the least commonly reported characteristic (8%), particularly by Cochrane reviews.
Individual intellectual COI: Figure 5 displays the distributions of the percentages of authors per paper reporting the different subtypes of individual intellectual COI. Table 7 presents the same distributions in the tabular format described in the Methods section. The most frequently reported subtype was ‘authorship of primary studies not included in the systematic review’ (10%), particularly by Cochrane reviews. Table 8 presents the characteristics of the individual intellectual COI reported in 24 systematic review papers. Only Cochrane reviews reported whether any type related to one of the products/interventions subject of the systematic review (63%), with a statistically significant difference compared with non-Cochrane reviews.
Institutional financial COI: Table 9 presents the characteristics of the institutional financial COI reported in 19 systematic reviews. Source of COI was the most frequently reported characteristic (90%). All non-Cochrane reviews tend to always report the source of any financial COI (100%). None of the Cochrane reviews specified the monetary value or duration for this type of COI.
Results of logistic regression analyses
Online supplementary appendix 4 presents the details of the multiple logistic regression analyses. These analyses were based on data from papers that reported authors' COI disclosures (n=194). The statistically significant associations for the respective models (ie, respective types of COI) were as follows:
Having at least one author reporting at least one type of COI: journal impact factor (OR=1.16; 95% CI 1.04 to 1.28).
Having at least one author reporting individual financial COI: journal impact factor (OR=1.14; 95% CI 1.04 to 1.25); and type of intervention being pharmacological (reference category being non-pharmacological; OR=2.02; 95% CI 1.04 to 3.91).
Having at least one author reporting individual intellectual COI: no statistically significant associations.
Having at least one author reporting institutional financial COI: journal impact factor (OR=1.09; 95% CI 1.00 to 1.18).
We did not conduct regression analyses for the variables ‘individual professional COI’ and ‘institutional advocatory COI’ due to the low number of papers that reported these COIs (n=5 and 7, respectively).
Summary of findings
The vast majority of systematic reviews report authors' COI disclosures, typically in the main document, and in a few cases either online or on request. Forty-one per cent of reviews had at least one author reporting any type of COI (with a higher percentage for Cochrane reviews). Institutional COIs were generally less frequently reported than individual COIs; non-financial COIs were generally less frequently reported than financial COIs; and Cochrane reviews were more likely to report individual intellectual COI compared with non-Cochrane reviews. We found variability in the reporting of the different types and characteristics of COI (eg, source, monetary value, duration).
Comparison to similar studies
The chart in table 10 compares the current study to five other methodological surveys of COI of authors of systematic reviews. In brief, our review was not restricted to a specific topic, included the highest number of papers and both Cochrane and non-Cochrane reviews, making it the most representative. In addition, it is the only review that included non-financial and institutional COI. Beyari et al28 found that 39% of systematic reviews provided a COI disclosure, whereas we found that 94% of non-Cochrane systematic reviews reported authors' COI disclosures. Four other surveys assessed financial but not other types of COI. Forbes,29 Dunn et al,19 Bes-Rastrollo et al18 and Ebrahim et al20 found that 17%, 27%, 35% and 65%, respectively, of systematic reviews reported individual financial COI. In our study, we found that 31% of systematic reviews reported individual financial COI. Although it is hard to discern any time trends, the variability in results can be attributed to differences in the areas and topics assessed in the different surveys.
A 2014 study found ‘substantial variability’ in the reporting of COI in meta-analyses published in dentistry journals.28 We found similar evidence during our thorough evaluation of the reported COI in the included systematic reviews and meta-analyses published in medical journals. This supports the fact that more stringent regulations on the reporting of COI ought to be adopted by all health-related journals. Such regulations in turn help set standardised systems for reporting and managing COI, two processes that at present seem to lack rigour and remain as sources of confusion to societies and academic investigators.8 ,30
The variability in reporting of COIs in medical journals may hinder the ability of readers to adequately judge those COIs and their implications. Indeed, as shown in previous experiments, a standard consistent section would aid readers in their perceptions of authors' COI disclosures and the reliability of articles.31 ,32 A paper by Maharaj33 discusses a method for scoring financial COI; such scoring scales could aid in adding more integrity to medical research, as well as minimising influence of possibly biased results on decision-making by patients, clinicians and policymakers.
This study assessed disclosed COI but did not verify the accuracy of these disclosures. The US Sunshine Act Open Payments database and the Danish Health and Medicines Authority public disclosure database could be very helpful in undertaking such verification.34 ,35 Unfortunately, such databases are not common, and the task of verification remains challenging and lacks valid tools and methodologies.
Reporting of COI
We found a positive association between reporting COI and journal impact factor. We did not identify any previously published study showing a similar association. One explanation of this association is that journals with higher impact factors are more likely to have disclosure policies. Indeed, two studies found that journals with high impact factors are more likely to have published policies for reporting COI.10 ,36 Another explanation of association is that studies that get published in higher impact journals tend to have more COI. The finding of a positive association between reporting individual financial COI and pharmacological intervention might reflect higher efforts by the pharmaceutical industry to partner with systematic reviewers.
Reporting of the review funding
We found that Cochrane reviews were more likely to report being funded compared with non-Cochrane reviews (86% vs 45%). However, this might be an overestimation because the source of funding in Cochrane reviews need to be inferred from the section on ‘source of support’, which might report on non-funding type of support. Indeed, Cochrane reviews were more likely to report receiving internal funding. It is quite likely that Cochrane reviewers include under ‘internal source of support’ relationships that might not be typically considered as a source of funding. On the other hand, Cochrane reviews were less likely to report on the role of the funder. For these reasons, the Cochrane Collaboration needs to consider a more explicit and specific system for reporting of funding sources.
Strengths and limitations
This is the first methodological survey of a large and representative sample of systematic reviews evaluating how frequently they report their authors' COI, and the types of those COIs. We used a comprehensive framework of COI that allowed us to assess different types of COIs, including non-financial COI. An additional strength is the use of systematic and transparent methods, for example, duplicate and independent processes in screening and data collection.
Although we included a widely representative sample of systematic reviews, it was limited to the clinical field. Thus, our results may not apply to other fields such as health policy. Also, we only describe the COI reported by authors of the systematic reviews. Indeed, authors may not accurately report their COI, or the journal requirements for COI disclosure may be limited.37 ,38
Implications for practice
Given the variability in the reporting of COI in systematic reviews, particularly non-Cochrane reviews, medical journals need to ensure the presence and implementation of COI disclosure policies. In addition, those policies need to better address institutional COIs and non-financial COIs such as professional and intellectual COIs. COI disclosures should also better report COI characteristics such as source, monetary value, duration and how they relate to the product subject of the paper.
Implications for future research
Assessment of the reporting of COI by authors of other types of publications (eg, randomised controlled trials) and in other fields (eg, policymaking) remains to be undertaken. Assessments in different areas may help to refine and validate our proposed COI framework. In addition, it would be important to explore whether the few proposed types of non-financial relationships actually introduce COI, that is, bias judgements. Additionally, there is a need for development of methods to allow checking and verifying the accuracy of COI disclosures.
The authors thank Lara A Kahale and Paul S Ramia for technical support.
Contributors EAA and GG conceived the study. EAA, GG and MBH designed the study. MBH coordinated the study throughout. EAA had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. MBH, SA and EAA screened papers for inclusion. MBH, MA-G, EAA-J, SA, DJH, LCL and AA acquired the data. MBH and EAA analysed and interpreted the data. MBH wrote the first draft of the manuscript with EAA. All authors critically revised the manuscript and approved the final version. The lead author EAA affirms that this manuscript is an honest, accurate and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.
Funding This project was funded by the American University of Beirut Faculty of Medicine's Medical Practice Plan (MPP) funds. The authors and their contributions to the manuscript are independent from the funder.
Competing interests EAA and GG are authors of a number of published papers on conflicts of interest, including non-financial conflicts of interest. All other authors declare no conflicts of interest.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.