Article Text

Download PDFPDF

Original research
Spin within systematic review abstracts on antiplatelet therapies after acute coronary syndrome: a cross-sectional study
  1. Audrey Wise1,
  2. Deepika Mannem2,
  3. Wade Arthur1,
  4. Ryan Ottwell1,
  5. Benjamin Greiner3,
  6. Derek Srouji4,
  7. Daniel Wildes4,
  8. Micah Hartwell1,5,
  9. Drew N Wright6,
  10. Jam Khojasteh7,
  11. Matthew Vassar1,5
  1. 1Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma, USA
  2. 2Arkansas College of Osteopathic Medicine, Fort Smith, Arkansas, USA
  3. 3The University of Texas Medical Branch at Galveston Department of Internal Medicine, Galveston, Texas, USA
  4. 4Cardiology, Oklahoma State University Medical Center, Tulsa, Oklahoma, USA
  5. 5Department of Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma, USA
  6. 6Samuel J. Wood Library & C.V. Starr Biomedical Information Center, Weill Cornell Medical College, New York, New York, USA
  7. 7School of Educational Foundations, Leadership and Aviation, Oklahoma State University - Tulsa, Tulsa, Oklahoma, USA
  1. Correspondence to Audrey Wise; audrey.wise{at}okstate.edu

Abstract

Objectives Spin is a reporting practice in which study results are misrepresented by overestimating efficacy or underestimating harm. Prevalence of spin varies between clinical specialties, and estimates are based almost entirely on clinical trials. Little is known about spin in systematic reviews.

Design We performed a cross-sectional analysis searching MEDLINE and Embase for systematic reviews and meta-analyses pertaining to antiplatelet therapies following acute coronary syndrome on 2 June 2020. Data were extracted evaluating the presence of spin and study characteristics, including methodological quality as rated by A MeaSurement Tool to Assess systematic Reviews (AMSTAR-2). All data extraction was conducted in a masked, duplicate manner from 2 June 2020 to 26 June 2020.

Participants and setting Not applicable.

Primary and secondary outcome measures We assessed abstracts of systematic reviews on antiplatelet therapy following acute coronary syndrome and evaluated the prevalence of the nine most severe types of spin. We additionally explored associations between spin and certain study characteristics, including quality.

Results Our searches returned 15 263 articles, and 185 systematic reviews met inclusion criteria. Of these 185 reviews, 31.9% (59/185) contained some form of spin in the abstract. Seven forms of spin (1, 2, 3, 4, 5, 7 and 9) among the nine most severe were identified. No instances of types 6 or 8 were found. There were no statistically significant relationships between spin and the evaluated study characteristics or AMSTAR-2 appraisals.

Conclusions Spin was present in abstracts for systematic reviews and meta-analyses; subsequent studies are needed to identify correlations between spin and specific study characteristics. There were no statistically significant associations between spin and study characteristics or AMSTAR-2 ratings; however, implementing changes will ensure that spin is reduced in the field of cardiology as well as other fields of medicine.

  • Myocardial infarction
  • Ischaemic heart disease
  • Adult cardiology
  • Coronary heart disease

Data availability statement

Data are available in a public, open access repository. Data are available upon reasonable request. To ensure that our study is transparent and reproducible, we have made the protocol, extraction forms, data analysis scripts and other study artefacts available via Open Science Framework.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • We strictly adhered to our protocol, which was published prior to data extraction on Open Science Framework. We made our protocol, data, analysis scripts, data extraction forms and other study artefacts available online to increase our transparency and reproducibility.

  • As recommended by Cochrane Collaboration, we used an independent group to verify our results and implemented both screening and data extraction in a masked, duplicate manner to strengthen our study and minimise bias.

  • The effect of spin on the reader’s opinions and medical recommendations could not be judged, and the degree of spin use may have affected readers differently. To help mitigate the subjectivity of spin, we used online coursework and training to prepare our team for screening and data extraction.

  • Our study had a cross-sectional design that did not allow for cause and effect analysis or generalisations to be made to other systematic reviews published in other journals.

  • Selection bias may have negatively influenced the data collection. However, this effect was reduced due to the extensive training on methodology and data collection performed prior to extraction.

Introduction

As the gold standard in evidence-based medicine, regulating bodies often use systematic reviews to develop clinical practice guidelines that physicians refer to for treatment decisions.1 2 Hence, the findings contained in systematic reviews and in their abstracts directly influence the care that patients receive and their prognosis.

The systematic reviews themselves serve as a means to collate all relevant primary data on a specific topic, critically appraise the literature, and condense the findings into a useful synthesis. A major issue for patient care is that the reviews often contain outcome reporting bias; that is, they share only positive findings, show selection bias, and/or transfer bias from primary data.3 This bias is further complicated by clinicians having limited access to full-text articles, as well as limited time to read them in decision-making contexts. A 2013 study, for example, found that nearly 75% of physicians read the abstract alone, rather than the full text of the article.4 Importantly, abstracts have been found to not always accurately represent the findings reported in articles. For example, Toma and colleagues reported that 41% of the abstracts of randomised controlled trials (RCTs) contained different efficacy estimates than the full text.5 Consistent with that report, other studies have shown that over 50% of abstracts had discrepancies in the results compared with the full text, and only 50% of abstracts discussed the side effects or harms experienced during clinical trials.6 7 Considering the importance of evidence-based medicine, the necessity for reducing bias, especially in the abstracts of systematic reviews, cannot be overstated.

One form of bias that is of increasing interest to systematic reviewers, clinical trial researchers and epidemiologists is spin. Spin involves misrepresenting research results to mislead readers into favouring a certain outcome. It has been identified in multiple studies, particularly in clinical trial research, accessible from these publicly available data sets.8 9

Accurate and objective research findings are necessary to guide patient care. For example, patients who have experienced acute coronary syndrome (ACS) with subsequent initiation of antiplatelet therapy are particularly at risk for morbidity and mortality arising from antiplatelet therapy inaccuracies. If clinicians misinterpret (or are misled by) clinical research on antiplatelet therapy, the results could be deadly.

However, no study has analysed spin in this body of literature. Thus, the primary objective of this study was to analyse the presence of spin in the abstracts of systematic reviews on antiplatelet therapy following ACS. The secondary objective was to assess the study characteristics, including journal impact factor, funding source and article type, associated with spin. We additionally describe methods for reducing spin in antiplatelet publications.

Methods

Search strategy

A systematic review librarian (DW) constructed the search strategies for the MEDLINE (Ovid) and Embase (Ovid) databases to locate systematic reviews and meta-analyses pertaining to antiplatelet therapies (figure 1). DW conducted these searches on 2 June 2020. Following the execution of these search strategies, the results were combined and uploaded into a screening platform, Rayyan (https://rayyan.qcri.org/). The duplicates were removed, and the titles and abstracts were independently screened for eligibility by two researchers (AW and DM).10 This process was performed in a masked, duplicate manner. Once the initial screening was complete, AW and DM resolved disagreements through discussion.

Eligibility criteria

The following inclusion criteria had to be met for an article to be eligible for this study. The article must (1) be a systematic review with or without a meta-analysis; (2) focus on antiplatelet, anticoagulation or thrombolytic therapies following ACS; (3) be conducted on human subjects only; (4) be available in English; and (5) have an abstract. The following study types were excluded from our study: observational studies (case-control, cohort, surveys), clinical trials, narrative reviews and systematic reviews not related to antiplatelet therapies. We also excluded duplicates, withdrawn or retracted studies, letters to the editor, and any remaining article that did not meet the inclusion criteria. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocols (PRISMA) definitions of systematic reviews and meta-analyses were used.11 The selected studies were then uploaded to Stata for randomisation, and as long as the previous criteria were met, the first 200 articles were sequentially extracted.

Training

Several levels of training were completed to ensure the quality of this study. Prior to the start of the study, all researchers were trained on an overview of study designs in clinical research, deduplication, and screening on Rayyan, and publication of protocols using Open Science Framework (OSF).12 The two researchers who conducted the screening (AW and DM) also completed an online training course on systematic reviews and meta-analyses offered by John Hopkins University on the Coursera platform.13 Following the course, the researchers undertook 2 days of online and in-person training. The training session covered the definition and interpretation of the nine most severe forms of spin in systematic review abstracts as defined by Yavchitz et al.14 Finally, data extractors were trained on how to analyse a systematic review via A MeaSurement Tool to Assess systematic Reviews (AMSTAR-2). The AMSTAR tool was used to determine the methodological quality of each included systematic review and meta-analysis. A detailed outline of the training can be found in our study protocol.

Identification of spin and data extraction

Data extraction was completed by AW and DM, using a pilot-tested Google form to evaluate for nine types of spin. These forms of spin as defined in Yavchitz et al14 studies were chosen and settled on following a Q-sort survey with members of the Cochrane Collaboration. This consisted of method review and bias method group members from over 400 invited participants. Members ranked examples on the Q-sort survey for more severe and prevalent forms of bias and were evaluated in our study. The selection of the nine types of spin in our study is derived from the most severe types of spin Yavchitz el al14 identified as well as from publicly available data in recent published literature15 on spin for completeness. The following general information was extracted from all reviews: the intervention type (surgery, pharmacologic and non-pharmacologic); the year in which the review was received by the journal; the funding source (industry, private, public, none, not-mentioned and hospital); whether the review discussed adherence to PRISMA16 or PRISMA for abstracts17; whether the journal recommended adherence to PRISMA; and the journal’s 5-year impact factor. The nine types of spin are defined in table 1. Researchers coded each of the nine spin types as being either present or absent from the systematic review, except for type 4 spin if safety was not addressed in the manuscript. If a systematic review did not address safety, it was not included in the analysis for type 4 spin. Any disagreements about the presence of spin were reconciled between AW and DM, and if an agreement could not be reached WA and MV acted as mediators. Following spin data extraction, the quality of each systematic review was analysed using AMSTAR-2 (https://amstar.ca/).18 AMSTAR-2 has previously been used to evaluate systematic reviews in multiple medical disciplines, including surgery, cancer, psychiatry, pain management and general medical interventions.18–22 Based on scores from the 16-item scale, reviews were given confidence ratings of high, moderate, low or critically low.18 Inter-rater reliability scores (assessed by kappa) for AMSTAR-2 ranged from .31 to 1.0,18 with additional studies supporting its reliability, finding moderate inter-rater reliability (median K=0.56) among multiple raters. Construct validity coefficients were high (AMSTAR, r=0.91; Risk of Bias in Systematic Reviews, r=0.84).23 All study characteristics, spin data and AMSTAR-2 data were extracted in a masked, duplicate manner.14

Table 1

Spin types and frequencies in abstracts (n=185)

Statistical analysis

The overall frequency of spin, study characteristics and AMSTAR-2 ratings were analysed using descriptive statistics, and the results are reported as both frequency counts and percentages. Following the screening, we included 185 systematic reviews. Binary logistic regression was used to determine the strength of associations via ORs between the occurrence of spin in a study’s abstract with journal impact factor, study characteristics and AMSTAR-2 appraisal rating, and a multivariable model was used to control for multiple characteristics. Prior to the start of the study, we determined that 185 systematic reviews were needed to power the analyses using GPower V.3.1.9.7. Our analytic decisions are included in our protocol, and Stata V.16.1 (StataCorp, College Station, Texas, USA) was used for all analyses.

Patient and public involvement

This study did not include any human participants or public involvement and therefore did not meet the regulatory definition of human subject research per the US Code of Federal Regulations and was not subject to Institutional Review Board oversight.

Oversight, data availability, transparency, reproducibility and reporting

To ensure that our study is transparent and reproducible, we have made the protocol, extraction forms, data analysis scripts and other study artefacts available via OSF. Our data and analysis scripts were sent to an independent investigation team and reanalysed in a masked fashion to further the reproducibility. This study was conducted in tandem with other studies evaluating the presence of spin in systematic reviews in other medical fields. These studies adhered to a common methodology, which has been described elsewhere. While drafting this article, we integrated the relevant reporting guidelines from PRISMA24 and Murad and Wang’s25 guidelines for meta-epidemiological studies.

Results

General characteristics

Our search queries returned 15 263 articles and 12 913 remained after deduplication. Of these, 12 372 were excluded after the initial title and abstract screening, which left 541 unique systematic reviews. A full-text review of the articles further excluded 356 reviews. Finally, 185 systematic reviews and meta-analyses pertaining to antiplatelet therapies following ACS remained and were included in data extraction. Our exclusion list from the screening process with rationale can be found in figure 2.

Figure 2

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram of study selection. ACS, acute coronary syndrome.

All studies meeting inclusion criteria analysed pharmacological interventions. Of the 185 reviews, 61 (32.9%) systematic reviews were published in journals that recommended adherence to PRISMA, and 79 (42.7%) of the papers mentioned adherence to PRISMA or PRISMA extension for Abstracts (PRISMA-A). The majority of the systematic reviews and meta-analyses did not mention a funding source (93/185, 50.2%), followed by systematic reviews that did not receive any kind of funding (44/185, 23.8%). Of the papers that mentioned a funding source (92/185, 49.7%), the most common funding source was private funding (20/185, 10.8%), followed by public funding (17/185, 9.2%) and industry funding (11/185, 6%).

Spin in abstracts

Among the included systematic reviews, 31.9% (59/185) contained at least one form of spin in the abstract. Some articles contained more than one type of spin; therefore, 61 instances of spin were found. The most common form of spin found in the 185 articles was type 3 in 42 articles (42/185, 22.7%). Following type 3 spin, 10 articles (10/185, 5.4%) contained type 1 spin. Regarding type 4 spin, safety was not assessed in 145 systematic reviews. Therefore, this spin type was only evaluated in 40 of our included studies. Of those 40 studies, 15% (6/40) contained type 4 spin. Spin type 2 (2/185, 1.1%), type 5 (6/185, 3.2%), type 7 (1/185, 0.5%) and type 9 (3/185, 1.6%) were found less frequently. No abstracts contained type 6 or type 8 spin. There was no significance between spin and any of the evaluated study characteristics. All study characteristics in the presence of spin are mentioned in table 2.

Table 2

General characteristics of systematic reviews and meta-analyses

AMSTAR-2

After appraising the systematic reviews with the AMSTAR-2 instrument, we rated 110 (59.5%) of the 185 studies as ‘critically low,’ 29 (15.7%) as ‘low,’ 22 (11.9%) as ‘moderate,’ and 24 (13%) as ‘high’. We found no statistical significance between spin and AMSTAR-2 appraisals (table 3).

Table 3

AMSTAR-2 Items and frequency of responses (n=185)

Discussion

Our study found spin in approximately one-third (31.9%) of the included systematic reviews. Spin has been shown to be prevalent in RCTs across many different fields of medicine.9 26 Recent studies have shown spin to be present not only within RCT abstracts but also within abstracts of systematic reviews.27 28 The results of the present study support the findings by Austin et al which identified one-third to one half of systematic review abstracts contained at least one form of spin, noted in their publicly available data set.26 Although multiple studies have evaluated spin in systematic reviews, to our knowledge, ours is the first to evaluate the use of spin within antiplatelet research. Here, we highlight the most common and the most severe forms of spin we found and the influence they have on the clinical use of antiplatelet therapies.

The most common type of spin found among the systematic reviews and meta-analyses that we analysed was type 3 spin (selective reporting of or overemphasis on efficacy outcome or analysis favouring the beneficial effect of the experimental intervention). To further describe type 3 spin, if efficacy outcomes are specified within the full text and are not reported in the abstract, spin is committed. Consider the following example noted by Dundar.29 They examined mortality and major adverse effects associated with thrombolytic agents (alteplase, tenecteplase, streptokinase and reteplase) in the treatment of acute myocardial infarction. In addition to mortality, their outcome measures included bleeding, stroke, reinfarction, allergy and anaphylaxis. Of the six outcomes, only morality and stroke were reported in the abstract. Interestingly, three of the four outcomes that were not reported in the abstract (allergic reaction, anaphylaxis and reinfarction) showed no statistical significance. Due to omission of these non-significant harm outcomes, the efficacy outcome of decreased mortality was selectively reported. Selectively reporting efficacy outcomes while leaving out harm outcomes can lead physicians to believe that an intervention is safer than it is; this belief may potentially lead to preventable adverse outcomes.

The most severe type of spin (conclusion contains recommendations for clinical practice not supported by the findings) was identified in our sample. This form of spin had the second greatest frequency among the included systematic reviews in our study. A study by Kinnaird et al provides an example of this form of spin. The authors claim that ‘This study suggests that bivalirudin is more effective and safer than heparin monotherapy and should, therefore, be preferred over heparin monotherapy’.30 However, the recommendation for using bivalirudin over heparin was not supported by all outcomes of interest. Non-significant outcomes of interest included risk for stroke and myocardial infarction. Without consistent significant findings for all outcome measures, the claim for clinical practice is spin. In our view, this spin type is the most severe because a recommendation for clinical practice can have an immediate influence on patient care.

The methodological quality of the included systematic reviews was analysed using AMSTAR-2. Among the reviews in our sample, over half were scored as critically low based on the AMSTAR-2 grading criteria. AMSTAR-2 defines this category as having at least one critical flaw rendering the systematic review unable to provide an accurate summary or recommendation.18 However, the presence of spin in the abstract was not significantly associated with a low AMSTAR-2 score for the systematic review. In light of these findings, spin may be found in systematic reviews regardless of their quality; there may be a multitude of factors propagating spin aside from quality, including abstract word limitations, funding conflicts of interest and lack of author awareness of spin. Although improving methodological quality may not eradicate the presence of spin, our findings support an opportunity for future systematic reviews to increase the quality of publications via adherence to AMSTAR-2.

Our study illustrates a current gap in transparency within the research pertaining to antiplatelet therapies. Whether intentional or not, motivating factors for the use of spin within manuscripts include confirmation bias, allegiance bias, and the misrepresentation of results in hopes for publication. Ioannidis et al suggest that the motives include a belief that positive results are associated with an increased likelihood of publication and greater clinical significance.31 Regardless of whether it is intentional or not, the misrepresentation of data may lead to suboptimal patient care.32 To help improve the reporting quality of systematic reviews on antiplatelet therapies following ACS, we have a few recommendations. First, readers must be made aware that spin exists within systematic reviews, especially as physicians often use systematic reviews to develop clinical practice guidelines. Given that Bero et al identified in their publicly available data set that spin is less prevalent in publications evaluating spin,33 we believe an awareness of spin is vital to decreasing spin in publications. Thus, by creating an online resource dedicated to teaching spin, as well as presenting on spin at medical conferences, we believe knowledge of spin would be increased among medical students, residents and physicians and its prevalence in publications may decrease. Next, we suggest that journal editors require the implementation of spin screening prior to author submission. Our rationale is that entities who control article publications have the greatest influence to change the amount of spin present in publications. Furthermore, in an effort to de-escalate spin, we urge journal editors to strive for inferential reproducibility by using a second discussant to verify study findings prior to publication.31 Inferential reproducibility is attained for a publication by an independent reviewer re-analysing the study to see if the same results and conclusions are reached.34 Additionally, we recommend journal editors emphasise the publication of studies with neutral results given that researchers believe positive results are more likely to be published.34 Lastly, PRISMA and PRISMA-A were designed to improve the reporting quality of systematic reviews. However, our study did not find less spin reported in publications that adhered to PRISMA. Thus, we suggest the addition of an item regarding spin to PRISMA and PRISMA-A to help reduce spin present in publications and systematic reviews.

Strengths and limitations

Our study contained key strengths and limitations. Regarding strengths, we strictly adhered to our protocol, which was published prior to data extraction on OSF. We made our protocol, data, analysis scripts, data extraction forms and other study artefacts available online to increase our transparency and reproducibility. Furthermore, we used an independent group to verify our results and implemented both screening and data extraction in a masked, duplicate manner to strengthen our study and minimise bias. This process is recommended by the Cochrane Collaboration. Regarding limitations, spin by nature is subjective and interpretation could vary between extractors. First, the effect of spin on the reader’s opinions and medical recommendations could not be judged. The degree of spin use may have affected readers differently. To help mitigate the subjectivity of spin, we used online coursework and training to prepare the team for screening and data extraction. An additional limitation is that our study had a cross-sectional design that did not allow for cause and effect analysis or generalisations to be made to other systematic reviews published in other journals. Furthermore, selection bias may have negatively influenced the data collection. However, this effect was reduced due to the extensive training on methodology and data collection performed prior to extraction. Lastly, our searches may not have returned all systematic reviews on antiplatelet therapies following ACS although we conducted our searches using the largest bibliographic databases (MEDLINE and Embase).

Conclusion

In conclusion, our study found spin is present in abstracts of systematic reviews regarding antiplatelet therapies following ACS. Although there were no statistically significant relationships between spin and the evaluated study characteristics or AMSTAR-2 appraisals, this misrepresentation of research data poses a threat to optimal evidence-based clinical use of antiplatelet therapy following ACS. There are steps that can be taken to improve the quality of antiplatelet therapy systematic reviews.

Data availability statement

Data are available in a public, open access repository. Data are available upon reasonable request. To ensure that our study is transparent and reproducible, we have made the protocol, extraction forms, data analysis scripts and other study artefacts available via Open Science Framework.

Ethics statements

Patient consent for publication

Ethics approval

Not applicable.

References

Footnotes

  • Twitter @audwise

  • Contributors AW: acquisition, analysis and interpretation of data, manuscript drafting and revision. DM: acquisition, analysis and interpretation of data, manuscript drafting. WA: conception and design of the work, drafting and revision of manuscript. RO: conception and design of the work, revision of manuscript. BG: drafting of the manuscript. DS: revision of manuscript. DW: revision of manuscript. MH: analysis and interpretation of data, revision of manuscript. DNW: conceptual design of the work. JK: analysis and interpretation of data. MV: conceptual design of the work, revision of manuscript, guarantor.

  • Funding This study was funded by the Oklahoma State University Center for Health Sciences Presidential Mentor-Mentee Research Fellowship Grant.

  • Competing interests None declared.

  • Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

  • Provenance and peer review Not commissioned; externally peer reviewed.