Article Text

Download PDFPDF

Quality of reporting in abstracts of RCTs published in emergency medicine journals: a protocol for a systematic survey of the literature
  1. Federico Germini1,2,
  2. Maura Marcucci3,4,
  3. Marta Fedele5,
  4. Maria Giulia Galli6,
  5. Lawrence Mbuagbaw7,8,
  6. Valentina Salvatori9,
  7. Giacomo Veronese10,
  8. Andrew Worster11,
  9. Lehana Thabane7,8,12,13,14
  1. 1 Department of Emergency, Fondazione IRCCS Ca’ Granda – Ospedale Maggiore Policlinico, Milano, Italy
  2. 2 Department of Health Sciences, Università degli Studi di Milano, Milano, Italy
  3. 3 Geriatric Unit, Fondazione IRCCS Ca’ Granda – Ospedale Maggiore Policlinico, Milano, Italy
  4. 4 Department of Clinical Sciences and Community Health, Università degli Studi di Milano, Milano, Italy
  5. 5 Department of Emergency, Area Nord, Azienda Unita Sanitaria Locale di Bologna, Bologna, Italy
  6. 6 Department of Clinical and Experimental Medicine, Università degli Studi di Parma, Parma, Italy
  7. 7 Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada
  8. 8 Biostatistics Unit, Father Sean O’Sullivan Research Centre, St Joseph’s Healthcare, Hamilton, Ontario, Canada
  9. 9 General Practitioner Course, Regione Marche – Servizio Sanità, Ancona, Italy
  10. 10 Department of Emergency Medicine, Ospedale Niguarda Ca’ Granda, Milano, Italy
  11. 11 Division of Emergency Medicine, McMaster University, Hamilton, Ontario, Canada
  12. 12 Departments of Paediatrics and Anaesthesia, McMaster University, Hamilton, Ontario, Canada
  13. 13 Centre for Evaluation of Medicine, St Joseph’s Healthcare, Hamilton, Ontario, Canada
  14. 14 Population Health Research Institute, Hamilton Health Sciences, Hamilton, Ontario, Canada
  1. Correspondence to Dr. Lehana Thabane; thabanl{at}mcmaster.ca

Abstract

Introduction The quality of reporting of abstracts of randomised controlled trials (RCTs) in major general medical journals and in some category-specific journals was shown to be poor before the publication of the ConsolidatedStandards of ReportingTrials (CONSORT) extension for abstracts in 2008, and an improvement in the quality of reporting of abstracts was observed after its publication. The effect of the publication of the CONSORT extension for abstracts on the quality of reporting of RCTs in emergency medicine journals has not been studied. In this paper, we present the protocol of a systematic survey of the literature, aimed at assessing the quality of reporting in abstracts of RCTs published in emergency medicine journals and at evaluating the effect of the publication of the CONSORT extension for abstracts on the quality of reporting.

Methods and analysis The Medline database will be searched for RCTs published in the years 2005–2007 and 2014–2015 in the top 10 emergency medicine journals, according to their impact factor. Candidate studies will be screened for inclusion in the review. Exclusion criteria will be the following: the abstract is not available, they are published only as abstracts, still recruiting, or duplicate publications. The study outcomes will be the overall quality of reporting (number of items reported) according to the CONSORT extension and the compliance with its individual items. Two independent reviewers will screen each article for inclusion and will extract data on the CONSORT items and on other variables, which can possibly affect the quality of reporting.

Ethics and dissemination This is a library-based study and therefore exempt from research ethics board review. The review results will be disseminated through abstract submission to conferences and publication in a peer-reviewed biomedical journal.

  • Accident & emergency medicine
  • Statistics &research methods
  • Abstract
  • Quality of reporting
  • Research methodology

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Strengths and limitations of this study

  • This is the first review on the quality of reporting of abstracts of randomised controlled trials in emergency medicine journals.

  • Study selection and quality assessment will be performed in duplicate.

  • This is a comparison of the quality of reporting before and after the publication of Consolidated Standards of Reporting Trials extension for abstracts to assess possible improvement.

  • Possible other factors contributing to the quality of reporting will be investigated.

  • Limited availability of data on explanatory variables could be a limitation.

Introduction

Randomised controlled trials (RCTs) are considered the best study design to test interventions in clinical research. Readers often rely on information reported on abstracts when they have to decide whether or not to go through the full text of an article, and in many cases abstracts can be the only source of information for physicians and researchers.1 This can be due to lack of access to the full texts, time limitations or inability to understand the technicalities of clinical trials. Moreover, many educational sources (eg, Evidence Update from BMJ and McMasterPLUS)2 provide only the abstracts of selected studies to their users. Hence, despite being encouraged to independently verify any information, readers may be prone to rely solely on abstracts to make their decisions, which is consistent with the process of article selection for systematic reviews, that usually starts with the screening of abstracts.3 For all these reasons, incomplete reporting of studies in their abstracts can lead to inaccurate interpretation of results, missed identification of potential sources of bias, and missed inclusion in systematic reviews. This can translate into an incorrect application of trial results in clinical practice. The Consolidated Standards of Reporting Trials (CONSORT) statement4 was developed by an international group of clinical trial methodologists, guideline developers, knowledge translation specialists and journal editors, and consists of a 25-item checklist intended to improve the reporting of RCTs. In an attempt to improve the reporting of abstracts, an extension of the CONSORT statement designed for abstracts was published by the CONSORT group in 2008.5 The quality of reporting of abstracts in major general medical journals before the publication of CONSORT statement was poor,6 and improved after it was published.7 Still, suboptimal adherence to CONSORT for abstracts guidelines was observed in general medical journals8 and in some category-specific journals.9–12 Emergency medicine publications are increasing worldwide, and RCTs follow this trend.13 Nevertheless, conducting research in emergency medicine can be challenging, likely more challenging than in other medical areas. The American College of Emergency Physician identified the following as key factors that contribute to making difficult the conduct of research in emergency medicine: time, money, personnel, disinterest in research, patient mistrust and regulatory burden.14 These factors, although not unique to emergency medicine, are amplified in such a dynamic and intensive clinical setting. This can lead to many methodological issues in the implementation of RCTs, which can eventually translate into suboptimal reporting, even to a larger extent than as already shown in general medicine and other category-specific publications. It is in this context that we planned our methodological systematic survey. To the best of our knowledge, the quality of reporting of abstracts for RCTs published in emergency medicine journals and the impact of the publication of the CONSORT guidelines for abstracts in this field of medicine have not been investigated. We conducted a scoping search in August 2016 and found only one conference abstract exploring the quality of reporting of abstracts in emergency medicine journals.15

Purpose and objectives

The purpose of the systematic survey is to inform physicians and researchers in the field of emergency medicine of the quality of reporting in abstracts of RCTs published in emergency medicine journals, and to assess the effect of the publication of the CONSORT extension for abstract on the quality of reporting. Specifically, our primary objective is to assess the compliance with the CONSORT extension items in emergency medicine journals, before comparing with after the publication of the CONSORT extension. Our secondary objective is to explore which factors, other than the year of publication, are associated with the quality of reporting of abstracts.

Hypothesis

We hypothesise that the quality of reporting of abstracts has improved after the publication of the dedicated CONSORT statement extension.

Methods

Study design

This study will be a systematic survey to assess quality of reporting. The Medline database will be searched for RCTs published in the years 2005–2007 and 2014–2015 in the top 10 exclusive emergency medicine journals, ranked on their impact factor according to the Journal Citation Report 2015 provided by Thomson Reuters16 (table 1). At first, we decided to take a 2-year period before and after CONSORT for abstracts (2006–2007 and 2014–2015). After a scoping search, we realised that probably the optimal time was a 3-year period before CONSORT for abstracts (2005–2007) to be able to reach the sample size. Journals created after 2006 will be excluded from the review, otherwise the period before the publication of the CONSORT extension for abstract (ie, before 2008) would not be sufficiently represented for those journals, not allowing a correct before–after comparison. Journals coming next in the Journal Citation Report list will be included to achieve the number of 10.

Table 1

Top exclusive emergency medicine journals based on impact factor, according to the Journal Citation Report 2015 provided by Thomson Reuters

The search strategy will include the Cochrane highly sensitive search strategy for identifying randomised trials in Medline (sensitivity-maximising and precision-maximising version, 2008 revision),3 journal names and limits set for the specific time periods of interest (years of publication 2005–2007 and 2014–2015). The search strategy is reported in detail in box 1.

Box 1

Search strategy adopted for randomised controlled trials published in top 10 emergency medicine journals in the years 2005–2007 and 2014–2015 in the Medline database

  1. randomized controlled trial [pt]

  2. controlled clinical trial [pt]

  3. randomized [tiab]

  4. placebo [tiab]

  5. clinical trials as topic [mesh: noexp]

  6. randomly [tiab]

  7. trial [ti]

  8. #1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7

  9. animals [mh] NOT humans [mh]

  10. #8 NOT #9

  11. (‘2005’[Date - Publication]: ‘2007’[Date - Publication])

  12. (‘2014’[Date - Publication]: ‘2015’[Date - Publication])

  13. #11 OR #12

  14. ‘Resuscitation’[Journal]

  15. ‘Annals of emergency medicine’[Journal]

  16. ‘Academic emergency medicine: official journal of the Society for Academic Emergency Medicine’[Journal]

  17. ‘Prehospital emergency care: official journal of the National Association of EMS Physicians and the National Association of State EMS Directors’[Journal]

  18. ‘European journal of emergency medicine: official journal of the European Society for Emergency Medicine’[Journal]

  19. ‘Injury’[Journal]

  20. ‘Emergency medicine journal: EMJ’[Journal]

  21. ‘World journal of emergency surgery: WJES’[Journal]

  22. ‘The American journal of emergency medicine’[Journal]

  23. ‘Emergency medicine Australasia: EMA’[Journal]

  24. #14 OR #15 OR #16 OR #17 OR #18 OR #19 OR #20 OR #21 OR #22 OR #23

  25. #10 AND #13 AND #24

Inclusion criteria

To be included in the review, the abstract must be a report of primary outcome(s) of an RCT and involves human subjects. Simulation based-studies (eg, study performed using manikins) will be considered for inclusion only if needed to achieve the sample size.

Exclusion criteria

Studies will be excluded if the abstract is not available, they are published only as abstracts (eg, conference proceedings), still recruiting, reporting on outcomes other than primary outcome or duplicate publications.

A flow diagram showing the study screening and selection procedures is reported in figure 1.

Figure 1

Flow diagram showing study selection procedure.

Outcomes

The overall quality of reporting (number of items reported) according to the CONSORT extension and the compliance with its individual items will serve as outcomes. The following checklist items will not be included: ‘Authors’ (contact details for the corresponding author) and ‘Recruitment’ (trial status), as these items are specific to conference abstracts.5 Thus we will eventually test the compliance with 15 items.

Explanatory variables

Timing of publication (pre-CONSORT vs post-CONSORT publication) will be the explanatory variable for the primary objective. Explanatory variables for the secondary objective will be the following: CONSORT endorsement by the journal (defined as the presence of the request to comply with the CONSORT statement, among the instructions for authors/authors guidelines section in the journal website), number of centres involved in the study (single-centre vs multicentre), study sample size (≤100 vs >100), type of intervention (pharmacological vs non-pharmacological), significance of results for primary outcome (significant vs non-significant), source of funding (industry-funded vs non-industry-funded) and study setting (out-of-hospital vs in-hospital).

Sample size calculation

The primary objective of this systematic survey is to compare the mean number of reported items in pre-publication versus postpublication of the CONSORT extension for abstracts based on the corresponding checklist.4 A review assessing the quality of abstracts in general medical journals found a mean difference in the number of items reported between after and before the publication of the CONSORT statement for abstracts of 3.05 (95% CI 2.44 to 3.65, p<0.001), going from 9.06 (SD: 2.15) in 2007 to 12.11 (SD: 2.22) in 2012.7 If we expected exactly the same results, the sample size required to show such a difference with a significance of 0.05 and a power of 0.8, considering an SD of about 2, would be eight in each study period (pre-CONSORT vs post-CONSORT). The sample size would be smaller if we consider that it is likely that from 2012 to 2015 a further improvement in the quality of reporting occurred. For example, assuming a difference of four, the sample size would be four in each study period. On the other hand, given that the generalisability of these results to studies in emergency medicine is uncertain, we hypothesise also a worse scenario in which we expect a mean difference of 1.5 items correctly reported between the two periods. The sample size required to show this effect size would be 28 in each study period. Because we are looking at the top 10 journals and so we are including articles published in the same journal, we adjusted the raw estimate in both the scenarios for a possible cluster effect multiplying by a factor of 1.42, obtained assuming an intraclass correlation coefficient of 0.03 and an average number of articles per journal of 15.17 Within the boundaries of the two scenarios, the sample size would range from 6 to 40 articles per period. Furthermore, the effect of the period will be explored in a multivariable model including also the following seven variables: endorsement of CONSORT statement by the journal, number of centres involved in the RCT, type of intervention, sample size, results of the trial, funding status and study setting. Thus, we further adjusted the sample size upward by adding five articles for each variable (35). Hence, we will be able to detect a mean difference as low as 1.5 items correctly reported as long as we find a number of eligible articles of at least 58 per period (116 in total). A smaller number of eligible articles would allow us to detect only larger effect sizes. If we find a number of articles higher than 58 per period, the articles will be randomly selected. If possible, we will select the same number of articles from each journal (6±1 per period).

Data extraction and synthesis

For each article, two independent reviewers will screen eligibility criteria and will extract data on the reporting of the 15 items of the CONSORT statement for abstracts that we have selected and data regarding the explanatory variables. When needed, full texts will be reviewed to retrieve those data. The reviewers will use the CONSORT elaboration and explanation document18 as a guide. Disagreements between reviewers will be resolved via discussion or, when necessary, with the intervention of a third reviewer. In order to increase accuracy and consistency between reviewers, an initial trial run involving 10 articles per reviewer will be undertaken.

Statistical analyses

The characteristics of the included articles will be synthesised using descriptive statistics such as mean (SD) or median (first quartile, third quartile) for continuous variables and count (frequency) for categorical variables. We will describe the frequency with which each item is reported and the mean number of reported items by period of publication (pre-CONSORT vs post-CONSORT). The primary outcome (mean difference of items reported) will be estimated first using a two-sample t-test (unadjusted mean difference) and then through generalised estimation equations (GEEs) in order to adjust for the prespecified explanatory variables.19 The mean differences and adjusted means will be reported with 95% CIs and p values. The same multivariable GEEs will be used to explore the effect of the explanatory variables other than the period of publication.

In order to strengthen our confidence in the results, the effect of the period of publication will be explored through two additional approaches. First, we will look at the compliance with each of the 16 items of the CONSORT statement for abstracts for years 2005–2007 versus 2014–2015 as expressed by the binary outcome item reported or not reported. We will perform both unadjusted analyses using χ2 tests and adjusted analyses using GEEs and assuming binomial distributions and unstructured correlation matrices. The results of the GEEs will be reported as adjusted ORs, with 95% CIs and p values. Then, the incidence rate ratios (IRRs) for reporting items for the period 2014–2015 compared with the period 2005–2007 will be estimated using GEEs based on a Poisson distribution and an unstructured correlation matrix. The results will be reported as adjusted IRRs, with 95% CIs and p values. The criterion for statistical significance will be set at alpha=0.05.

In the multivariable analyses, adjustments will be made for (1) CONSORT endorsement by the journal, (2) number of centres involved in the study (single-centre vs multicentre), (3) study sample size (≤100 vs >100), (4) type of intervention (pharmacological vs non-pharmacological), (5) significance of results for primary outcome (significant vs non-significant), (6) source of funding (industry-funded vs non-industry-funded) and (7) study setting (out-of-hospital vs in-hospital), with journal as a grouping factor to adjust for potential clustering or similarity in articles published in the same journal. Descriptive data will be presented as counts and percentages. Between-reviewer agreements will be measured using the kappa statistic.20 Data will be analysed using STATA/SE V.12.0.

Discussion and dissemination

Our systematic survey is aimed at assessing the quality of reporting of abstracts of RCTs published in emergency medicine journals, and the influence on that of the publication of the CONSORT extension for abstracts. Adequate reporting of abstract of RCTs would improve transparency, correct assessment and interpretation of trials, accurate indexing, retrieval, and appropriate inclusion in systematic reviews. We expect the quality of reporting of articles published after the publication of the CONSORT extension for abstract to be better than before. Besides the publication of the CONSORT extension for abstracts, other variables can affect the quality of reporting. That is why we will explore the role of the following: journal’s endorsement of the CONSORT statement, number of study centres, sample size, type of intervention, significance of the result of primary outcome, funding of the study and setting (out-of-hospital vs in-hospital). Based on previous experiences, RCTs from journals endorsing CONSORT,21 22 multicentre studies,22 23 studies with larger sample size,22 24 involving pharmacological intervention,22 25 reporting positive results for their primary outcome26 and industry-sponsored studies26 are more prone to fit the CONSORT extension for abstracts. We will also inspect the relationship between quality of reporting and study setting. Indeed, to conduct studies on interventions performed in the out-of-hospital environment is one of the main challenges posed by the emergency medicine to clinical researchers, and we think that this might further affect the quality of reporting compared with studies in the in-hospital setting.

We decided to include in this review simulation-based RCTs, if they will be needed to achieve the sample size. The CONSORT extension for healthcare simulation-based studies has recently been published.27 However, this extension refers to the CONSORT guidelines for full texts, and only one recommendation is given relating to abstracts: ‘(In abstract or key terms, the MESH or searchable keyword term must have the word ‘simulation’ or ‘simulated’)’. Therefore, the CONSORT extension for abstract is still the best tool for evaluating the quality of reporting of simulation-based studies.

To design the present study, we referred to a previously published protocol of a systematic survey on the reporting quality of abstracts of trials published in pain journals.12 With respect to the previous study, we decided to extend the survey to 10 journals instead of 5, and to consider only 15 of the 17 items of the checklist (as explained in the Outcomes section).

Ethical committee approval is not required for this review because we will only be dealing with secondary data.

The review results will be disseminated through publication in a peer-reviewed biomedical journal and abstract submission to conferences.

In conclusion, with the present review we will investigate the quality of reporting of abstracts of RCTs in emergency medicine journals, and the eventual improvement due to the publication of the CONSORT statement for abstract. Quality of reporting might be found to be inadequate; in that case, we will enlighten factors associated with better quality and try to suggest strategies for its improvement.

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.
  22. 22.
  23. 23.
  24. 24.
  25. 25.
  26. 26.
  27. 27.

Footnotes

  • Contributors FG is the guarantor of the review. FG, MM, LM, AW and LT drafted the manuscript. FG, MF, MGG, MM, LM, VS, GV, AW and LT contributed to the development of the selection criteria, the risk of bias assessment strategy and data extraction criteria. FG developed the search strategy. MM, LM and LT provided statistical expertise. AW provided expertise on emergency medicine literature. LM and LT provided expertise on quality of reporting. FG, MF, MGG, MM, LM, VS, GV, AW and LT read, provided feedback and approved the final manuscript.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.