Study design

This study will be a systematic survey to assess quality of reporting. The Medline database will be searched for RCTs published in the years 2005–2007 and 2014–2015 in the top 10 exclusive emergency medicine journals, ranked on their impact factor according to the Journal Citation Report 2015 provided by Thomson Reuters16 (table 1). At first, we decided to take a 2-year period before and after CONSORT for abstracts (2006–2007 and 2014–2015). After a scoping search, we realised that probably the optimal time was a 3-year period before CONSORT for abstracts (2005–2007) to be able to reach the sample size. Journals created after 2006 will be excluded from the review, otherwise the period before the publication of the CONSORT extension for abstract (ie, before 2008) would not be sufficiently represented for those journals, not allowing a correct before–after comparison. Journals coming next in the Journal Citation Report list will be included to achieve the number of 10.

The search strategy will include the Cochrane highly sensitive search strategy for identifying randomised trials in Medline (sensitivity-maximising and precision-maximising version, 2008 revision),3 journal names and limits set for the specific time periods of interest (years of publication 2005–2007 and 2014–2015). The search strategy is reported in detail in box 1.

Inclusion criteria

To be included in the review, the abstract must be a report of primary outcome(s) of an RCT and involves human subjects. Simulation based-studies (eg, study performed using manikins) will be considered for inclusion only if needed to achieve the sample size.

Exclusion criteria

Studies will be excluded if the abstract is not available, they are published only as abstracts (eg, conference proceedings), still recruiting, reporting on outcomes other than primary outcome or duplicate publications.

A flow diagram showing the study screening and selection procedures is reported in figure 1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Flow diagram showing study selection procedure.

Outcomes

The overall quality of reporting (number of items reported) according to the CONSORT extension and the compliance with its individual items will serve as outcomes. The following checklist items will not be included: ‘Authors’ (contact details for the corresponding author) and ‘Recruitment’ (trial status), as these items are specific to conference abstracts.5 Thus we will eventually test the compliance with 15 items.

Explanatory variables

Timing of publication (pre-CONSORT vs post-CONSORT publication) will be the explanatory variable for the primary objective. Explanatory variables for the secondary objective will be the following: CONSORT endorsement by the journal (defined as the presence of the request to comply with the CONSORT statement, among the instructions for authors/authors guidelines section in the journal website), number of centres involved in the study (single-centre vs multicentre), study sample size (≤100 vs >100), type of intervention (pharmacological vs non-pharmacological), significance of results for primary outcome (significant vs non-significant), source of funding (industry-funded vs non-industry-funded) and study setting (out-of-hospital vs in-hospital).

Sample size calculation

The primary objective of this systematic survey is to compare the mean number of reported items in pre-publication versus postpublication of the CONSORT extension for abstracts based on the corresponding checklist.4 A review assessing the quality of abstracts in general medical journals found a mean difference in the number of items reported between after and before the publication of the CONSORT statement for abstracts of 3.05 (95% CI 2.44 to 3.65, p<0.001), going from 9.06 (SD: 2.15) in 2007 to 12.11 (SD: 2.22) in 2012.7 If we expected exactly the same results, the sample size required to show such a difference with a significance of 0.05 and a power of 0.8, considering an SD of about 2, would be eight in each study period (pre-CONSORT vs post-CONSORT). The sample size would be smaller if we consider that it is likely that from 2012 to 2015 a further improvement in the quality of reporting occurred. For example, assuming a difference of four, the sample size would be four in each study period. On the other hand, given that the generalisability of these results to studies in emergency medicine is uncertain, we hypothesise also a worse scenario in which we expect a mean difference of 1.5 items correctly reported between the two periods. The sample size required to show this effect size would be 28 in each study period. Because we are looking at the top 10 journals and so we are including articles published in the same journal, we adjusted the raw estimate in both the scenarios for a possible cluster effect multiplying by a factor of 1.42, obtained assuming an intraclass correlation coefficient of 0.03 and an average number of articles per journal of 15.17 Within the boundaries of the two scenarios, the sample size would range from 6 to 40 articles per period. Furthermore, the effect of the period will be explored in a multivariable model including also the following seven variables: endorsement of CONSORT statement by the journal, number of centres involved in the RCT, type of intervention, sample size, results of the trial, funding status and study setting. Thus, we further adjusted the sample size upward by adding five articles for each variable (35). Hence, we will be able to detect a mean difference as low as 1.5 items correctly reported as long as we find a number of eligible articles of at least 58 per period (116 in total). A smaller number of eligible articles would allow us to detect only larger effect sizes. If we find a number of articles higher than 58 per period, the articles will be randomly selected. If possible, we will select the same number of articles from each journal (6±1 per period).

Data extraction and synthesis

For each article, two independent reviewers will screen eligibility criteria and will extract data on the reporting of the 15 items of the CONSORT statement for abstracts that we have selected and data regarding the explanatory variables. When needed, full texts will be reviewed to retrieve those data. The reviewers will use the CONSORT elaboration and explanation document18 as a guide. Disagreements between reviewers will be resolved via discussion or, when necessary, with the intervention of a third reviewer. In order to increase accuracy and consistency between reviewers, an initial trial run involving 10 articles per reviewer will be undertaken.

Statistical analyses

The characteristics of the included articles will be synthesised using descriptive statistics such as mean (SD) or median (first quartile, third quartile) for continuous variables and count (frequency) for categorical variables. We will describe the frequency with which each item is reported and the mean number of reported items by period of publication (pre-CONSORT vs post-CONSORT). The primary outcome (mean difference of items reported) will be estimated first using a two-sample t-test (unadjusted mean difference) and then through generalised estimation equations (GEEs) in order to adjust for the prespecified explanatory variables.19 The mean differences and adjusted means will be reported with 95% CIs and p values. The same multivariable GEEs will be used to explore the effect of the explanatory variables other than the period of publication.

In order to strengthen our confidence in the results, the effect of the period of publication will be explored through two additional approaches. First, we will look at the compliance with each of the 16 items of the CONSORT statement for abstracts for years 2005–2007 versus 2014–2015 as expressed by the binary outcome item reported or not reported. We will perform both unadjusted analyses using χ² tests and adjusted analyses using GEEs and assuming binomial distributions and unstructured correlation matrices. The results of the GEEs will be reported as adjusted ORs, with 95% CIs and p values. Then, the incidence rate ratios (IRRs) for reporting items for the period 2014–2015 compared with the period 2005–2007 will be estimated using GEEs based on a Poisson distribution and an unstructured correlation matrix. The results will be reported as adjusted IRRs, with 95% CIs and p values. The criterion for statistical significance will be set at alpha=0.05.

In the multivariable analyses, adjustments will be made for (1) CONSORT endorsement by the journal, (2) number of centres involved in the study (single-centre vs multicentre), (3) study sample size (≤100 vs >100), (4) type of intervention (pharmacological vs non-pharmacological), (5) significance of results for primary outcome (significant vs non-significant), (6) source of funding (industry-funded vs non-industry-funded) and (7) study setting (out-of-hospital vs in-hospital), with journal as a grouping factor to adjust for potential clustering or similarity in articles published in the same journal. Descriptive data will be presented as counts and percentages. Between-reviewer agreements will be measured using the kappa statistic.20 Data will be analysed using STATA/SE V.12.0.

Discussion and dissemination

Our systematic survey is aimed at assessing the quality of reporting of abstracts of RCTs published in emergency medicine journals, and the influence on that of the publication of the CONSORT extension for abstracts. Adequate reporting of abstract of RCTs would improve transparency, correct assessment and interpretation of trials, accurate indexing, retrieval, and appropriate inclusion in systematic reviews. We expect the quality of reporting of articles published after the publication of the CONSORT extension for abstract to be better than before. Besides the publication of the CONSORT extension for abstracts, other variables can affect the quality of reporting. That is why we will explore the role of the following: journal’s endorsement of the CONSORT statement, number of study centres, sample size, type of intervention, significance of the result of primary outcome, funding of the study and setting (out-of-hospital vs in-hospital). Based on previous experiences, RCTs from journals endorsing CONSORT,21 22 multicentre studies,22 23 studies with larger sample size,22 24 involving pharmacological intervention,22 25 reporting positive results for their primary outcome26 and industry-sponsored studies26 are more prone to fit the CONSORT extension for abstracts. We will also inspect the relationship between quality of reporting and study setting. Indeed, to conduct studies on interventions performed in the out-of-hospital environment is one of the main challenges posed by the emergency medicine to clinical researchers, and we think that this might further affect the quality of reporting compared with studies in the in-hospital setting.

We decided to include in this review simulation-based RCTs, if they will be needed to achieve the sample size. The CONSORT extension for healthcare simulation-based studies has recently been published.27 However, this extension refers to the CONSORT guidelines for full texts, and only one recommendation is given relating to abstracts: ‘(In abstract or key terms, the MESH or searchable keyword term must have the word ‘simulation’ or ‘simulated’)’. Therefore, the CONSORT extension for abstract is still the best tool for evaluating the quality of reporting of simulation-based studies.

To design the present study, we referred to a previously published protocol of a systematic survey on the reporting quality of abstracts of trials published in pain journals.12 With respect to the previous study, we decided to extend the survey to 10 journals instead of 5, and to consider only 15 of the 17 items of the checklist (as explained in the Outcomes section).

Ethical committee approval is not required for this review because we will only be dealing with secondary data.

The review results will be disseminated through publication in a peer-reviewed biomedical journal and abstract submission to conferences.

In conclusion, with the present review we will investigate the quality of reporting of abstracts of RCTs in emergency medicine journals, and the eventual improvement due to the publication of the CONSORT statement for abstract. Quality of reporting might be found to be inadequate; in that case, we will enlighten factors associated with better quality and try to suggest strategies for its improvement.