Article Text
PDF
Abstract
Objectives To evaluate the reporting quality of randomised controlled trial (RCT) abstracts presented at a leading international conference in sleep medicine (the SLEEP Annual Meeting), and to investigate the association between potential predictors and the reporting quality of trial abstracts in this field.
Design Cross-sectional, research on research study.
Methods A handsearch of the 2016–2018 SLEEP Annual Meeting abstract books was carried out to identify abstracts describing RCTs. Quality of reporting was assessed with the original 17-item CONSORT for Abstracts checklist. Univariable and multivariable linear regression analyses were performed to identify significant predictors of reporting quality. In addition, risk ratios were used to analyse the adequate reporting rate of each quality item by type of intervention and funding status.
Primary and secondary outcome measures The overall quality score (OQS, range 0–17) in accordance with the CONSORT for Abstracts checklist (primary outcome), and the adequate reporting rate of each checklist item (secondary outcome).
Results A total of 176 RCT abstracts were included and assessed. The mean OQS was 5.53 (95% CI 5.30 to 5.76). Only three quality items (objective, conclusions and funding) were adequately reported in most abstracts (>75%). None of the abstracts adequately reported authors, randomisation or outcome in the results section. According to the multivariable analysis, pharmacological interventions (p=0.018) and funding from the industry (p=0.025) were significantly associated with better reporting quality.
Conclusions The reporting quality of RCT abstracts presented at SLEEP Annual Meetings was suboptimal. Pharmacological intervention and funding from industry were significant predictors of better reporting quality. Joint efforts by authors and conference committees are needed to enhance the reporting quality of RCT abstracts presented at sleep medicine conferences, and thereby reduce relevant research waste in this field.
- randomized controlled trial as topic
- data reporting
- sleep medicine specialty
- congresses as topic
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
Quality assessment was carried out using the original Consolidated Standards of Reporting Trials for Abstracts checklist, which facilitates the replication of this study and its inclusion in future meta-analyses.
Assessors were blinded to author names and author affiliations of included abstracts during the quality assessment process.
Only one major conference was included, thus, our findings may not be applicable for other conferences in sleep medicine.
Introduction
It is a shared obligation and responsibility for all researchers to ensure completeness, transparency and accuracy when reporting health research.1 However, a previous article indicated that about half of research in biomedicine were poorly reported and therefore unusable, resulting in an avoidable waste of billions of dollars of funding every year.2 Thus, research into the current reporting quality of medical research publications (articles, abstracts and protocols etc) and potential methods to reduce the relevant research waste are warranted.
Since the emergence of evidence-based medicine, high-quality randomised controlled trials (RCTs) have been considered as the highest level in the hierarchy of primary studies,3 and also the gold standard for the evaluation of healthcare interventions.4 The results of RCTs are often first presented at medical conferences in the form of abstracts, but only about one-third of these abstracts were subsequently published as full-length articles.5 6 Therefore, abstracts presented at conferences might be the only reports of the corresponding research that are available for potential readers. In addition, as systematic reviews often include such unpublished data,7 RCT abstracts presented at conferences can have significant impact on evidence-based clinical practice.
Considering the importance of RCT abstracts, an extension of the Consolidated Standards of Reporting Trials (CONSORT) statement was released in 2008. It summarised the essential items of RCTs that should be reported in journal and conference abstracts.8 However, in spite of the release of this statement (also known as the CONSORT for Abstracts guidelines), recent studies have suggested that the reporting quality of RCT abstracts presented at major dental, geriatric and urological conferences remained suboptimal.9–11
The SLEEP Annual Meeting is an international forum for the communication of novel developments in clinical sleep medicine as well as sleep and circadian science, covering many important topics such as the treatments for insomnia and sleep-related breathing disorders (SBD). However, to our knowledge, there has been no assessment of the reporting quality of abstracts presented at this major conference. Therefore, we carried out a study to assess the reporting quality of RCT abstracts presented at the 2016–2018 SLEEP Annual Meetings, and to identify potential factors associated with the abstract reporting quality in this field.
Methods
Retrieval and selection of abstracts
The abstract books of the 2016–2018 SLEEP Annual Meetings were obtained from the official website (www.sleepmeeting.org). Thereafter, one author (QS) handsearched these books to identify all abstracts describing RCTs, with the eligibility of each identified abstract verified by two experts (FH and HH). Based on the Cochrane criteria for RCT selection, we aimed to include healthcare-related studies that were interventional, conducted in human participants and randomly allocated participants to different intervention groups.12 Secondary analyses of RCTs and studies only included baseline data were excluded. All eligible abstracts were compiled into a Word document with all author names and affiliations removed to facilitate blinded quality assessment.13
Assessment of reporting quality
As determined a priori, the original CONSORT for Abstract guidelines and associated explanations, which included 17 quality items for conference abstracts8 was used to assess reporting quality. Before the assessment of all included abstracts, an internal pilot study was carried out to calibrate one author (QS) and an expert (FH) in iterative rounds of 20 randomly selected abstracts, until the inter-rater agreement was excellent or better (Cohen κ>0.75).14
Thereafter, one author (QS) assessed the reporting quality of the remained RCT abstracts, with all encountered issues and uncertainties resolved through a discussion with two experts (FH and HH).15 A score of ‘0’ (inadequate) or ‘1’ (adequate) was recorded according to the level of reporting for each checklist item. Thereafter, an overall quality score (OQS) was calculated by totalling the score of 17 items (score range: 0–17). Additionally, we recorded the reporting of 11 important subitems to provide as indicated in the explanatory document of CONSORT for Abstracts.8
Data extraction
After quality assessment was completed, two authors (QS and TZ) extracted the following information from each abstract, which could be potential predictors of abstract reporting quality: publication date (year), number of authors, geographical origin (first author), type of intervention (pharmacological vs non-pharmacological), number of centres (single centre vs multicentre), word count, research area (insomnia, SBD or others), sample size, as well as funding status (funded by industry vs not funded by industry). Research area was categorised in reference to the International Classification of Sleep Disorders, Third Edition.16 Another author (FH) verified the extracted data and all disagreements were resolved through discussions.
Statistical analyses
Cohen’s kappa coefficient was calculated to measure inter-rater agreement. Descriptive statistics were used to summarise the characteristics of all included abstracts, as well as the overall reporting quality and the adequate reporting rate of each checklist item/subitem. Additionally, linear regression analyses were performed to determine the association between potential predictors and abstract reporting quality (dependent variable, OQS). We conducted univariate analyses first, and then entered all significant predictors in the univariable analyses into multivariable modelling. Assessment of residuals did not indicate significant violation of normality. For the multivariable analysis, tolerance and variance inflation factor (VIF) were used to detect multicollinearity. We would have excluded any predictor from the final model, if it had a tolerance below 0.1 and/or a VIF above 10.17 Risk ratios (RRs) and their 95% CIs were calculated to illustrate differences in the adequate reporting rate of each item by types of intervention (pharmacological vs non-pharmacological) and funding status (funded by industry vs not funded by industry). For all analyses, statistical significance was defined as two-sided p<0.05.
Patient and public involvement
The present work was based on an assessment of presented RCT abstracts in the SLEEP Annual Meetings and did not include any participant, patient advisers or original data from patients. Patients were not involved in the design, recruitment and conduct of this study.
Results
Characteristics of included abstracts
A total of 176 RCT abstracts were identified and included (see online supplementary information for the flow chart). As demonstrated in table 1, most of the included abstracts were presented by authors from North America (76.7%), conducted in a single centre (90.9%), written in 301–350 words (71.0%) and describing non-pharmacological interventions (75.0%). Half of the included abstracts had 4–7 authors (50.0%). The majority of these abstracts were about insomnia (54.5%) and SBD (27.3%), and about one-fourth (26.1%) of them were funded by the industry.
Supplementary material 1
Characteristics of included abstracts
Pilot study
The internal pilot study was completed after round 2, in which excellent inter-rater agreement was reached (κ=0.841).
Reporting of general items
The adequate reporting rates of each CONSORT item and subitem among the 176 included abstracts are presented in table 2. Only 52 abstracts (29.5%) can be identified as randomised through their titles. Less than one-third (32.4%) of the abstracts described their trial design explicitly. Only six abstracts (3.4%) provided details about trial registration. Although authors’ names and their institutions were provided in the abstracts, no abstract reported contact details of the corresponding author. Nevertheless, a majority of the included abstracts (92.0%) reported their source of funding.
Reporting of each CONSORT checklist item and subitem in the included 176 abstracts
Reporting of trial methodology
Most abstracts adequately reported the CONSORT items objective (96.6%) and interventions (73.9%). However, none of the included abstracts described the methods used for sequence generation and allocation concealment. Therefore, no abstract reported the item randomisation adequately. In terms of information regarding trial participants, although eligible criteria were reported in 96.6% of included abstracts, only 9.1% described the settings in which participants were studied. Besides, only 28 abstracts (15.9%) clearly defined the primary outcome of the trial. Forty-five abstracts (25.6%) provided information about blinding, but only 14 abstracts (8.0%) specified who (eg, participants, caregivers and outcome assessors) were blinded.
Reporting of trial results
Seventy-nine RCT abstracts (44.9%) reported the number of participants randomised to each group, but only 24 abstracts (13.6%) reported the number of participants analysed in each group. Only 6.8% of included abstracts stated the adoption of intention-to-treat or per-protocol analysis. Although 28 abstracts (15.9%) defined the primary outcome in their Methods sections, no abstract provided all the details required for the primary outcome in the Results section, including result for each group, the estimated effect size and its precision. Additionally, adverse events or side effects were reported by only 12.5% of the included abstracts. Forty-one abstracts (23.3%) stated the trial status (eg, ongoing, preliminary analysis and interim analysis).
Reporting of trial conclusions
Almost all included abstracts (97.7%) adequately reported conclusions that were consistent with the trial results. However, only 10 RCT abstracts (5.7%) balanced the benefits and harms of interventions in their conclusions.
OQS and associated factors
The mean OQS of the included 176 abstracts was 5.53 (SD 0.12; 95% CI 5.30 to 5.76). Table 3 demonstrates the results of linear regression analyses. In univariable analyses, four factors were significantly associated with better reporting quality of included abstracts: more authors (p=0.019), pharmacological interventions (p<0.001), research regarding SBD (p=0.004) and funding from the industry (p<0.001).
Linear regression-derived coefficients (B) and 95% CIs, with overall quality score as the dependent variable for included abstracts
All these four predictors were entered into a multivariable model (p<0.001; R2=0.153, adjusted R2=0.128; constant=6.106). It was indicated that both pharmacological intervention (p=0.018) and funding from the industry (p=0.025) remained significant predictors of greater OQS. However, the low R2 (15.3%) and adjusted R2 (12.8%) values of the final model suggested that the influence of other factors not included in this model was likely to exist.
Reporting quality by type of intervention
According to the calculated RRs, five CONSORT items were reported significantly better in abstracts describing pharmacological interventions than those describing non-pharmacological interventions: title (RR 0.53; 95% CI 0.34 to 0.83), interventions (RR 0.78; 95% CI 0.67 to 0.91), objective (RR 0.95; 95% CI 0.92 to 0.99), outcome in methods section (RR 0.44; 95% CI 0.23 to 0.87) and harms (RR 0.10; 95% CI 0.04 to 0.25) (table 4).
Reporting quality of each CONSORT checklist item and subitem by type of intervention
Reporting quality by funding status
The adequate reporting rates of six CONSORT items were significantly higher in abstracts funded by industry: title (RR 0.41; 95% CI 0.27 to 0.63), trial design (RR 0.56; 95% CI 0.37 to 0.85), outcome in methods (RR 0.47; 95% CI 0.24 to 0.92), blinding (RR 0.27; 95% CI 0.10 to 0.72), harms (RR 0.17; 95% CI 0.07 to 0.38) and funding (RR 0.89; 95% CI 0.84 to 0.95) (table 5).
Reporting quality of each CONSORT checklist item and subitem by funding status
Discussion
Our study identified and evaluated 176 RCT abstracts presented at the 2016–2018 SLEEP Annual Meetings, whose overall reporting quality turned out to be suboptimal. Of the 17 CONSORT quality items, only three items (objective, conclusions and funding) were adequately reported in most abstracts (>75%). This pattern of reporting is slightly different from the findings of several previous studies regarding conference abstracts,9 18 in which intervention, objective and conclusions were well reported but the reporting of funding was poor. Such differences might result from the required structured format of SLEEP Meeting abstracts, which included the headings of ‘Introduction, Methods, Results, Conclusion and Support (if any)’. Almost all of the included SLEEP Meeting abstracts had a ‘Support’ section where authors reported the sources of financial support, which clearly demonstrated that instructions for authors can play an important role in abstracts reporting quality.
In the present study, none of the included abstracts provided sufficient details for corresponding author, randomisation and outcome in results section. Less than 10% of the included abstracts adequately presented participant, blinding and trial registration. For these items, the findings of two previous studies in dentistry9 and gerontology10 were generally in line with ours, which indicates that the inadequate reporting of corresponding author, participant, randomisation, outcome in results section and trial registration could be universal across medical specialties. However, compared with our results, similar studies in sports injury,19 burn20 and endourology21 found greater adequate reporting rate in participant, randomisation and outcome in results section. Such discrepancies might be attributed to different levels of awareness and usage of the CONSORT and CONSORT for Abstracts guidelines in different medical fields.
The importance of titles in the identification of RCTs has been shown in the Cochrane Highly Sensitive Search Strategy. When carrying out a systematic review, researchers need to search controlled vocabulary (eg, Medical Subject Headings) and free-text terms related to the word ‘random’ and specific trial designs (eg, parallel, cross-over) to retrieve RCTs.22 Furthermore, considering that more than half of conference abstracts do not reach full publication,5 6 the corresponding studies can only be searched by title and abstracts. Therefore, it is important for conference abstracts to mention randomisation in an explicit way, to ensure correct indexing and identification through common search strategies. In the index section of abstract books of the 2016–2018 SLEEP Meetings, only nine abstracts were labelled as RCTs. However, as shown in our study, there were actually 176 RCT abstracts presented during these meetings. This may be related to the fact that only 30% of the included abstracts mentioned randomisation in their titles, and such lack of identifiability could hinder the dissemination of sleep medicine trials in conferences as well as their translation into clinical practice.
Conference abstracts are more likely to report preliminary or interim analyses, and the number of participants randomised and analysed is likely to change during the course of a trial.23 Thus, the reporting of trial status is important for readers to critically appraise a trial. In addition, if a reader or systematic reviewer needs additional information about a trial, contacting the corresponding author or searching registration website could be the only viable methods to obtain the information. However, in our study, less than one-fourth provided details about recruitment, less than 4% of the abstracts reported authors and trial registration adequately.
It is impossible for a reviewer to appraise a study comprehensively if related information has not been provided. One previous study demonstrated that inadequate allocation concealment may lead to larger estimates of treatment effects.24 In our study, no abstract provided details about sequence generation and allocation concealment. It is, therefore, impossible to determine for the included abstracts whether there were any systematic differences between study groups at baseline (selection bias).
According to the Cochrane handbook,25 the main cause of performance bias was inadequate blinding of study participants and personnel, lack of sufficient blinding of outcome assessors may lead to detection bias. Thus, it is important to provide explicit details about blinding in the RCT. However, among the included abstracts of this study, only 8% of abstracts contained explicit description of blinding. Thirty-one abstracts used the terms ‘single blind’ or ‘double blind’, without specifying whether participants, caregivers or those assessing the outcomes were blinded. Such reporting deficiencies may be confusing for readers and systematic reviewers, and should therefore be avoided.
Large and unbalanced discrepancies in the number of participants randomised and the number analysed may be a hint of attrition bias. If the prespecified primary outcome was not reported, the trial might have a high risk of selective reporting bias.26 However, in the present study, only 14% of the included abstracts reported the number of participants analysed in each group, only 16% clearly defined their primary outcome measure. In other words, about 85% of the included abstracts did not provide enough information for a quick assessment of their attrition and selective reporting bias.
The applicability of findings of an abstract depends on explicit description of eligibility criteria and the settings of data collection. Although 97% of abstracts adequately reported eligible criteria for participants, less than 10% reported their study settings. This makes it difficult for readers to assess the external validity of a trial and to determine its applicability to their own setting.8 Besides, less than 3% of included abstracts reported estimated effect size and precision of the estimate (eg, 95% CI) for the specified primary outcome. Although p values can provide information about statistical significance, full evaluation of clinical interventions needs to take into account the effect size and precision.27
In this study, we explored the association between abstract reporting quality and nine factors. Most of these factors, including date of publication,12 28 number of authors,29 type of intervention,29 number of centres,28 word count,9 30 research area18 and funding status,31 had been previously identified as significant predictors of RCT abstract reporting quality. However, in our study, only two factors remained significant predictors in the multivariable analysis. First, abstracts of pharmacological trials were better reported than non-pharmacological trials. This is generally in keeping with the finding of several previous studied regarding the reporting of trial abstracts28 31 32 and full articles,27 but was in contradiction with a recent study regarding abstracts of HIV/AIDS-related RCTs.29 Second, funding from the industry was associated with better abstract reporting. A recent study on RCT abstracts in psychiatry observed the same association,31 while in some other studies source of funding was not a significant predictor of abstract reporting quality.32 To what extent can the above-mentioned differences be attributed to subject variations is still unknown. Further studies from the field of sleep medicine are needed to confirm our results and explore the relevant causes and implications.
The present study is not without limitations. First, our final model of multivariable regression can only explain 12.8%–15.3% of the variation of OQS. Other factors, such as involvement of statisticians, awareness of CONSORT for Abstracts among the reviewers might also influence abstract reporting quality.33 34 However, we were not able to obtain such information from SLEEP Meeting abstracts books. Second, the quality assessment was carried out by only one author. However, similar to previous studies in this area,15 before formal assessment a pilot study was carried out to calibrate the assessor (QS) and an expert (FH).9 12 35 In addition, all discrepancies and uncertainties in the assessment process were resolved through a discussion with two experts (FH and HH). Third, only one major conference was included in this study, therefore, our findings may not be applicable to other academic conferences in the field of sleep medicine. Nevertheless, to our knowledge, this was the first study to assess the reporting quality of RCT abstracts and to explore relevant predictors in the field of sleep medicine. Our findings should be helpful in reducing relevant avoidable research waste,2 and facilitate further studies into the reporting of sleep medicine research.
The SLEEP Annual Meeting is a major platform for disseminating the latest research in sleep medicine. Authors of meeting abstracts should take the responsibility of reporting their research adequately. In addition, we recommend that the organising committees of conferences in this field endorse the CONSORT for Abstracts guidelines in their online instructions for authors. Active implementation of the CONSORT for Abstracts guidelines and the adoption of a highly structured format could lead to significant improvements in the reporting of RCT abstracts.35 36
In summary, the reporting quality of RCT abstracts presented at SLEEP Annual Meetings was suboptimal. Pharmacological intervention and industry funding were found to be significantly associated with better reporting quality. There is a need for efforts by authors and conference committees to improve the reporting of RCT abstracts presented at conferences in sleep medicine.
References
Footnotes
FH and QS contributed equally.
Contributors Grant application: FH. Study conception: FH and QS. Study design: FH, QS, XC and HH. Abstract selection: FH, QS and HH. Data extraction: FH, QS and TZ. Data analysis: FH and QS. Data interpretation: FH, QS, TZ, XC and HH. Manuscript drafting: FH and QS. Critical revision of the manuscript: TZ, XC and HH. Approval of final version: all authors.
Funding This work was supported by the China Postdoctoral Science Foundation (No. 2018M630884) and the Fundamental Research Funds for the Central Universities (No. 2042018kf0070, Wuhan University).
Disclaimer The funding source had no involvement in the study design, collection, analysis, and interpretation of data, preparation of the manuscript, or in the decision to publish.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement The data may be obtained from the authors for academic purposes.
Patient consent for publication Not required.