Discussion

In the present study, 136 RCT abstracts concerning the treatment of AMD were identified and assessed. The overall quality of reporting based on the CONSORT for abstracts checklist criteria was with median 7 reported items out of 16 items poor. Only two CONSORT items (‘interventions’ and ‘objectives’) were adequately reported in most abstracts (>90%). However, no abstract provided complete information on outcomes as required. The main problem was found in the reporting of effect size and CI (for each trial arm/group as well as in total). Less than 25% of abstracts included sufficiently reported ‘recruitment’, ‘numbers analysed’, ‘conclusions’, ‘trial registration’ and ‘funding’. Information on ‘randomisation’ was available in 29% which also implies that reporting was not transparent. In the conclusions section, most abstracts (96%) reported conclusions consistent with the results but just a few of them (18%) addressed potential limitations of the study or noted whether additional studies were required due to different reasons.

In particular, none of the 136 abstracts presented sufficient reporting of result outcomes, which demonstrates the crucial need for improvement in this field by, for example, provision of explicit instructions for authors in terms of standard reporting formats. The non-reporting of the effect size and its CI for primary outcome has to be identified as the main problem.

Two further studies of Hua (0% pre-CONSORT and 2.3% post-CONSORT)21 and Chen (1%)10 presented similar results. Other studies showed better but also improvable reporting of outcomes (Bigna 25.2% pre and 42.5% post3; Can 5.6%–18.4% pre and 20.4%–38.8% post, depending on the journal16; Berwanger 62.3%11). From our point of view, the abstract is crucial, and therefore, the outcomes are essential in an abstract. Strict fulfilment of all criteria as indicated by the CONSORT statement for abstracts was therefore required by the authors: primary outcome result for each group and for primary outcome as well as effect size and CI reported appropriate for trials with binary outcome resp. with continuous outcome. In this respect, we have not allowed any scope for interpretation and the parallel reader evaluation followed a strict ‘no tolerance strategy’, which might explain the embarrassing result for this item in our investigation. In addition, in a randomised study with at least two different therapies, it should be possible to present an effect size between them. RCTs are expected to provide a high degree of evidence. However, 136 RCT abstracts did not contain sufficiently detailed data on the study outcome (effect size in combination with significance or confidence measure) in such explicit terms as required by the CONSORT statement. This lack of reporting quality certainly means a more crucial loss in transparency and reproducibility than, for example, suboptimal fulfilment of rather editorial recommendations. In summary, physicians conducting clinical research are strongly advised to involve a biometrician in the formulation of their results.

AMD is a common eye disease with an increasing need for research in the last years. However, the reporting quality of RCT abstracts concerning AMD therapy has not been assessed until we conducted our study. Our findings were disillusioning but consistent with previous studies assessing the quality of reporting of journal RCT abstracts concerning different diseases: Bigna,3 for example, found in 2016 in 312 abstracts concerning HIV median 6 reported items pre-CONSORT and median 7 items post-CONSORT, and stated that this suboptimal improvement was associated with the journal’s high IF, large number of authors and non-pharmacological/vaccine intervention in the trial. Also in 2016, Chhapola found in 891 abstracts in three leading paediatrics journals18 median 7 to 8 items (depending on the journal) pre-CONSORT and median 7 to 10 items post-CONSORT. The authors assumed that the reporting quality may be affected by constraints of space and word limit as well as structured versus unstructured abstracts. Ghimire23 investigated 956 phase III oncology trial abstracts published pre-CONSORT and post-CONSORT for abstracts and found median 8.2 (95% CI 8.0 to 8.3) and 9.9 (95% CI 9.7 to 10.2) in the pre-CONSORT and post-CONSORT periods, respectively. A high IF and the journal of publication were independent factors significantly associated with higher reporting quality on their multivariate analysis. In our study, the multivariate analysis showed that only the effective number of words had an influence on the quality of abstracts. Hopewell5 showed that with a word limit of 250 to 300 words, the checklist items could be easily incorporated. This was also reflected in our investigation: Abstracts with >250 words showed better reporting than abstracts with 250 words or less. It is conceivable that with a word limit of maximum 250 words medical authors focus on medical aspects and less on methodological aspects such as randomisation, blindness, trial status or a correct outcome presentation as recommended by CONSORT. On the other hand, Berwanger11 impressively demonstrated the possibility of expressing maximum information about methodological aspects in just a few words.

In this study, the IF of journals or structuring of abstracts was not found associated with the quality of reporting in AMD abstracts as was found in other studies. One reason for this could be the inclusion of 36 different journals, in which RCTs concerning AMD were published between 2004 and 2013. We had determined the IF for each journal depending on the year of publication: Therefore, we had a great variability in the IFs ranging from 0 to 52.414 and a median IF of 3.125. Most other studies relate to abstracts in preselected journals10–12 14 16–18 that specialised in the particular disease. These studies refer to a handful of preselected journals with only a few different IFs, which possibly explains this influence in contrast to our study. Furthermore, only eight journals with nine abstracts contained unstructured abstracts in this investigation. This could be the reason why an influence of abstract structure was not demonstrated here. The present investigation also failed to demonstrate an improvement of quality after publication of the CONSORT extension for abstracts in 2008. This result is similar to several previous surveys,18–20 26 whereas some other studies using a simple pre–post comparison showed a slight improvement.3 14 16 23 Due to the inclusion of all journals without preselection and over a long period of publication, the authors were not able to identify whether the respective journal made a reference to the CONSORT statement at the publication time of each RCT, and especially whether the journal contained a reference to the CONSORT extension for abstracts. This is certainly a limitation of this investigation which, however, was not preventable due to a lack of (online) information.

Our investigation shows high process validity from the methodological perspective, as the identification of RCTs, eligibility decision and data extraction were performed by two independent readers, both having several years of experience in the publication and review of clinical trials. In addition, all items were clearly parameterised before the investigation by means of a standard operating procedure on the CONSORT items’ evaluation, and then discussed and mutually validated by means of these procedures. For this purpose, a pilot study in a different indication (cataract surgery) was carried out25 in order to identify possible weaknesses and interferences in advance. Nevertheless, kappa values for some items only showed moderate interobserver reliability. This is partly explained by the fact that it was difficult—based only on the written information—to decide whether abstracts contained information on ‘numbers randomised’ or ‘numbers analysed’. Only in a few cases, both information were presented explicitly. Therefore, the ‘moderate’ interobserver reliability in these items was not only based on different ratings, but also on non-transparent and unclear reporting in the respective abstract.

We did not take a random sample of RCTs, but rather used systematic review methodology to identify eligible RCT publication abstracts for our purpose, and then decided to perform a full cross-sectional census evaluation on these abstracts. The use of a sensitive PubMed research strategy and the data collection from 2004 to 2013 led to the inclusion of all published RCTs in AMD with no selection of special journals. This represents the whole body of scientific evidence in the field of AMD until the end of the publishing period under consideration—and actually, as we presume—until today. Of course, we are aware that the limitation of publication period (2004–2013) must be viewed critically: We sought to depict a 10-year period in our study, which covered both the period before and after the publication date of the CONSORT guidelines for abstracts (‘CONSORT for abstracts’ was published in January 20085). Since we assumed that these recommendations did not find immediate uptake in the year of publication, we defined 2004–2008 as a 5 years pre-CONSORT and 2009–2013 as a sufficiently long-term post-CONSORT 5-year period. It must be admitted that even after a sufficient uptake time after the ‘CONSORT for abstracts’ being published, the publication date is only a proxy for the actual uptake of the recommendation. For example, a publication first submitted in 2007—thereby not underlying the recommendations’ content—might have been published in 2009 after a 2-year review and editing process. As a consequence, a 2009 ‘post-CONSORT’ publication would have been misclassified as actually being written in the pre-CONSORT period. However, during the literature research on this topic, we found that various authors had already examined the pre–post comparison.3 14 18 21 For this reason, we included the 2008 cut-point via the binary variable ‘publication date before/after 2008’ in the Poisson regression model. Consequently, the pre–post comparison was thought to be a possible factor influencing the quality of the abstracts. This, however, was not confirmed, which could possibly be due to the selected time periods. A further investigation covering the next 5-year period (2014–2018) could possibly show a trend towards CONSORT uptake.

An undeniable limitation is that both readers were not blinded to the journal and publication period so the possibility of assessor bias cannot be ruled out entirely. Furthermore, as in other studies,3 10 11 14 16–18 all 16 CONSORT items were equally weighted and only the presence of an item was rated as ‘yes/no’. However, it can be noted that certainly not all items have the same impact on the transparency of the reporting process, and some items, for example, on explicit result presentation, are certainly more important than others such as details on funding. Nevertheless, all items of the CONSORT checklist should be reported in an abstract and a median number of reported items of 7 (95% CI 7 to 8) must be considered as poor reporting quality.

A further limitation is that this study was carried out without preparing a study protocol in advance, as its evaluation was based on the previously tried and tested procedure of the pilot study.25 Nevertheless, there remains a risk of selective reporting.

In summary, the reporting quality of RCT abstracts on AMD healthcare showed a considerable potential for improvement to meet the CONSORT abstract reporting recommendations. The reporting of the outcomes in particular was disillusioning, even though CONSORT provide detailed information in its checklist explanations on how these results should be presented. Since these further explanations are very detailed, it seems as if these notes are hardly being read, and therefore lack the degree of awareness they deserve. One possibility would be to provide elaborate examples/templates by means of commented ‘best-practice abstracts’ as supplementary material to a journal’s ‘instructions for authors’. These abstracts would be held in the authors’ terminology and clinical context, and thereby would become much more accessible than easier to imitate for the clinical author than the transfer of recommendation explanations.

Furthermore, word count limits for abstracts of 250 were identified as a significant determinant of the overall abstract reporting quality. These word count limits could possibly be neglected if annotated best-practice abstracts were made available, or else they could be modified according to the requirements of the respective journals in order to improve quality. To reduce the amount of research waste,27 improvements in abstract quality are urgently needed. In particular, the correct statement of the exact effect size is of utmost importance in an abstract, in order to be able to transfer a statement into everyday clinical practice.