Article Text
PDF
Abstract
Introduction The STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) Statement was developed in response to inadequate reporting of observational studies. In recent years, several extensions to STROBE have been created to provide more nuanced field-specific guidance for authors. The content and the prevalence of extension endorsement have not yet been assessed. Accordingly, there are two aims: (1) to classify changes made in the extensions to identify strengths and weaknesses of the original STROBE checklist and (2) to determine the prevalence and typology of endorsement by journals in fields related to extensions.
Methods and analysis Two independent researchers will assess additions in each extension. Additions will be coded as ‘field specific’ (FS) or ‘not field specific’ (NFS). FS is defined as particularly relevant information for a single field and guidance provided generally cannot be extrapolated beyond that field. NFS is defined as information that reflects epidemiological or methodological tenets and can be generalised to most, if not all, types of observational research studies. Intraclass correlation will be calculated to measure reviewers’ concordance. On disagreement, consensus will be sought. Individual additions will be grouped by STROBE checklist items to identify the frequency and distribution of changes.
Journals in fields related to extensions will be identified through National Library of Medicine PubMed Broad Subject Terms, screened for eligibility and further distilled via Ovid MEDLINE® search strategies for observational studies. Text describing endorsement will be extracted from each journal’s website. A classification scheme will be created for endorsement types and the prevalence of endorsement will be estimated. Analyses will use NVivo V.11 and SAS University Edition.
Ethics and dissemination This study does not require ethical approval as it does not involve human participants. This study has been preregistered on Open Science Framework.
- reporting guidelines
- strobe
- observational studies
- information dissemination/methods
- bibliometrics
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
Our systematic approach to qualitatively assess the content of the additions made in the STrengthening the Reporting of OBservational studies in Epidemiology extensions provides a comprehensive overview of the types of changes made and can identify redundancies and problem areas.
Our method involves standardised search strategies in Ovid Medline, designed to capture a representative sample and circumvent issues of subjectivity in the identification of eligible journals.
This study will create an open source corpus of recent observational studies spanning seven fields which future researchers can use to assess completeness of reporting or other topics of interest.
The bibliometric aspect of this study only focuses on seven extensions and fields so results are not generalisable to other studies.
Introduction
The STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) Statement was developed in 2007 in response to the pervasiveness of inadequate reporting of observational studies. STROBE provides a checklist of items that serve as a reference for how to report sufficient information for observational research involving cohort, case–control and cross-sectional studies.1 The guidelines have been endorsed by the International Committee of Medical Journal Editors (ICMJE) and the accompanying checklist is sometimes explicitly used as a requirement for manuscript submission.2 However, there is no standard method of endorsement by journals and little is known about the most effective ways to apply the guidelines in practice.3–5
Regarding the reporting of clinical trials, requiring a completed Consolidated Standards of Reporting Trials (CONSORT) checklist on submission of a manuscript has been shown to lead to improvements in reporting.6 However, some journals do not want to take responsibility for guideline enforcement and many overlook non-adherence to guidelines; editors have expressed beliefs that their journal’s current policies are adequate or that they fear losing authors to other journals that have less strict requirements for publication.7–9 Editors may also be unaware of the existence of guidelines, as demonstrated by low endorsement rates by journals in dentistry,10 veterinary medicine7 and urology.11 On the other hand, the evidence for the endorsement of STROBE is also mixed. Endorsement was not shown to be associated with better reporting for items related to confounding, regardless of strength.12
Several field-specific extensions to STROBE have been designed in recent years in an effort to promote complete reporting, provide more nuanced guidance for authors and perhaps address editor’s concerns that STROBE is not focused enough for their journal. Extensions for other reporting guidelines are common; however, the creation of extensions for STROBE seems to outpace those for other reporting guidelines such as the CONSORT.13 Since the publication of STROBE in 2007, 13 extensions have been published and indexed by the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network, an international collaboration that promotes transparent and accurate reporting and indexes reporting guidelines.14 In contrast, CONSORT was first published in 1996, updated in 2001 and further revised in 2010, yet only 17 extensions have been published during that period.15 The reason behind the difference in the pacing of publications of extensions is unclear. Perhaps the concept of field-specific extensions to reporting guidelines was pioneered by CONSORT, thus making the idea more commonplace for subsequent reporting guidelines. Alternatively, the complexity of the types of observational research studies may require more guidance due to the wide variety of methods employed in observational studies. Regardless of the reasoning, it is evident that authors are still perceiving a need to provide more guidance on how to report information about their studies. However, until now, many of these initiatives have not been evaluated.
Extensions to STROBE offer a potential new avenue for promoting more complete reporting but their use has been largely unassessed and, similar to STROBE, they may face implementation and usage problems.3 7 Being intended as general guidelines for observational studies, STROBE should include necessary information that is sufficient to most observational studies. For some fields, however, STROBE guidelines may not be sufficient due to specific requirements within the field. This gap is then covered by an extension for that field. However, when extensions include non-specific guidance that can be extrapolated to most observational studies (eg, details about participants, settings, confounders, follow-up, biases or any other general epidemiological constructs), it suggests potential deficiencies in STROBE checklist. If the content is already in STROBE, extension authors may have thought that it was not clearly communicated, or that it is necessary to include it in the checklist instead of being only in the explanation and elaboration document. While, if the content is not already in STROBE, extension authors may have identified a gap or insufficiency which should be considered as an addendum to STROBE. Therefore, by identifying non-specific or redundant guidance suggested in the STROBE extensions, we will be able to identify perceived gaps and deficiencies in the current STROBE checklist and potentially reduce future waste in the process of extension creation.
A perceived lack of confidence in reporting guidelines can impact journal editors’ willingness to endorse reporting guidelines. Currently, it is unclear if and how journals are encouraging or requiring authors to use STROBE extensions. As journals are key players influencing the use and uptake of extensions, the prevalence and typology of extension endorsement is needed to understand the variety of methods employed to encourage transparent reporting. Data collected from this study can later be used as the groundwork for an evaluation of the impact of endorsement on the completeness of reporting.
Aims
The objectives of this study are twofold. First, to qualitatively assess and classify the changes made in the extensions to help to identify the strengths and weaknesses of the original STROBE checklist; this will identify potential problem areas or deficiencies conveyed in extension additions. Second, we will estimate the prevalence of endorsement in journals that publish observational studies from extension-related fields and create an endorsement typology to provide a finer detailed view of the promotion of the STROBE extensions.
Methods and analysis
Qualitative assessment and analysis
The main focus of this phase will be on coding the additions that are made in each extension. Coded additions will help to identify the strengths, weaknesses and redundancies conveyed in the STROBE extensions to provide guidance for modifications to the original STROBE checklist and to identify target areas for future educational interventions.
We will assess the content of 13 STROBE extensions which were identified through the EQUATOR Network website as well as through a PubMed search for STROBE-related publications. Two independent reviewers (DH, MKS) will code the additions made in each STROBE extension; disagreement will be resolved by consensus. Each subitem on an extension that is attached to a STROBE checklist item will be coded individually by the relevant content area (eg, item five subitem additions a, b and c will be counted and coded as three separate items). Each subitem will also be coded as ‘field specific’ (FS) or ‘not field specific’ (NFS). FS is defined as information that is particularly relevant for a single field and guidance provided cannot be generalised beyond that particular extension’s field. Items which note phrases such as ‘including,’ ‘specifically,’ ‘for example’ and ‘eg,’ followed by a field-specific example, generally are considered to be field specific as these items are adding additional information specific to a certain topic area. NFS is defined as information that reflects general epidemiological or methodological tenets and can be extrapolated to most, if not all, types of observational research studies.
For the subjective assessments of the field-specific or not field-specific nature of the additions (rated as binary yes or no), intraclass correlation (ICC) will be used to assess the inter-rater reliability (IRR). The ICC for the two raters will be calculated for ratings across all 13 extensions that involve the subjective assessment of an item as FS or not. This method was chosen because ICC does not take an all-or-nothing approach to agreement but rather it ‘incorporates the magnitude of disagreement to compute IRR estimates’.16 Descriptive statistics such as counts, means and percentages will be given.
Endorsement survey
Eligibility criteria
Extensions to the STROBE guidelines were identified through the EQUATOR Network website as well as through a search on PubMed. Extensions are eligible for assessment if at least 1 year has passed since publication as this allows for some time for endorsement and implementation. In the case of multiple publications of an extension, the earliest publication/availability date will be used to determine eligibility. As of 1 March 2017, eligible extensions are detailed in table 1, while ineligible extensions are detailed in table 2.
Extensions eligible for assessment
Extensions not eligible for assessment
Identification of journals
Journals in fields related to extensions will be identified using the National Library of Medicine (NLM) catalogue which contains, among other things, ‘biomedical and health-related life sciences journals’ indexed in Medline. As of March 2017, there are over 5600 journals indexed.17 This database was chosen for two primary reasons: (1) broad subject terms are used which allows for easy identification and segmentation of research fields for journals and topic areas for articles and (2) the segmentation of other search engines, namely Clarivate Analytics Web of Science Journal List,18 did not clearly align with extension fields and would result in more overwhelming searches with less certainty that potentially eligible journals would be identified.
Journals will be identified using the following search string in the NLM catalogue: pubmed (‘Broad subject terms’). If an extension reports search terms in their publication, these will be considered as a starting point. All search strategies were developed in collaboration with a medical librarian. Further details listing the individual broad subject terms used for each extension are detailed in table 3.
Broad subject terms
Screening
Journals will be manually screened to confirm that they publish in English, are in a relevant format (eg, not a textbook, magazine, etc) and are currently publishing. From the remaining list of journals that are indexed in Medline, search strategies will be used to identify observational studies in the relevant topic areas (see online supplementary file 1). The filter for observational studies is a combination of a study design search filter for cohort and case–control studies by BMJ Evidence Centre information specialists, Fraser et al’s work on identifying observational studies in surgical interventions and consultations with a medical librarian.19 20
Supplementary file 1
From the remaining list of journals that publish observational studies, FS search strategies (detailed in online supplementary file 1) will be used. Extensions were used as a starting point and extant systematic reviews provided additional guidance, particularly for RECORD and STROBE—AntiMicrobial Stewardship (STROBE-AMS).21 22 In the case of European League Against Rheumatism, a combination approach will not be used as this is the only extension where the broad subject term is the exact focus of the extension; the search strategy for observational studies will still be used.
The results of the Ovid Medline FS and observational search strategies will be compared with the list of journals that the search was run on to determine inclusion and exclusion. This combination approach will be used for several reasons. First, journal information from NLM is given in more structured manner and allows for easy matching between sets with overlapping Broad Subject Terms. For example, both STROBE-AMS and Strengthening the Reporting of Molecular Epidemiology for Infectious Diseases (STROME-ID) use the term ‘anti-infective agents’ while both STROBE-ME and STROME-ID use ‘molecular biology.’ This approach is also less resource intensive and allows us to more easily identify how many journals in each field publish observational studies, thus establishing the extent and importance of the issue.
Data extraction
Eligible journals and their websites will be searched exhaustively for any mention of STROBE extensions in their instructions for authors, guidelines for reviewers, other guidance documents or ethical policies. Data will be extracted by the first author (MKS). To inspect reliability, another researcher (DH) will extract data from 10% of the sample and agreement will be calculated. Primary data sources (ie, website pages) will be downloaded in pdf format and relevant text describing guideline endorsement will be extracted and coded into a standard data extraction sheet in Excel. Although STROBE and its extensions are the main focus of this investigation, we will also collect information about endorsement of other common guidelines such as CONSORT, Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), ICMJE’s Recommendations for the Conduct, Reporting, Editing and Publication of Scholarly work in Medical Journals and mentions of organisations like EQUATOR and Committee on Publication Ethics.13 15 23–25 This information will be gathered to see if journals that endorse other reporting guidelines or ethical reporting guidance are more likely to endorse STROBE or an extension.
Altman and Hopewell’s classification schema will be used as a starting point for the development of a typology of endorsement for STROBE and extensions.6 26 27 The initial approach will be to codify endorsements into several categories of ranging from active, passive and not endorsing. Some examples include a requirement of a completed checklist with manuscript submission (eg, active), a suggestion that authors ‘should’ reference or follow a specific guideline (eg, passive strong), a vague suggestion that author should adhere to reporting guidelines (eg, passive moderate), a vague suggestion that authors should adhere to certain standards which include reference to reporting guidelines (eg, passive weak) or no explicit mention at all (eg, not endorsing).
In addition to information regarding support for STROBE and its extensions, general information about the journal such as impact factor, publisher and contact information for the editorial offices will be collected. For the purposes of future analyses focused on completeness of reporting, it will also be noted if journals have recently launched and have not been publishing for at least 2 years prior to the publication of its related extension; this will ensure the ability to establish baseline data on the completeness of reporting. For example, STREGA was published in 2009, therefore journals must have begun publishing by 2007 to be included in latter assessments.
As publishers often provide additional resources for authors, we will collect information from the websites of publishers about their methods of endorsement. Endorsement from publishers will be considered to be indirect methods of support as they require significant effort on the part of the user seeking the information. Information communicated directly through the journal’s website will be considered to be direct if it is supplied in immediately available resources to authors.
Statistical analyses
Endorsement, types of endorsement and journal characteristics (eg, Impact factor, publisher) will be expressed using descriptive statistics such as counts, means/medians and percentages. For analyses comparing two binary variables (ie, endorsement of extensions and endorsement of other reporting guidelines), unadjusted ORs and their associated 95% CIs will be conducted. Differences in impact factors between endorsing and non-endorsing journals will be assessed with the Wilcoxon test of ranks, equivalent to the c-stat, c-index or area under the receiver operating characteristic curve. All CIs will be provided at the two-sided 95% level.
Discussion
An evaluation of the extensions provides a deeper understanding of content areas that are adequately detailed or in need of elaboration. By identifying the content areas that authors have difficulties with, the groundwork will be laid for an assessment into how authors currently use and understand STROBE and what difficulties they encounter with its implementation. This study will provide us with potential hypotheses for future survey for authors, focused both on the perceived sufficiency of STROBE and the extensions as this could be a barrier to use. For example, if we find non-specific additions in parts of STROBE, we may focus on those parts when inquiring authors’ opinions about adequacy of STROBE. The qualitative assessment will also allow us to identify key areas (eg, particular sections of the methods, results, conclusion) that may be commonly misunderstood to specifically probe authors about these points.
Results from this study will also provide estimates of the frequency and typology of endorsement. This dataset will allow journals to be targeted to promote guideline usage and will establish a groundwork for follow-up studies on attitudes related to endorsement of STROBE and its extensions. Perhaps most importantly, this study will provide the foundation for assessing the impact that endorsement has on the completeness of reporting. The data collected through this study will generate important insights for the design of future studies such as feasibility or pilot studies to estimate the effects of endorsement. Perceived lack of tangible benefit due to a weak evidence base can be a major barrier to guideline use. Testing a relationship between endorsement and an increase in completeness of reporting can provide the much-needed data to address sceptic’s concerns about the tangible value of supporting STROBE and its extensions.
This study will solidify the scope of the problem of insufficient support and use of STROBE extensions, detail variability in endorsement typology and establish data for future studies focused on the effects of endorsement on completeness of reporting and attitudes towards STROBE and its extensions.
Acknowledgments
The authors would like to acknowledge the Methods in Research on Research (MiRoR) consortium for their support and guidance.
References
Footnotes
Contributors All authors have made substantive intellectual contributions to the development of this protocol. MKS conceptualised the study and led the writing of the manuscript. DH led the supervision of the manuscript preparation. MKS and AU developed the search strategies. All authors provided detailed comments on earlier drafts and approved the final manuscript.
Funding This work was supported by the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 676207.
Competing interests EW is a Fellow of the UK EQUATOR Centre. This is an unpaid position but she has been paid to run training courses associated with the EQUATOR Network, and by other organisations, which promote the use of reporting guidelines such as STROBE. The EQUATOR Network is also a member of the Methods in Research on Research Network, which MKS, DH, EW, LG and EC are members. MKS has a placement with the EQUATOR Network as part of her doctoral studies. The other authors declare that they have no competing interests.
Provenance and peer review Not commissioned; externally peer reviewed.