Article Text
PDF
Abstract
Objectives To explore the accuracy of application of the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) tool in epidemiological studies focused on the evaluation of the role of antibiotics in selecting resistance, and to derive and test an extension of STROBE to improve the suitability of the tool in evaluating the quality of reporting in these area.
Methods A three-step study was performed. First, a systematic review of the literature analysing the association between antimicrobial exposure and acquisition of methicillin-resistant Staphylococcus aureus and/or multidrug-resistant Acinetobacter baumannii was performed. Second, articles were reviewed according to the STROBE checklist for epidemiological studies. Third, a set of potential new items focused on antimicrobial-resistance quality indicators was derived through an expert two-round RAND-modified Delphi procedure and tested on the articles selected through the literature review.
Results The literature search identified 78 studies. Overall, the quality of reporting appeared to be poor in most areas. Five STROBE items, comprising statistical analysis and study objectives, were satisfactory in <25% of the studies. Informative abstract, reporting of bias, control of confounding, generalisability and description of study size were missing in more than half the articles. A set of 21 new items was developed and tested. The new items focused particularly on the study setting, antimicrobial usage indicators, and patients epidemiological and clinical characteristics. The performance of the new items in included studies was very low (<25%).
Conclusions Our paper reveals that reporting in epidemiological papers analysing the association between antimicrobial usage and development of resistance is poor. The implementation of the newly developed STROBE for antimicrobial stewardship (AMS) tool should enhance appropriate study design and reporting, and therefore contribute to the improvement of evidence to be used for AMS programme development and assessment.
- Reporting
- Antibiotic resistance
- MRSA
- MDR-Acinetobacter
- Antimicrobial stewarsdhip
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strength and limitations of this study
This study developed a revised version of Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) to increase appropriateness of reporting for epidemiological studies focused on the link between antimicrobial-resistant bacteria and antibiotic usage.
STROBE for antimicrobial stewardship (STROBE-AMS) was developed through a 2-round Delphi experts approach. The final set included 21 new variables and four areas were graded as the most relevant: definition of infection and/or colonisation; setting; antibiotic usage at patient’s and hospital’s level; molecular resistance mechanisms; and infection control.
The STROBE-AMS application was tested only in epidemiological studies on methicillin-resistant Staphylococcus aureus and multidrug-resistant Acinetobacter baumannii. Since the postdevelopment test was performed only for observational reports, no specific bias was introduced.
Introduction
The rate of infections caused by antimicrobial-resistant microorganisms is seen increasingly by the public and healthcare inspection organisations as an indicator of quality of healthcare and patient safety.1 Several studies have explored risk factors associated with infection or colonisation due to antimicrobial-resistant microorganisms.2 ,3 Among them, antimicrobial exposure is a well-known risk factor for subsequent infections due to methicillin-resistant Staphylococcus aureus (MRSA), vancomycin-resistant enterococci (VRE) and multidrug-resistant (MDR) Gram-negative bacteria.4–6 Antimicrobial stewardship (AMS), included in a multifaceted approach, represents an essential tool to combat the spread of resistance in the healthcare setting.7 ,8 An effective AMS programme should include the appropriate choice of empirical therapy (including decision on no antimicrobial treatment) as well as duration and dosage of the drug in order to provide the best outcome for the patient, to reduce adverse effects, and to prevent and control the rate of antimicrobial resistance.9 As with intervention measures, AMS should be based on evidence. However, although randomised controlled trials (RCTs) are considered to be the gold standard of epidemiological research, analyses of risk factors for antimicrobial-resistant infections, especially those in which we can intervene, are usually not tested by RCTs, and rely mainly on observational studies. The major limitations of epidemiological studies on antimicrobial-resistant infections are the retrospective design, poor controlling for confounding factors (ie, duration and dosage of antibiotics, knowledge of carrier status, combination and sequential antibiotic therapy) and heterogeneity in several definitions (ie, definition of infection and of antimicrobial-resistant microorganisms, time-at-risk period for previous antimicrobial exposure, selection of control group).10–13 A further limitation affects external validity, since prevalence of resistance changes in different locations and in time. In an era of increasing antimicrobial resistance, reducing these flaws and increasing appropriate reporting is critical to the application of findings to AMS programmes.
The ‘Strengthening the Reporting of Observational Studies in Epidemiology’ (STROBE) was an initiative to improve the quality of reporting evidence.14 It aimed to provide useful guidance by establishing a checklist consisting of 22 items for the transparent and complete publication of observational studies, facilitating their critical evaluation.14 Recently, an extension of the STROBE statement (STROME-ID) established recommendations to support good scientific reporting of molecular epidemiological studies.15 However, the peculiarity of the epidemiology of antimicrobial-resistant infections being strongly connected to patients’ characteristics as well as to different settings (hospital, community and healthcare centres) as well as surveillance methods of antibiotic usage (ie, defined daily dose or packets or prescriptions) cannot be properly covered through the items currently included in STROBE.
Therefore, we performed a study with two main objectives: (1) to explore the accuracy of application of STROBE in epidemiological studies focused on the link between antibiotics usage and development of resistance; and (2) to derive and test an extension of STROBE that could improve the suitability of the tool in evaluating the quality of reporting of epidemiological studies in this area. Studies addressing MRSA and the MDR-Acinetobacter baumannii group were selected for the frequency with which they have been used worldwide as indicators of healthcare associated infections.2 ,16
Study design
Methods
There were three-steps to the study design. First, to explore the STROBE application, we searched the literature (1976–2013) for articles analysing the association between antimicrobial exposure, and acquisition of MRSA and/or MDR-A. baumannii. Second, all the articles were reviewed according to the STROBE checklist.14 Third, a set of potential new items focused on antimicrobial-resistance quality indicators for proper reporting was derived through a two-round RAND-modified Delphi procedure, involving experts in the field of antimicrobial prescribing.17 The extension of STROBE for AMS (STROBE-AMS) was then tested on the articles selected in the first study step.
Selection of articles
Published human studies concerning the role of previous antimicrobial therapy as a risk factor for developing MRSA and MDR-A. baumannii colonisation or infection in hospitalised patients were identified through computerised literature searches using MEDLINE (National Library of Medicine, Bethesda, Maryland, USA) and EMBASE (Excerpta Medica Database), and by reviewing the references of the retrieved articles. Index search terms included the following Medical Subject Headings: ‘risk factors’, ‘resistance’, ‘antimicrobial therapy’, ‘Acinetobacter’, ‘outbreak’, ‘Staphylococcus aureus’ and ‘methicillin-resistant’. The search was carried out with no language restriction and followed PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines. Studies were considered eligible if they included adult patients (>16 years old) and presented data pertaining to the relationship between antimicrobial use and the development of MRSA or MDR-A. baumannii colonisation or infection. Authors were contacted for missing information.
Data extraction
Data extraction was performed by four independent researchers in two centres. This process involved infectious disease specialists as well as epidemiologists. In case of disagreement among the reviewers, a senior reviewer was consulted. Reviews, letters, editorials and case reports were excluded. Data extraction included also the impact factor (IF) of the journal (2013 Journal Citation Report, Thomson Institute for Scientific Information), which is a measure of average citation frequency for all the articles over a given period of time.
Reporting assessment
The assessment of the quality of reporting was performed according to the STROBE statement.14 Data from each study were entered into standardised forms, verified for consistency and accuracy, and then entered into a computerised database. Entries for each item of the STROBE statement were categorised as ‘Yes’, if they were completely in agreement with the STROBE statement explanation and elaboration documentation, ‘No’, if this was not satisfied, and ‘Partly’, if evident only in part of the text. Absolute and relative frequencies were used to describe the results of these judgments of quality of reporting according to the STROBE statement.
New items development
The new items development was carried out through a two-round Delphi approach. A list of experts in the field of antimicrobial prescribing was collated from the network of the European Society of Infectious Diseases and Clinical Microbiology (ESCMID) and the Impact of Specific Antibiotic Therapies on the prevalence of hUman host ResisteNt bacteria (SATURN) Project on antimicrobial usage and selection of resistance in hospitalised patients (EU-7th FP7-241796). To develop a first set of quality indicators, the literature was reviewed to define the major limitations of current research on the association between antimicrobial usage and antimicrobial resistance,11 ,18 ,19 and a first set of indicators, developed from the authors, was sent out in December 2013. Respondents were asked to rate each new item against two continuous 1–9 integer scales and provide their comments. The second round was performed in May 2014, according to the same indications. Ethic consent was not required because no patient data were used and the study was based on literature review.
Test of the new items
The new developed items were then applied to the papers selected for the systematic review. The same reporting assessment used for the STROBE evaluation was applied.
Statistical analysis
Scores were analysed using a non-parametric test (Fischer's test). Analysis was performed with STATA V.12.1 (Stata Corporation, Texas, USA).
Results
Identification of relevant articles
The search identified 1008 potentially relevant studies; 311 were excluded because they were reviews, case reports or letters; 233 were excluded because they did not investigate previous antibiotic therapy. Among the remaining 464, 386 articles were excluded because they did not analyse the relationship between antibiotic usage and development of resistance, were duplicates, or their authors could not provide missing information. Overall, 78 studies (57 on MRSA and 21 on MDR-Acinetobacter) fulfilled inclusion criteria for the association between antimicrobial usage and antimicrobial resistance: 39 (51%) case–control, 28 (35%) cohort and 11 (14%) cross-sectional studies. The list of reviewed articles and the flow chart are reported in online supplementary annex 1—figure 1 and table 1.
Reporting assessment
The results of the assessment of quality of reporting through the STROBE statement of these 78 studies are shown in table 1. Overall, the quality of reporting appeared to be poor in most areas. Best performing areas of the STROBE tool were seven items including the description of setting, participants and generic review of statistical methods that were well satisfied in >75% of the studies. However, five items, comprising statistical details for the analysis of quantitative variables, subgroup analysis, sample size calculation, addressing potential sources of biases and, most notably, the description of main study objectives, were satisfactory in <25% of the studies.
STROBE item checklist for 78 epidemiological studies analysing antimicrobial usage and development of colonisation/infections due to MRSA and MDR-Acinetobacter baumannii
Table 2 describes the association between the value of the IF and the grade of satisfaction of the STROBE statement's items. Informative abstract, explanation of objectives, reporting of bias, statistical methods, control of confounding, generalisability, definition of quantitative variables, giving of estimates and description of study size, were missing or incomplete in more than half the articles published in 13 journals with IF greater than four. The cut-off of four was selected based on the IF median distribution. There was no association between type of journal (general medicine vs clinical infectious diseases and microbiology-dedicated journal) and year of study publication (before and after 2007, when the STROBE tool was introduced).
Association between satisfaction of the STROBE items and journal's IF
Development of new items
In the first round, 16 new items were sent to the experts. No indicators were discarded between rounds and five new items were added. In the second round, participants were provided with the frequency of distribution of scores and qualitative comments. The final set included 21 new variables, presented in table 3. The main new items referred to the description of the study setting and participants. In particular, the following four areas were graded as the most relevant for studies reporting on antimicrobial usage and resistance development: definition of infection and/or colonisation, and evidence of robustness of the new definition (if not a validated reference); definition of setting (epidemic or endemic); definition of antibiotic usage at patient’s level including type, dosage, duration, route of administration and combinations; description of antimicrobial formulary at the hospital level and measurement of antibiotic usage (defined as daily dosage, packet daily dosage, treatments, units), definition of how antibiotic consumption data were obtained (pharmacy, patients’ charts, etc) and if it was actually used or purchased/dispensed. The remaining three items focused on infection control measures applied at the study location: definition of resistance, cross-resistance and molecular resistance mechanisms, for example, plasmids, chromosomes, integrons, transposons; and statistical methods including controlling for confounders, rationale for grouping of antimicrobials and time at risk for antimicrobial exposure. Among the confounders, items reporting on previous stay in long term care facilities, nursing home and other hospitals, were added.
New checklist items proposed to be included in the STROBE statement for deepening the assessment of epidemiological studies analysing the impact of antimicrobial usage on the development of antimicrobial-resistant infections
Test of new items
The performance of reviewed articles to the new items was very poor. Overall, the new items were satisfied in <25% of the papers reviewed (item 5.3, 6.2, 7.1–4, 8, 11, 17 and 21).
Discussion
With increasing reporting of antibiotic-resistant infections worldwide, epidemiological investigations on risk factors for resistance require rather special attention regarding designing and reporting in order to maximise their ability to inform AMS programmes. Our paper reveals that current reporting in epidemiological studies focusing on the association between antibiotic usage and development of resistance is very poor. Although some items including the description of setting and generic review of statistical methods were well satisfied in >75% of the studies, more than half the 22 domains included in the STROBE checklist were not satisfied in the majority of studies analysing the role of antibiotics in selecting MRSA and MDR-A. baumannii.
The major consequence of this inaccuracy is that a significant part of the evidence contributing to the development of AMS programmes is currently of a very low level. Improvement of reporting could be achieved at two different levels: on the one side, it is essential to increase the awareness of researchers and publishers on the importance of the appropriate reporting for epidemiological studies on antimicrobial resistance, and on the other, availability of items specifically developed for this field could play a pivotal role, and increase feasibility and implementation. The new items developed by a group of experts through a Delphi approach, which combined evidence and expert opinion, mainly focuses on the methods section and includes specific epidemiological items (ie, definition of the epidemiological setting in terms of endemicity or epidemicity) as well as a set definitions for antimicrobial usage (ie, local formulary, antibiotic measurements, presence or absence of any AMS, etc). To further underline the innovation of the items and the importance of introducing the new STROBE-AMS, our results showed that the new items were not reported in more than two-thirds of 78 already published studies.
Interestingly, although a significant correlation was observed between the IF of the journal and satisfactory compliance with the STROBE statement criteria, there were very important areas such as an informative abstract, explanation of objectives, reporting of statistical methods and generalisability, that were missing or incomplete in >50% of articles published even in higher quality journals with IF factors >4. This result underlines the need for involvement of major scientific journals on clinical infectious diseases in a re-evaluation process concerning the review process of such papers. Indeed, RCTs are not accepted for publication in high-ranked journals if they do not comply with the CONSORT guidelines (Consolidated Standards for Reporting Trials).20 Similarly, systematic reviews and meta-analyses need to comply with the PRISMA guidelines.21 We would advocate that, for observational trials on antibiotic-resistant infections, where there is a greater potential for bias, this adaptation of the STROBE tool must be an essential review requirement, significantly contributing to the global efforts to combat the spread of antimicrobial-resistant microorganisms with improvement of the quality of evidence of the reports. Implicit in this requirement is that the studies will have had to consider these checklists at the design stage as well.
Our study has limitations. The analysis of STROBE application in the field of antimicrobial-resistant microorganisms was limited to epidemiological studies on MRSA and MDR-A. baumannii. These microorganisms were selected as their occurrence is very common, the majority of published epidemiological studies are focused on these bacteria, and their prevention and control are important potential indicators of the success, or otherwise, of healthcare associated interventions. The testing of the new items was performed on the same group of articles tested for satisfying the STROBE. However, we believe that, since the postdevelopment test was performed only for observational reports, no specific bias was introduced. Generalisability of our study applies therefore only to epidemiological studies exploring the association between previous antibiotic therapy and development of infection and/or colonisation due to antimicrobial-resistant strains.
Our study has shown, through an expert's Delphi approach, that the current version of the STROBE tool does not describe all the components considered essential to define the association between antibiotic usage and resistance. That these studies should be conducted and reported effectively has become an imperative, given the global threat imposed by antimicrobial-resistant microorganisms, for future effective therapies.1 ,18 ,19 Increasing proper reporting will reduce heterogeneity between papers, and assist in the evaluation of the evidence through systematic reviews and peer reviewing for journal and grant proposals. The implementation of the STROBE-AMS will also impact on other aspects of antimicrobial-resistant research, including study design, extraction of data and generalisability of results. We do believe that the introduction of STROBE-AMS will increase the quality of available evidence to policymakers, relevant healthcare workers and the research community, and ultimately ensure that there are sustained improvements to AMS throughout the world.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
- Data supplement 1 - Online supplement
Footnotes
Contributors ET and MAC conceived and designed the study. ET, NTM, FF, PG, JK, LL and MP designed the structure and content of the statement. FF, WS, CC and CDW performed the literature review. ET, BC and GDA wrote the first draft of the statement. All the authors contributed to the writing of the statement, and agree with its content and conclusions.
Funding The study was partially funded through the DZIF (Deuschte Zentrum für Infektion Forschung; German Center for Infectious Diseases Research) funding for the Clinical Trial Unit for healthcare-associated infection and the DRIVE-AB study (IMI 115618).
Disclaimer The funding source had no role in the analysis of data and results reporting.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.