Article Text
PDF
Abstract
Introduction Acute rhinosinusitis (ARS) is a prime reason for doctor visits and among the conditions with highest antibiotic overprescribing rates in adults. To reduce inappropriate prescribing, we aim to predict the absolute benefit of antibiotic treatment for individual adult patients with ARS by applying multivariable risk prediction methods to individual patient data (IPD) of multiple randomised placebo-controlled trials.
Methods and analysis This is an update and re-analysis of a 2008 IPD meta-analysis on antibiotics for adults with clinically diagnosed ARS. First, the reference list of the 2018 Cochrane review on antibiotics for ARS will be reviewed for relevant studies published since 2008. Next, the systematic searches of CENTRAL, MEDLINE and Embase of the Cochrane review will be updated to 1 September 2020. Methodological quality of eligible studies will be assessed using the Cochrane Risk of Bias 2 tool. The primary outcome is cure at 8–15 days. Regression-based methods will be used to model the risk of being cured based on relevant predictors and treatment, while accounting for clustering. Such model allows for risk predictions as a function of treatment and individual patient characteristics and hence gives insight into individualised absolute benefit. Candidate predictors will be based on literature, clinical reasoning and availability. Calibration and discrimination will be evaluated to assess model performance. Resampling techniques will be used to assess internal validation. In addition, internal–external cross-validation procedures will be used to inform on between-study differences and estimate out-of-sample model performance. Secondarily, we will study possible heterogeneity of treatment effect as a function of outcome risk.
Ethics and dissemination In this study, no identifiable patient data will be used. As such, the Medical Research Involving Humans Subject Act (WMO) does not apply and official ethical approval is not required. Results will be submitted for publication in international peer-reviewed journals.
PROSPERO registration number CRD42020220108.
- primary care
- epidemiology
- otolaryngology
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
By applying multivariable risk prediction modelling approaches to individual patient data (IPD) of multiple randomised placebo-controlled trials, this study aims to identify relevant differential absolute treatment effects of antibiotics in adults with acute rhinosinusitis in primary care.
The large number of participants and events available in the IPD set allow us to evaluate up to 25 parameters in the model.
Due to the retrospective nature, it is possible that some potentially helpful predictors cannot be included in our analyses due to unavailability in the majority of studies.
Since an externally developed risk model is not available, we will develop a model using an internal–external cross validation procedure.
Introduction
Acute rhinosinusitis (ARS) is a prime reason for primary care visits in adults.1–3 At present, accurate diagnosis of ARS is challenging and general practitioners have to rely on a specific signs and symptoms.4 Although viral of origin and self-limiting in the majority of cases,5–7 ARS is among the conditions with highest antibiotic overprescription rates in adults.1 8 The lack of evidence to support management decisions in adults with ARS might explain the apparent overuse of diagnostic tools and medications such as antibiotics, intranasal corticosteroids, anthistamines and mucolytics in everyday practice.9 Tailored guidance based on identification of patients that are most likely to benefit from antibiotics has the potential to substantially reduce inappropriate prescribing in this common condition.10
A previous individual patient data (IPD) meta-analysis of randomised controlled trials (RCTs) comparing antibiotics with placebo in adults with clinically diagnosed ARS aimed to provide such guidance.7 This IPD meta-analysis combined data from 2547 participants in nine trials and investigated whether common signs and symptoms such as duration of symptoms, body temperature, pain on bending and purulent nasal discharge modified antibiotic effectiveness. By applying conventional (one-variable-at-a-time) subgroup analysis,11–13 no single sign or symptoms could be identified to predict antibiotic benefit.7 However, evidence is accumulating that underlying clinical heterogeneity is likely under-represented and potential important differences in treatment effects may be obscured when taking only single variables into account.11–13 Prediction modelling approaches which allow for simultaneous analysis of multiple baseline variables that may influence treatment effect have the potential to overcome these problems and can be used to predict differential absolute treatment effects, the most useful scale for clinical decision making.13–16 These approaches are considered most valuable when (1) an overall treatment effect exists, (2) treatment may lead to substantial harm/burden, (3) substantial heterogeneity in the trial population is anticipated (broad case mix), (4) multiple RCTs are available and appropriate for pooling in IPD meta-analysis and (5) covariates in the model are routinely available in everyday practice.15 16 With antibiotics for clinically diagnosed ARS in adults meeting all these criteria, we aim to predict the benefit of antibiotic treatment for individual patients by applying multivariable risk prediction modelling approaches to IPD of multiple placebo-controlled trials in this field. This may ultimately improve patient care by more judicious prescription of antibiotics in adults with ARS and thereby curb antimicrobial resistance.
Methods
The protocol is reported according to the Preferred Reporting Items for Systematic Review and Meta-Analysis protocols (PRISMA-P) statement.17 The IPD meta-analysis will follow the recommendations provided in the Predictive Approaches to Treatment effect Heterogeneity (PATH) statement15 and the guidance on the use of IPD meta-analyses of diagnostic and prognostic modelling studies18 and reported according to the PRISMA-IPD statement.19
Patient and public involvement
Patients were not involved in development of the protocol.
Identification of relevant studies: a systematic review
A systematic review will be performed to identify and select any relevant studies published since the 2008 IPD meta-analysis.7
Study eligibility criteria
All RCTs comparing antibiotics with placebo in patients with clinically diagnosed ARS will be eligible. Study participants must be ≥16 years of age suspected by their physician of having uncomplicated ARS based on clinical signs and symptoms and presenting in a primary healthcare setting (ie, patients who are not referred by a physician because of the current ARS episode, or patients presenting through self-referral in ambulatory care or an emergency department). No language or publication date restrictions will be applied. Studies involving children (<16 years), referred patients, hospitalised patients as well as those involving highly specialised populations (eg, those with immunodeficiency, odontogenic sinusitis or malignancy) will be excluded.
Search strategy
First, the reference list of the 2018 Cochrane review on antibiotics for ARS in adults6 will be reviewed for any relevant studies published since the 2008 IPD meta-analysis.7 Next, the systematic electronic searches of the Cochrane review will be updated from 18 January 2018 (date of last search) to 1 September 2020 to increase the yield of potentially relevant trials (online supplemental file 1). In addition, reference lists of all eligible studies as well as those from relevant systematic reviews will be screened for any further potential studies. Finally, contributing review authors will be asked if they are aware of any additional (published or unpublished) studies.
Supplemental material
Study selection
One review author (RPV) will review the reference list of the 2018 Cochrane review6 for any additional trials, where relevant full texts will be retrieved. Next, two review authors (RPV, JH) will independently screen titles and abstracts of the unique records obtained from the electronic database searches to assess their potential relevance for reviewing the full text. The same review authors will independently review the full text of all potentially eligible articles against the predefined inclusion and exclusion criteria. Any disagreements will be resolved by discussion and if necessary, a third review author (JBR) will be consulted.
Data extraction and management
Corresponding authors of eligible trials published since the 2008 IPD meta-analysis7 will be contacted via an email. They will be invited to collaborate and share their de-identified, complete IPD of their original study in their preferred format. A data sharing agreement will be provided. Study data will be considered unavailable when none of the authors respond to multiple contact attempts or if study authors indicate that the requested data are not available or cannot be shared.
On retrieval, the IPD of individual studies will be reviewed against published data by comparing key variables (eg, number of participants, descriptive analysis of demographic characteristics). We will ask collaborators for clarification in case important discrepancies are identified. The amount of missing data within each study will be assessed and discussed with collaborators to reduce missing data as much as possible.
SPSS V.25 and R (R Foundation for Statistical Computing, Vienna, Austria) will be used for the statistical analyses. An aggregated database will be created containing a trial ID variable (to identify participants from the same study), patient demographics and characteristics including clinical signs and symptoms, treatment allocation (antibiotics or placebo) and outcome measure of interest (whether or not proportion of patients were clinically cured at days 8–157). The aggregated database will have a multilevel structure (with individual trials as levels or clusters). Missing data will be studied and appropriate methods for handling them, such as multiple imputation, will be considered.20–22
Quality assessment of included studies
Two review authors (RPV, JH) will independently assess the methodological quality of the included studies using the Cochrane Risk of Bias 2 tool.23 This tool allows to judge the risk of bias for the primary outcome of interest as ‘low’, ‘high’ or ‘some’ concerns for following five domains: (1) bias arising from the randomisation process, (2) bias due to deviations from the intended interventions, (3) bias due to missing outcome data, (4) bias in measurement of outcome, (5) bias in selection of the reported results. Any disagreements will be resolved by discussion and if necessary, a third review author (JBR) will be consulted. If information regarding study quality is unclear or undisclosed, individual study authors will be contacted.
Data synthesis
Descriptive analysis and evidence synthesis
For all studies contributing IPD, study and participants’ characteristics will be provided. We will review the characteristics of eligible studies that did not contribute IPD to investigate whether there is any evidence of selection (availability) bias.
IPD meta-analysis using outcome risk prediction
Outcome of interest
The main outcome of interest will be whether or not patients are cured at 8–15 days as per the definition applied in the 2008 IPD meta-analysis, that is, cure as defined in an individual trial or by agreement with the primary investigator in advance of the analysis with patients receiving an antibiotic in addition to or as replacement for the randomly assigned treatment regarded as not cured.7
Candidate predictors
Candidate predictors will be based on clinical reasoning, knowledge from existing literature and availability in the IPD set.
There is currently no externally developed prediction model available and insufficient evidence on potentially useful predictors of prolonged illness duration in adults with ARS.4 7 The 2020 European Position Paper on Rhinosinusitis and Nasal Polyps states that the presence of three or more of the following five symptoms increases the likelihood of acute bacterial rhinosinusitis (ABRS): (1) the presence of fever (>38°C), (2) double sickening, (3) unilateral disease, (4) severe pain, (5) raised C reactive protein (CRP) or erythrocyte sedimentation rate (ESR) values.4 However, further research into combinations of signs and symptoms predictive of ABRS in primary care is warranted.4 24
A recent systematic review found limited evidence that overall clinical impression, cacosmia and pain in teeth are the best predictors of ABRS.25 Using data from a single diagnostic accuracy study,26 preceding upper respiratory tract infection (URTI), preceding ARS, unilateral maxillary pain, pain in teeth, purulent nasal discharge and CRP >15 mg/L were predictive of positive bacterial culture of fluid from antral puncture.27 Of these variables, only raised CRP or ESR rates have been proven useful to predict antibiotic benefit in adult patients with clinically diagnosed ARS in one trial.28
Based on these considerations and clinical reasoning, potential candidate predictors include: (1) preceding URTI, (2) preceding ARS, (3) age, (4) duration of symptoms, (5) symptom severity, (6) maxillary pain (any, unilateral, bilateral), (7) pain in teeth, (8) pain on bending, (9) anosmia, (10) cacosmia, (11) double sickening, (12) overall clinical impression, (13) fever (>38°C), (14) purulent nasal discharge on examination, (15) purulent discharge in pharynx on examination, (16) CRP and (17) ESR. Continuous variables (age, CRP, ESR) will be kept continuous and the functional relationship with the outcome will be assessed through restricted cubic splines.
Sample size considerations
Recent methodological work has provided guidance with respect to sample size considerations for developing a clinical prediction model.29 The key rationale is to link desired risk prediction accuracy to sample size. We anticipate IPD availability for 2541 participants and an outcome proportion of 0.4.7 Due to the clustered nature of the data, the effective sample size will be smaller than 2541, but the available guidance does not yet extend to this setting. Conservatively estimating the effective sample size to be 1250, this allows for evaluation of up to 25 parameters in the model for a desired 0.05 accuracy in terms of mean absolute prediction error. Based on an anticipated Cox-Snell R2 of 0.175, this is also expected to keep shrinkage below 10% and the expected Cox-Snell R2 within 5%.
Model development and evaluation
The aggregated IPD set will be used to inform which predictors can be included in our model. Reasons for dropping variables will be based on non-availability in >50% of the studies and critical between-study heterogeneity in definition of a variable. The final decision to drop a variable based on this criterion will be based on consensus within research team. Next, remaining predictors will be included in a mixed effect logistic regression model30 together with treatment.15 16 Currently, there is no strong evidence in favour of clear pre-specification of treatment–covariate interactions in this setting. We will therefore include treatment–covariate interaction terms for variables indicative of ABRS according to the literature and perform a single pooled test for their combined significance. Based on this test, we either keep or remove the entire group of treatment–covariate interactions from the model. The overall ability of the model to discriminate between patients with and without clinical cure at 8–15 days will be quantified using the c-statistic. Calibration of the model will be assessed visually with calibration plots.
Since an initial prediction model commonly shows a too optimistic discrimination and calibration, that is, overfitting, we will use bootstrapping for internal validation and to guide possible adjustment for overfitting. Large unexplained between-study heterogeneity in predictive accuracy of the resulting model will be explored and discussed. Ultimately this may lead to adjustment of the model by dropping the variables that contribute most to the observed heterogeneity. Lastly, we will perform an internal–external cross validation procedure31 to estimate model performance outside the context of the acquired IPD. In this procedure, each study will be left out once for evaluation purposes while the model is fitted on the remaining studies. The final model will be based on the entire pooled dataset. The resulting absolute risk predictions under both treatments (antibiotics and placebo) and the predicted absolute treatment effects will be displayed graphically.
Secondary analysis
As a secondary analysis, we will study possible treatment effect heterogeneity as a function of outcome risk.15 16 The underlying rationale is that the absolute benefit of antibiotic treatment may continue to increase as the risk of an unfavourable outcome increases, while this is not reflected by a common OR. To this end, we will first develop a model for outcome risk and subsequently evaluate a mixed logistic model of outcome risk, treatment, and a possibly non-linear interaction between the two. Methods for evaluation and presentation of the results of the latter model are conform the primary analysis.
It is worth noting that a model including all possible treatment interactions, as opposed to the risk score summary used here, would be both more likely to capture the true underlying mechanism (since it allows relative treatment effect to vary as a function of each predictor) and overfit the data (for the same reason). Due to this risk of overfitting, even given the anticipated set of IPD, we chose to evaluate differential treatment effect as a function of a limited number of high potential treatment interactions in the primary analysis, and as a function of an overall risk score in this secondary analysis.
Ethics and dissemination
There will be no identifiable patient data in any of the datasets. As such, the Medical Research Involving Humans Subject Act (WMO) does not apply to this study. The Medical Research Ethics Committee Utrecht, the Netherlands, reviewed the study protocol (protocol 20-719/C) and concluded that an official approval was not required.
All principal investigators will provide written confirmation that all participants included in the original studies had given full written informed consent. Data will be stored within a secured folder of the data management department of the Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht. On completion of the study, data will be stored for the at least 15 years at a central drive of the data management department of the Julius Center and will be made available for the use by third parties on request and approval of the research team.
Research findings will be published in international peer-reviewed journals and will be presented at national and international scientific conferences.
Review registration and anticipated end date of study
The review has been registered in PROSPERO; CRD42020220108. The anticipated end date of data collection is March 2021 and the anticipated end date of the study is October 2021.
Discussion
High-quality data indicate that antibiotics on average have only limited beneficial effects in adults with clinically diagnosed ARS.6 7 Nevertheless, antibiotic prescribing rates remain at a high level in this common condition; antibiotics are prescribed in around 50% of patients in the Netherlands and up to 90% of patients in the UK and the USA.1–3 This routine practice exposes individual patients to common side effects32 and both the individual as well as the population to emerging antimicrobial resistance.33 One of the reasons for the persistant habit to prescribe antibiotics to patients with ARS might be attributed to clinicians’ gut feeling that there is a subgroup of patients that do benefit from antibiotics.10 There is some evidence to substantiate this impression; antibiotics seems to have larger effects in those with a radiologically confirmed diagnosis, in particular those with fluid level or total opacification in any sinus on CT.6 Unfortunately, current attemps to identify these subgroups of patients including a previous IPD meta-analysis were not successful.7
In this update and re-analysis of the IPD meta-analysis,8 we aim to predict the benefit of antibiotic treatment for individual patients by applying multivariable risk prediction methods.13–15 Missing data, in particular those missing for all individuals in one or more trials, pose a particular challenge in this project. Despite using multiple imputation methods to address missing data, it is likely that some potentially helpful predictors cannot be included in our IPD meta-analysis due to non-availability in >50% of the studies. Nonetheless, the large number of participants available in the IPD set will allow us to include a substantial number of candidate predictors in our model, thereby providing a unique opportunity to identify potential heterogeneity of antibiotic effectiveness in adults with clinically diagnosed ARS.
As per recent guidance, we propose to incorporate only a limited number of multiplicative treatment-by-coviarte interaction terms into the model.15 16 Further methodological research will be needed to identify methods that can effectively explore a wider spectrum of treatment–covariate interactions, while minimising the risk of modelling spurious associations also known as overfitting. Further remaining uncertainties and questions when it comes to best practices for PATH include: (1) what is the optimal way to examine multiplicative treatment-by-continuous covariate interactions, (2) how will results vary between different approaches for covariate selection and shrinkage, (3) how to best examine and express the impact of heterogeneity of treatment effects between individual studies, and do (4) other statistical approaches such as latent-class analysis and (5) more recent data-analytical approaches such as artificial intelligence have any benefit in modelling heterogeneity of treatment effect.
Ethics statements
Acknowledgments
We gratefully thank Dr James Robert Young for commenting on the draft protocol.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @MaroeskaRovers
Contributors RPV is the guarantor. RPV conceptualised and designed the study and acquired funding. RPV drafted the first version of the protocol. JH, MvS, MMR, JBR contributed to the concept and design of the study and critically revised the draft of the protocol for important intellectual content. AIdS, DM, GAvE, LK, HL, PL, HCCB critically revised the draft of the protocol for important intellectual content. All authors provided final approval for submission.
Funding This study was supported by The Netherlands Organisation for Health Research and Development (grant 91618026).
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.