Article Text
PDF
Abstract
Introduction This protocol outlines a diagnostic individual patient data (IPD) meta-analysis aimed at developing simple prediction models based on readily available signs, symptoms and blood tests to accurately predict acute bacterial rhinosinusitis and CT-confirmed (fluid level or total opacification in any sinus) acute rhinosinusitis (ARS) in adults presenting to primary care with clinically diagnosed ARS, target conditions associated with antibiotic benefit.
Methods and analysis The systematic searches of PubMed and Embase of a review on the accuracy of signs and symptoms for diagnosing ARS in ambulatory care will be updated to April 2020 to identify relevant studies. Authors of eligible studies will be contacted and invited to provide IPD. Methodological quality of the studies will be assessed using the Quality Assessment of Diagnostic Accuracy Studies-2 tool. Candidate predictor selection will be based on knowledge from existing literature, clinical reasoning and availability. Multivariable logistic regression analyses will be used to develop prediction models aimed at calculating absolute risk estimates. Large unexplained between-study heterogeneity in predictive accuracy of the models will be explored and may lead to either model adjustment or derivation of separate context-specific models. Calibration and discrimination will be evaluated to assess the models’ performance. Bootstrap resampling techniques will be used to assess internal validation and to inform on possible adjustment for overfitting. In addition, we aim to perform internal–external cross-validation procedures.
Ethics and dissemination In this IPD meta-analysis, no identifiable patient data will be used. As such, the Medical Research Involving Humans Subject Act does not apply, and official ethical approval is not required. Findings will be published in international peer-reviewed journals and presented at scientific conferences.
PROSPERO registration number PROSPERO CRD42020175659.
- primary care
- epidemiology
- otolaryngology
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Strengths and limitations of this study
Combining individual patient data (IPD) from multiple diagnostic accuracy studies allows for more statistical power and therefore more precise predictions.
The study will be conducted according to the most recent guidance on the use of IPD meta-analyses of diagnostic modelling studies.
Some potentially useful predictors may not be available in >50% of the included studies.
There might be differences in measurement of index and reference standard tests between studies.
Introduction
Acute rhinosinusitis (ARS) is a very common infection and an important reason for consultations by adults in ambulatory care.1 2 The condition places a huge burden on society; it has been associated with impaired quality of life for patients and with high direct and indirect costs.3
Clinicians have to rely on clinical signs and symptoms such as nasal discharge or congestion, facial pain or pressure, and tooth pain for diagnosing ARS.3 However, these symptoms overlap considerably with other conditions, making accurate diagnosis challenging.
Despite most sinus infections being viral in origin4 and antibiotics having only marginal beneficial effects in otherwise healthy adults with uncomplicated ARS,5 6 antibiotic prescribing rates vary from around 50% in the Netherlands to up to 90% in the UK and the USA.2 7 8 Importantly, ARS is one of the conditions with highest antibiotic overprescribing rates,1 9 thereby exposing patients to avoidable side effects10 and the population to emerging antimicrobial resistance.11
With antimicrobial resistance posing an increasing threat to public health globally,12 effective strategies to encourage judicious use of antibiotics in common infections such as ARS are urgently needed. So far, educational programmes focusing on targeted or delayed antibiotic prescribing in primary care and patient information strategies have been effective in clinical studies,13–15 but adoption of these strategies in day-to-day practice has proven challenging with antibiotic use in adults remaining at high levels2 16 17
Among other reasons, clinicians tend to prescribe antibiotics because they feel unable to recognise patients that are most likely to benefit from antibiotics.18 Unfortunately, a meta-analysis of individual patient data (IPD) of 10 randomised controlled trials (RCTs) on the effectiveness of antibiotics in patients with clinically diagnosed ARS failed to identify signs and symptoms that predicted antibiotic benefit.5 Identification of specific signs and symptoms predictive of acute bacterial rhinosinusitis (ABRS) diagnosis has therefore been recognised as an important unmet research need.3 Placebo-controlled trial evidence indicates that this target condition is indeed associated with antibiotic benefit in adult outpatients. First, antibiotics are associated with improved cure rates in adult patients with clinically diagnosed ARS and elevated values of C reactive protein (CRP) or erythrocyte sedimentation rate (ESR),19 which in turn have been associated with positive bacterial culture of antral fluid.20 Second, antibiotics lead to significantly faster and better recovery than placebo (number needed to treat to benefit=3) in those with evidence of a fluid level or total opacification in any sinus on CT,21 which in turn has been shown highly predictive of pus or mucopus by antral puncture.22 Importantly, antibiotics do not lead to improved recovery compared with placebo in outpatient adults with clinically diagnosed ARS and mucosal thickening but without fluid level or total opacification on CT scan.23 This latter finding is consistent with those from a CT study of the common cold showing that 27 of 31 patients (87%) who had a CT performed within a few days after symptom onset had abnormalities of one or both maxillary sinuses, which were almost exclusively related to mucosal thickening.24
Routine CT scanning or antral puncture in a common and generally self-limiting condition such as ARS is however neither feasible nor cost-effective. However, a simple model based on readily available signs, symptoms and blood tests that can accurately predict ABRS or CT-confirmed (fluid level or total opacification in any sinus) ARS has the potential to help clinicians identify patients most likely to benefit from antibiotics and thereby substantially reduce inappropriate antibiotic prescribing in this common condition. While previous work has indicated that such models are promising,25 recent insights into sample size calculations for studies aimed at developing binary logistic prediction models26–28 reveal that the sample sizes of the individual studies in this field22 29–32 do not reach the minimum sample size required to ensure precision of the model predictions and minimise the risk for overfitting. This problem can however be overcome when combining IPD from these studies. In this paper, we describe the protocol of a diagnostic IPD meta-analysis aimed at developing such prediction models.
Methods
The protocol is reported according to the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols statement.33 The IPD meta-analysis will follow the guidance on the use of IPD meta-analyses of diagnostic and prognostic modelling studies34 and will be reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement for diagnostic test accuracy studies (PRISMA-DTA) and the PRISMA-IPD statement.35 36
Patient and public involvement
Patients were not involved in development of the protocol.
Identification of relevant studies: a systematic review
A systematic review will be performed to identify and select any relevant studies. Study authors of eligible papers will be contacted and invited to provide IPD of their original study.
Study eligibility criteria
All studies on diagnostic accuracy of readily available signs, symptoms and blood tests for diagnosing ARS using a fluid level or total opacification in any sinus on CT scan and/or bacterial culture of antral fluid and/or bacterial culture of endoscopically guided middle meatal aspiration or swabs37 38 as the reference standard tests (table 1) will be eligible. Study participants must be adults (≥15 years) suspected by their physician of having uncomplicated ARS based on clinical signs and symptoms and presenting in a primary healthcare setting (ie, non-referred patients consulting their physician for the first time or patients presenting through self-referral in ambulatory care or an emergency department). No language restrictions will be applied. Studies involving children (<15 years), referred patients, hospitalised patients as well as those involving highly specialised populations (eg, those with immunodeficiency, odontogenic sinusitis or malignancy) will be excluded. To avoid verification bias, only studies in which all patients received the reference standard tests of interest (ie, CT scan, culture of endoscopically guided middle meatal aspiration or swabs or culture of fluid from antral puncture as part of clinical work-up) will be included.
Definition of index and reference standard tests
Search strategy
First, the reference list of a recent systematic review on the accuracy of signs and symptoms for diagnosing ARS in ambulatory care using any reference standard4 will be reviewed for relevant studies. Next, to increase the yield of potentially relevant studies, the systematic searches of PubMed and Embase of this review (online supplemental file 1) will be updated from 1 January 2015 to 1 April 2020. In addition, reference lists of all eligible studies as well as those from relevant systematic reviews will be screened for any further potential studies. Contributing review authors, experts in the field and the task force from the Danish Medicine Council working on the diagnosis and treatment of ARS will be asked if they have, or are aware of, any additional (published or unpublished) studies. Where necessary, individual study authors will be contacted.
Supplemental material
Study selection
First, two review authors (RV and MHE) will independently review the reference list of the recent systematic review4 for eligible papers; where relevant, full texts will be retrieved. Next, two review authors (RV and MHE) will independently screen titles and abstracts of the unique records obtained from the electronic database searches and the reference lists of the relevant systematic reviews to assess their potential relevance for reviewing the full text. The same review authors will independently review the full text of all potentially eligible articles against the predefined inclusion and exclusion criteria. Any disagreements will be resolved by discussion.
Data extraction and management
Authors of eligible studies will be contacted via an email outlining the main study aim and inviting them to collaborate and share their deidentified, complete IPD of their original study in their preferred format. A data sharing agreement will be provided. Study data will be considered unavailable when none of the authors respond to multiple contact attempts or if all contacted study authors indicate they do not have access to the requested data and are unable to provide contact details of a colleague who does have access.
Retrieved IPD of individual studies will be checked by comparing key variables (eg, number of participants, descriptive analysis of demographic characteristics, index tests and reference standard test) with published data. We will contact collaborators for clarification if important discrepancies are identified. The level of missing data within each study will be assessed and discussed with collaborators with the aim to reduce missing data as much as possible. Finally, missing data will be investigated and appropriate methods for handling them, such as multiple imputation, will be considered.39
All IPD data to be included in the meta-analysis will be converted to SPSS V.25 and R (R Foundation for Statistical Computing, Vienna, Austria). A database will be created containing study ID, patient ID, index tests (readily available signs, symptoms and blood tests including CRP, ESR and reference standard tests. Any critical differences in coding and/or measurement of index tests or the reference standard tests between studies will be discussed with the entire research team to decide how to resolve this issue most adequately. Harmonised data from the individual studies will be aggregated into single IPD sets with a study ID variable to identify participants from the same study. The aggregated database will be rechecked for accuracy.
Quality assessment of included studies
Two review authors (RV and MHE) will independently assess the methodological quality of the included studies using the Quality Assessment of Diagnostic Accuracy Studies-2 framework adapted to our research question.40 This tool assess internal validity (risk of bias) and external validity (concerns about applicability) in four domains: patient selection, index test, reference standard test and flow and timing. Any disagreements will be resolved by discussion. If information regarding study quality is unclear or undisclosed, individual study authors will be contacted.
Data synthesis
Descriptive analysis and evidence synthesis
Study and participants’ characteristics will be provided for all studies that contribute IPD. Characteristics of eligible studies that did not contribute IPD will be considered to assess whether there is any evidence of selection bias.
IPD meta-analysis
Sample size considerations
Traditionally, sample size calculations for studies aiming to develop binary logistic prediction models use the widely known ‘10 events per variable’ rule of thumb.41 The validity of this rule of thumb has however been debated,26 and recent research revealed alternative approaches to calculate the minimum sample size required when primary interest is in prediction accuracy.27 28
An interactive sample size calculation tool (https://mvansmeden.shinyapps.io/BeyondEPV/) is now available to ensure predictive accuracy of a new prediction model with ≤30 candidate predictors.24 Anticipating IPD availability for at least 375 participants (from two studies using CT scan as reference standard test)22 31 with an outcome proportion of 0.5 and the mean absolute prediction error set at 0.054, six candidate predictors could be included in our final model.
Alternatively, it has been suggested to identify the sample size and number of candidate predictors that minimises the amount of desired shrinkage (≤10%) during model development.28 Sample size calculation using this approach requires prespecification of: (1) the number of candidate predictors, (2) outcome proportion and (3) the model’s anticipated Cox-Snell R-squared statistic, which reflects the signal:noise ratio (value between 0–1 with lower values indicating more difficulties in detecting patterns thereby increasing the potential for overfitting that ultimately leads to higher required sample sizes and fewer candidate predictors to be estimated reliably).28 Anticipating IPD availability for at least 375 participants22 31 and an anticipated Cox-Snell R-squared statistic of 0.15, six candidate predictors can be included in our final model to target an expected shrinkage of 0.9 (which reflects an events per candidate predictor parameter of 27.4 assuming an outcome proportion of 0.5).28
Selection of candidate predictors
Candidate predictors will be based on knowledge from existing literature and clinical reasoning. Previous research indicated that the following variables most likely predict the likelihood of ABRS or CT-confirmed ARS: (1) preceding upper respiratory tract infection, (2) previous diagnosis of ARS, (3) maxillary pain (any, unilateral and bilateral), (4) pain in teeth, (5) purulent nasal discharge on examination, (6) anosmia, (7) cacosmia, (8) overall clinical impression, (9) double sickening, (10) CRP and (11) ESR (table 1).4 20 22 25 32 42 Where possible, we will add the following, clinically sensible, predictors to the models to investigate their added diagnostic value: duration of illness (>10 days), fever (>38°C) and severe pain (table 1).
Continuous variables (CRP and ESR) will ideally be kept continuous and the functional relationship with the outcome will be assessed through restricted cubic splines.
Model development
The aggregated IPD dataset will be used to inform which predictors will be included in our multivariable logistic regression models.43 Reasons for dropping variables will be based on non-availability in >50% of the studies and critical between-study heterogeneity in measuring a variable.
If potential relevant differences in how the reference standard has been performed between studies remain unsolved, we will explore whether this leads to differences in regression coefficients and the performance of models between studies. Meaningful heterogeneity in results will be reported and incorporated in our modelling strategy (eg, stratified models). Large unexplained between-study heterogeneity in predictive accuracy of the models will also be explored and discussed and may ultimately lead to either adjustment of the models (by dropping the variables that contribute most to the observed heterogeneity) or derivation of separate context-specific models.
The overall ability of the models to discriminate between patients with and without ABRS or abnormal CT finding will be quantified using the c-statistic. Calibration of the models will be assessed graphically with calibration plots.
Since an initial prediction model commonly shows a too optimistic discrimination and calibration, that is, overfitting, we will use bootstrap resampling techniques for internal validation and, where appropriate, to obtain an average shrinkage factor to adjust regression coefficients for overfitting. In addition, we aim to perform an internal–external cross-validation procedure44 to estimate models’ diagnostic performance outside the context of the acquired IPD; every study will be left out once for validation of the models on the remaining studies. The final models will based on the entire pooled dataset and will generate continuous risk estimates. The final models will be presented in nomograms to allow for user-friendly decision making at an individual level45 using Shiny web applications. We aim to use net benefit and decision curves to explore the clinical value of the models.46
In exploratory analyses, we aim to apply other statistical approaches such as classification and regression trees, artificial neural networks, and fast and frugal trees to explore whether they generate additional insight.
Ethics and dissemination
There will be no identifiable patient data in any of the datasets. As such, the Medical Research Involving Humans Subject Act (WMO) does not apply to this study. The Medical Ethics Review Committee Utrecht, the Netherlands, reviewed the study protocol (protocol 20–331/C) and concluded that an official approval was not required.
All principal investigators will provide written confirmation that all participants included in the original studies had given full written informed consent. Data will be stored within a secured folder of the data management department of the Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht. On completion of the study, data will be stored for at least 15 years at a central drive of the data management department of the Julius Center and will be made available for the use by third parties on request and approval of the research team.
Research findings will be published in international peer-reviewed journals and will be presented at national and international scientific conferences.
Review registration and anticipated end date of study
The review has been registered in PROSPERO; CRD42020175659. The anticipated end date of data collection is January 2021, and the anticipated end date of the study is August 2021.
Discussion
With antibiotic overprescribing for ARS in adults estimated at 51%–54%,1 9 continuing efforts are needed to substantially reduce inappropriate antibiotic overprescribing in this condition. Tailored guidance based on identification of subgroups that are most likely to benefit from antibiotics has the potential to do so; such guidance for clinicians47 led to reductions in inappropriate antibiotic prescribing for childhood acute otitis media with current overprescribing estimated at 4%.9 For ARS, such evidence-based guidance is still lacking,5 which leaves clinicians with a feeling of uncertainty and thereby creating a situation in which overuse of diagnostic tools and medications commonly occur.48
The recently updated European Position Paper on Rhinosinusitis and Nasal Polyps has highlighted the most urgent unmet research needs in ARS. Among further RCTs evaluating the effectiveness of the full range of medications in ABRS, research into combinations of symptoms and signs predictive of ABRS has been recognised as top priority.3 The proposed diagnostic IPD meta-analysis aims to fill this gap by developing simple multivariable models based on readily available signs, symptoms and blood tests that can accurately predict ABRS or a fluid level or total opacification in any sinus on a CT scan in adults presenting to primary care with ARS, target conditions associated with antibiotic benefit.19–21 Previous work from single studies indicated that such models are promising,25 but combining IPD from multiple diagnostic accuracy studies allows for more statistical power and therefore more precise predictions. An initial scoping exercise also revealed that most of the candidate predictors were measured consistently across the individual studies that facilitate meta-analysis. Using state-of-the art diagnostic IPD meta-analysis methods, this project will ultimately reveal whether simple and easy-to-implement models can be derived to accurately predict ABRS or abnormal CT findings in adults (≥15 years) presenting to primary care with suspected, uncomplicated ARS to help clinicians to tailor antibiotic prescribing in this very common condition. A recent paper debunked common myths about risk threshold for prediction models.45 To avoid inappropriate risk stratification of patients, we will calculate continuous risk estimates. Where appropriate, we aim to derive context-specific and clinical sensible risk thresholds by exploring test and treatment thresholds for ARS in a subsequent project.49 50
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors RV is the guarantor. RV conceptualised and designed the study and acquired funding. JGH, MHE and ML contributed to the concept of the study and provided clinical input. JGH, JBR, MHE and ML contributed to the design of the study with JBR and MHE providing methodological expert input. RV drafted the first version of the protocol. All other authors critically revised the draft of the protocol for important intellectual content. All authors provided final approval for submission.
Funding This study was supported by The Netherlands Organisation for Health Research and Development (grant 91618026). The funder did not participate in the design of the study and will have no role in the study conduct, data analysis, interpretation and publication of the data.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.