Article Text
PDF
Abstract
Objective Acute infection is the most common presentation of children in primary care with only few having a serious infection (eg, sepsis, meningitis, pneumonia). To avoid complications or death, early recognition and adequate referral are essential. Clinical prediction rules have the potential to improve diagnostic decision-making for rare but serious conditions. In this study, we aimed to validate a recently developed decision tree in a new but similar population.
Design Diagnostic accuracy study validating a clinical prediction rule.
Setting and participants Acutely ill children presenting to ambulatory care in Flanders, Belgium, consisting of general practice and paediatric assessment in outpatient clinics or the emergency department.
Intervention Physicians were asked to score the decision tree in every child.
Primary outcome measures The outcome of interest was hospital admission for at least 24 h with a serious infection within 5 days after initial presentation. We report the diagnostic accuracy of the decision tree in sensitivity, specificity, likelihood ratios and predictive values.
Results In total, 8962 acute illness episodes were included, of which 283 lead to admission to hospital with a serious infection. Sensitivity of the decision tree was 100% (95% CI 71.5% to 100%) at a specificity of 83.6% (95% CI 82.3% to 84.9%) in the general practitioner setting with 17% of children testing positive. In the paediatric outpatient and emergency department setting, sensitivities were below 92%, with specificities below 44.8%.
Conclusions In an independent validation cohort, this clinical prediction rule has shown to be extremely sensitive to identify children at risk of hospital admission for a serious infection in general practice, making it suitable for ruling out.
Trial registration number NCT02024282.
- PRIMARY CARE
- EPIDEMIOLOGY
- ACCIDENT & EMERGENCY MEDICINE
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
Prospective multicentre validation study in almost 9000 illness episodes in children.
Examining sensitivity and specificity, that is, the proportion of true positives (sensitivity) and true negatives (specificity), which are correctly identified by the four-step decision tree.
Consecutive recruitment in three different settings covering the whole spectrum of acutely ill children seen at first contact.
Measuring standardised clinical features could have led to a workup bias.
Identification of admissions for serious infection depended on the quality of medical records and follow-up.
Introduction
Acute infection is the most common reason for children to attend ambulatory care and represents an important proportion of a general practitioner's workload.1
However, in primary care, <1% of children will be diagnosed with a serious infection.2 The incidence is assumed to be 5–10 times higher at the emergency department (ED).3
Serious infections in children are usually defined as sepsis (including bacteraemia), meningitis, pneumonia, complicated urinary tract infection, bacterial gastroenteritis with dehydration, osteomyelitis and cellulitis.4
These serious infections need to be distinguished from the vast majority of self-limiting infections in children because, although rare in children in developed countries, they are associated with considerable morbidity (eg, hearing loss, neurologic disability) and mortality.5
Furthermore, early recognition could improve prognosis of seriously ill children and prevent avoidable investigations and referrals in children without serious infection.5
Clinicians use signs and symptoms to initially assess the probability of a serious infection and decide on further management. On the basis of a prospective cohort of 4000 children, Van den Bruel et al6 derived a symptom-based four-step decision tree consisting of: the clinician's gut feeling ‘something is wrong’, ‘dyspnoea’, ‘temperature >39.95°C’ and ‘diarrhoea in children aged 1–2.5 years’.
The tree is considered positive if yes to any of these four sequential items is positive, with a sensitivity and negative predictive value (NPV) of nearly 100% in the original derivation study.6 Although the tree also demonstrated high sensitivity in a retrospective validation in another primary care data set using approximations for gut feeling and dyspnoea, prospective validation had not been performed as yet.7
In this study, we aim to prospectively validate this decision tree in a new and independent population of acutely ill children in ambulatory care.
Methods
Setting
This is a diagnostic accuracy study in ambulatory care (defined as general practice, paediatric outpatient clinics or ED).
Patients
Children aged 1 month to 16 years, presenting to a general practitioner (GP) or paediatrician in Flanders, Belgium, with an acute illness for a maximum of 5 days were included consecutively from 15 February 2013 to 28 February 2014. Children were excluded if the acute illness was caused by purely traumatic or neurological conditions, intoxication, a psychiatric problem or an exacerbation of a known chronic condition.
If a physician recruited less than five children over the 1-year study period, the assumption of consecutive inclusion was assumed to have been violated, leading to the exclusion of his or her data from the analysis.
When the same child was recruited twice within 5 days, we considered the second registration a consequence of the same illness episode and discarded the second registration from the analyses.
Index tests
We asked physicians to register diagnostic features based on previous research and consensus of an international team of clinicians and researchers,7 including all items of the National Institute of Health and Care Excellence (NICE) traffic light system, and vital signs (heart and breathing rate, temperature and capillary refill time) and pulse oximetry.7–9
In total, 74 diagnostic features were scored: 28 features obtained by history taking, 36 by clinical examination and 10 items relating to clinical decision-making (see online supplementary file 1).
In addition to the clinical prediction rule, clinicians were asked to rate whether the child appeared seriously ill and whether the parents considered their child's illness different from previous illnesses.6 All features were scored as ‘yes’ when present, ‘no’ when absent, and ‘?’ when they could not be evaluated.
Four-step decision tree
We asked physicians to score variables included in the four-step decision tree, as developed by Van den Bruel et al6 (figure 1).
Four-step decision tree developed by Van den Bruel et al.6 SI, serious infections; GP, general practitioner; yrs, years; red boxes, children testing positive on the decision tree; green box, children testing negative on the decision tree.
‘Something is wrong’ was defined as a subjective gut feeling of the physician that something is out of the ordinary. ‘Dyspnoea’ was defined as difficult or laboured breathing. ‘Body temperature’ was defined as the highest body temperature measured by parents or the physician during the illness episode. Before analysis, 0.5°C was added to temperatures measured under the axilla, or with a tympanic thermometer.10 ,11
‘Diarrhoea’ was defined as loose or watery stools, increased in frequency and volume.12
Vital signs
Temperature, respiratory rate, heart rate, oxygen saturation and capillary refill time were measured, each according to their respective standardised method.13
All GPs were provided with a paediatric finger pulse oximeter (CMS50QA, Contec Medical Systems, China) for use in children at least 3 years old (due to device limitations). Paediatricians were given the choice to use the provided finger pulse oximeter, or rather use their own large-size pulse oximeter appropriate for all ages.
Target condition
The target condition was hospital admission (>24 h) for a serious infection, which was one of the following:
Sepsis (including bacteraemia) with pathogenic bacteria isolated from haemoculture;
Meningitis with a positive lumbar puncture (pleocytosis in cerebrospinal fluid or identification of bacteria or a virus);
Appendicitis with a positive histological diagnosis;
Pneumonia with an infiltrate seen on chest X-ray;
Osteomyelitis (pathogens from bone aspirate or MRI or bone scan suggestive for osteomyelitis);
Cellulitis (acute suppurative inflammation of subcutaneous tissues);
Bacterial gastroenteritis with dehydration (pathogen isolated from stool culture);
Complicated urinary tract infection (>105/mL pathogens of a single species isolated from urine culture and systemic effects such as fever).
The outcome was verified by three complementary methods:
A search of the electronic medical records of all regional hospitals;
An interview with each participating GP;
A diary completed by parents for children recruited in general practice, recording the date of recovery.
If methods II and III showed evidence of a hospital admission initially not captured by method I, attempts were made to obtain information for this additional hospital admission. Children were considered as not having a serious infection if hospital records showed no evidence of serious infection. In cases when no definitive adjudication could be made on the basis of the above criteria, an adjudication committee consisting of clinicians with expertise in acute paediatric care assigned outcome by consensus, using all available information.
Sample size
Sample size calculations were based on the assumption that the prevalence and diagnostic value of the decision tree would be similar to those reported by Van den Bruel et al.
Assuming a prevalence of 0.9%, recruiting 6500 children would result in 59 cases. This would provide us with an error margin of 12% around an expected sensitivity of 97% (95% CI 85% to 100%).14
Statistical analysis
Accuracy of individual features
First, the accuracy of each diagnostic feature was analysed and reported using sensitivity, specificity, likelihood ratios and predictive values for both the GP and specialist setting (paediatric outpatient and ED). A correction of 0.5 was added to every cell in case of an empty cell in a 2×2 table.
We constructed receiver operating characteristic (ROC) curves for temperature, breathing rate, heart rate and oxygen saturation. In addition, these features were dichotomised on the basis of NICE guidance.13
Validation of the four-step decision tree
The four-step decision tree for any serious infection was validated in the entire group and in the three predefined settings separately being general practice, ambulatory paediatric care and EDs. In addition, we performed subgroup analyses for three infectious categories: pneumonia, complicated urinary tract infections and sepsis/meningitis.
We applied the same missing value categorisations for every decision tree variable as in the derivation study, namely missing values in the same category as ‘no’ or ‘unknown’.6
Optimised thresholds
We optimised the tree by recalibrating the thresholds of body temperature and age for the current data, using classification and regression tree (CART) analysis, and maximising sensitivity with a weighing factor of 75 for false negatives, while keeping the structure of the tree constant.
Pragmatic thresholds
To facilitate implementation in routine care, we created a decision tree with easy-to-remember thresholds for temperature and age:
Temperature of 40°C in the GP setting or 39.5°C in the specialist setting (instead of 39.95°C or 39.2°C);
Age below 3 years of age (instead of 3.3).
Sensitivity analyses were performed comparing the results of all illness episodes versus the first illness episode only to explore the effect, if any, of clustering based on recurring admissions in the same children.
Analyses were performed with Stata software (V.11.2; Stata Corp) and JMP Statistical Discovery (version Pro 11.1.1; SAS Institute Inc).
Ethics
Formal written informed consent was obtained for each child. We provided age-appropriate information leaflets and assent forms for minors below and above 12 years of age.
Results
Baseline characteristics
Children were recruited across Flanders at 92 GP surgeries, 6 outpatient paediatric clinics and 6 EDs, involving 276 physicians (170 GPs and 106 paediatricians): 33% were male, with a median clinical practice experience of 13 years (range 0–40 years).
We included 8664 new illness episodes in 7355 children between 15 February 2013 and 28 February 2014 (figure 2); 1322 children were included with two separate illness episodes, 525 children with three episodes and 379 with four or more episodes.
Flow chart of inclusions in recruited children (GP, general practitioner).
The children's median age was 2 years (IQR 1–4.1; total age range: 1 month to 16.9 years) and 3897 were boys (53%).
Outcome verification
We identified 1025 admissions to hospital for >24 h, of which 283 were for a serious infection (table 1). No patient died during this study.
Baseline characteristics for children with or without a serious infection
The prevalence of serious infections was 3.3% (95% CI 2.9% to 3.7%), increasing significantly from 0.3% (95% CI 0.1% to 0.6%) in general practice over 2.6% (95% CI 2.0% to 3.2%) in paediatric outpatients to 7.5% (95% CI 6.5% to 8.5%) in the ED setting.
There were only 11 cases of serious infection in the GP setting, of which 8 had pneumonia, 2 had complicated urinary tract infections and 1 had appendicitis. Of the 27 cases of sepsis and meningitis identified in the specialist setting, 16 children had a viral meningitis (mostly enterovirus or herpes simplex), 1 had a bacterial meningeal infection (group B Streptococcus), 5 had Streptococcus pneumoniae sepsis, 1 had Haemophilus influenzae type B sepsis (despite evidence of prior immunisation), 1 had Neisseria meningitidis sepsis and 3 had uropathogenic sepsis (eg, Escherichia coli).
Accuracy of individual features
In the GP setting, only gut feeling, fever >1 day, eating or drinking less, and being less active had sensitivities above 80% (see online supplementary file 2). In ambulatory paediatrics and the ED, overall sensitivities were even lower, with only fever duration >1 day and fever not reducing to normal temperatures after antipyretics having sensitivities above 80%.
Red flags (specificity >99%) included reduced consciousness, bloody diarrhoea, inconsistent speech, abnormal skin turgor and fontanel tension, petechial rash, meningeal irritation, nasal flaring, cyanosis, reduced peripheral circulation and peritoneal irritation.
The areas under the ROC curves (AUC) for temperature, breathing and heart rate per setting were low (0.58–0.69), except for breathing rate in the GP setting (AUC=0.80; 95% CI 0.63 to 0.97), probably due to the high number of pneumonia cases in this setting (see online supplementary file 3).
Validation of the four-step decision tree
Figure 3 shows all diagnostic properties of the decision tree per setting. In general practice, sensitivity was 100% (95% CI 71.5% to 100%) and specificity 77.7% (95% CI 76.2% to 79.1%), and 23% of children seen by the GP tested positive on the tree. Sensitivity and specificity were lower in both specialist settings, although CIs overlap.
Validation results of the four-step decision tree for all serious infections. GP, general practice; Paed, paediatric outpatient clinic; ED, emergency department; prevalence, prevalence of serious infection within this setting; LR, likelihood ratio; PV, predictive value.
The diagnostic value of the tree for pneumonia, urinary tract infection and sepsis/meningitis is reported in table 2. For pneumonia, the diagnostic characteristics were almost identical to those for the composite outcome of serious infections, which is unsurprising since pneumonia cases made up 58% of all serious infections. Specificity was higher for complicated urinary tract infection (88.5%, 95% CI 87.3% to 89.5%).
Results for pneumonia, urinary tract infection and sepsis/meningitis
For sepsis and meningitis, sensitivity was 69.6% (95% CI 47.1% to 86.8%) in the ED where a large majority of cases were seen.
Optimised and pragmatic thresholds
Figure 4 illustrates the threshold changes when (1) optimising the splits of the decision tree variables using CART, and (2) applying the pragmatic approach.
Validation results after applying optimised and pragmatic thresholds to the four-step decision tree. Yellow boxes, threshold changes after applying the optimisation using classification and regression tree analysis (CART); orange boxes, additional threshold changes after applying the pragmatic approach; sensitivity and specificity are given for every tree with their respective 95% CIs in brackets; y, years.
In the GP setting, using the pragmatic ‘temperature’ threshold of 40°C, sensitivity remained at 100% (95% CI 71.5% to 100%) and specificity was 83.6% (95% CI 82.3% to 84.9%), which is higher than the value obtained with the original tree (but lower than that with the optimal threshold (40.7°C) of 85.4% (95% CI 84.1% to 86.6%)).
In the specialist settings, these strategies increased sensitivity up to 92.0% (95% CI 83.4% to 97.0%), however at the expense of a lower specificity up to 44.8% (95% CI 43.0% to 46.7%).
The sensitivity analyses revealed similar sensitivities and specificities with overlapping CIs for all settings and chosen thresholds in the 7355 first hospital admissions only (84.9% of all episodes).
Discussion
Summary
Validating the four-step decision tree in a new and independent but similar population 9 years after the derivation study demonstrated a sensitivity and NPV of 100% in the GP setting, thus confirming its usefulness to rule out serious infections in general practice. This perfect sensitivity suggests that current practice could be improved by using the tree since 4 of the 11 children with a serious infection were initially not identified at first presentation.
A clinical decision tree that is able to rule out serious infections is especially useful in low prevalence situations. There were only 11 hospital admissions for a serious infection in the GP setting (0.3%), most of which were pneumonia (8 cases), and there were no cases of sepsis or meningitis. This very low prevalence is comparable to that in the derivation study (0.4% in the GP setting).6
In the paediatric outpatient clinic and ED settings, the tree did not provide a useful rule-out value, although sensitivity rose considerably to 92% in the paediatric outpatient clinic setting if the thresholds were optimised.
Using pragmatic thresholds allowed us to enhance overall clarity and ease of use, without losing diagnostic accuracy in the GP and paediatric outpatient settings.
Strengths and limitations
This was a prospective multicentre validation study of the four-step decision tree in a large and similar population of children. We included almost 9000 illness episodes, which makes this study one of the largest cohorts of children with acute illness.15 ,16
The Belgian healthcare system allows for unlimited access to paediatric outpatient clinics and EDs, alongside general practice. This provides us with a unique opportunity to examine acutely ill children in different urgent-access settings.
To ensure identification of all admissions for serious infection, the outcome was measured through three complementary strategies. Nonetheless, this verification depended on the quality of medical records and follow-up.
We asked the participating physicians to record a list of standardised clinical features, which could lead to additional testing and potentially facilitate a diagnosis of serious infection (workup bias), inflating sensitivity and specificity.17 For this reason, the outcome was defined as hospital admission for a serious infection, rather than hospital admission or serious infection in isolation.
Comparison with existing literature
Very few studies have validated clinical prediction rules of vital signs and symptoms in acutely ill children in primary care.5 Most research has been performed in secondary care, with varying results.8 ,15 ,18 ,19 To the best of our knowledge, there is only one prior study that conducted a retrospective validation in a low prevalence setting and found a sensitivity of 90% for the four-step decision tree.7
Implications for clinicians
Signs and symptoms are the first available tests to support clinical decision-making in primary care.20 The clinician's feeling that ‘something is wrong’ (gut feeling) is confirmed to be an important predictor of serious infection.6 Other red flags, such as cyanosis, poor peripheral circulation, meningeal irritation and petechial rash, are useful as they raise the probability of serious infections, but are rarely present.5
Physicians often choose not to measure vital signs, assuming them to be normal. However, vital signs might act as a red flag for serious infection, as suggested by most recent guidelines.13 ,21 The results of our study confirm this assumption.
The decision tree consisting of a gut feeling, dyspnoea, temperature >40°C and diarrhoea is able to safely exclude serious infection that warrants hospital admission in children in general practice.
However, 17% of acutely ill children will be labelled as potentially at risk of a serious infection, of whom 98% will be false positive. Consequently, appropriate additional strategies such as rapid laboratory testing or watchful waiting with adequate safety netting need to be put in place to reduce unnecessary referrals.
Implications for research
Blood tests are currently rarely performed in acutely ill children in primary care, because the result becomes available too late to influence clinical decision-making. In adults, rapid laboratory tests such as C reactive protein have been shown to be useful in improving the management of lower respiratory tract infections.22
Future research might be able to establish the exact role of such tests in the management of acutely ill children presenting to ambulatory care.
Acknowledgments
The authors would like to thank all participating GPs. They would like to thank all participating paediatricians, at UZLeuven under the supervision of Professor Christel Van Geet and Professor Dominique Bullens, at AZTurnhout under the supervision of Dr Luc Pattyn, at Jessa Hasselt under the supervision of Dr Marc Raes, at UZGent under the supervision of Professor Frans De Baets, at AZ Maria Middelares under the supervision of Dr Jasmine Leus and Dr Katrien De Schynkel, at AZ Sint-Vincentius Deinze under the supervision of Dr Lut Van den Berghe, at Stedelijk Ziekenhuis Roeselare under the supervision of Dr Karin Decaestecker, and at Heilig Hart Ziekenhuis Roeselare under the supervision of Dr Karl Logghe. The authors would also like to thank Frederick Albert, Greet Delvou and Annelien Poppe for the daily follow-up during the study. Last but not least, they would like to thank all the children and parents who took part in this study.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
- Data supplement 1 - Online supplement
Footnotes
Contributors JYV, MBL, ADS, AVdB, BA and FB conceived the study. JYV, MBL and TDB supervised the data collection and performed the data follow-up and data cleaning. JYV performed the analyses, which were discussed with AVdB, BS and FB. JYV drafted this report and MBL, TDB, ADS, DMAB, BA, BS, AVdB and FB co-drafted and commented on the final version. All authors have read and approved the final manuscript.
Funding This study was funded by the National Institute for Health and Disability Insurance (RIZIV, Belgium) under reference CGV n° 2012/235 and the Research Foundation Flanders (FWO) under reference n° G067509N.
Competing interests DMAB is a recipient of a senior clinical investigator fellowship from the Research Foundation Flanders (FWO).
Ethics approval The study was approved by the Ethical Review Board of the University Hospitals/KU Leuven under reference ML8601, as well as by all participating hospitals. The study authors obtained ethics approval from their regional research ethics committees before the study for the initial data collection of the included data sets.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.