Article Text
PDF
Abstract
Introduction WHO treatment guidelines are widely recommended for guiding treatment for millions of children with pneumonia every year across multiple low-income and middle-income countries. Guidelines are based on synthesis of available evidence that provides moderate certainty in evidence of effects for forms of pneumonia that can result in hospitalisation. However, trials have included fewer children from Africa than other settings, and it is suggested that African children with pneumonia have higher mortality. Thus, despite improving access to recommended treatments and deployment with high coverage of childhood vaccines, pneumonia remains one of the top causes of mortality for children in Kenya. Establishing whether there are benefits of alternative treatment regimens to help reduce mortality would require pragmatic clinical trials. However, these remain relatively expensive and time consuming. This protocol describes an approach to using secondary analysis of a new, large observational dataset as a potentially cheaper and quicker way to examine the comparative effectiveness of penicillin versus penicillin plus gentamicin in treatment of indrawing pneumonia. Addressing this question is important, as although it is now recommended that this form of pneumonia is treated with oral medication as an outpatient, it remains associated with non-trivial mortality that may be higher outside trial populations.
Methods and analysis We will use a large observational dataset that captures data on all admissions to 13 Kenyan county hospitals. These data represent the findings of clinicians in practice and, because the system was developed for large observational research, pose challenges of non-random treatment allocation and missing data. To overcome these challenges, this analysis will use a rigorous approach to study design, propensity score methods and multiple imputation to minimise bias.
Ethics and dissemination The primary data are held by hospitals participating in the Kenyan Clinical Information Network project with de-identifed data shared with the Kenya Medical Research Institute (KEMRI)-Wellcome Trust Research Programme for agreed analyses. The use of data for the analysis described received ethical clearance from the KEMRI scientific and ethical review committee. The findings of this analysis will be published.
- comparative effectiveness
- propensity scores
- pneumonia
- observational
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This study will be used as a platform to explore effectiveness of alternative treatments in routine care in a low-income setting to improve health outcomes for children
The analysis will be limited to the variables in the observational dataset—and therefore risk bias due to unmeasured key variables
The influence of any resulting bias, to alter results, will however be assessed through the use of alternative methods as instrumental variables
Introduction
Kenya has developed and disseminated national treatment guidelines largely drawing on those of WHO for a number of childhood diseases including pneumonia.1 2 These pneumonia guideline recommendations are based on synthesis of available evidence that provides moderate certainty in evidence of effects of treatments for forms of pneumonia that can result in hospitalisation.1 2 Such guidelines have been shown to be effective in reducing pneumonia-related mortality, and thus, Kenyan clinicians are supposed to use them in routine practice to treat pneumonia (and other diseases).3 4 However, although the guidelines are based on the best available evidence, the evidence available from trials conducted in Africa remains limited.5 There has also been little thorough investigation of the effectiveness of treatments in non-trial populations in routine settings that may often differ from those enrolled in formal clinical trials. For example, many children admitted with pneumonia may have comorbidity that might exclude them from trials.6 These issues can prove problematic when making national guidelines where study generalisability can be contested.7
The WHO and Kenyan pneumonia treatment guidelines are implicitly based on risk stratification of illness with children deemed at higher risk of severe illness and mortality offered broad spectrum antibiotic regimens and those at lower risk, narrow spectrum antibiotics.2 8–10 This risk stratification approach is operationalised by requiring clinicians to look for specific features in the clinical history and examination that are used to define illness severity and therefore recommended treatment (box 1). Previous studies conducted in Kenya have, however, indicated that clinicians do not always follow guideline recommendations in treating pneumonia.4 Variation from the guideline recommended approach can occur at the point of pneumonia severity assignment (clinicians do not follow the rules linking clinical signs and severity category) and at the point of treatment assignment (clinicians do not follow the rules linking treatment and severity). This variability in treatment assignment provides the opportunity for comparative effectiveness evaluation if similar populations of children with pneumonia are prescribed different treatments. Clinicians may create such a situation by not following recommendations because they have inadequate knowledge or if they believe (potentially contrary to the evidence) that certain treatments result in better health outcomes.
Pneumonia treatment algorithm
The pneumonia severity classification that was recommended by Kenyan guidelines up to March 20169 (and previously by WHO guidelines1) defined the following three severity classes:
Very severe pneumonia: If a child had either oxygen saturation less than 90% or central cyanosis or was grunting or unable to drink or not alert, then s/he was classified as having very severe pneumonia, put on oxygen and treated with a combination of gentamicin and penicillin. (The new WHO2 and Kenyan guidelines9 renamed this class as ‘severe pneumonia’—and currently recommend treatment with a combination of ampicillin (or penicillin) with gentamicin plus oxygen.)
Severe pneumonia: If a child had lower chest wall indrawing (but did not have any of qualifying signs for very severe pneumonia above) and was alert, then s/he was to be classified as having severe pneumonia and be treated with benzyl penicillin only. Note: the term indrawing pneumonia is hereafter used in this protocol to define this category of children to avoid confusion.
(Non-severe) Pneumonia: If a child had none of the mentioned signs but had cough or difficulty breathing and a respiratory rate greater than or equal to 50 breaths/min (for age between 2 and 11 months) or respiratory rate greater than or equal to 40 breaths/min (for age above 12 months), then s/he was classified as having non-severe pneumonia and treated with co-trimoxazole or amoxicillin if previously treated with co-trimoxazole. (The current WHO and Kenyan guidelines collapsed severity classes 2 and 3 into one category referred to as ‘non-severe pneumonia.’ This group of patients is currently treated with oral amoxicillin—partly informed by a local trial19).
In particular, a previous study showed that clinicians overprescribed gentamicin, adding this to penicillin for the treatment of pneumonia characterised by lower chest wall indrawing but no other signs of severe illness instead of penicillin alone as was recommended.* Therefore, this protocol is for a study that seeks to explore whether there is any benefit from adding gentamicin to penicillin in treating children with indrawing pneumonia. Such a benefit could accrue if bacterial causes of pneumonia that were previously (prior to introduction of new vaccines) proportionately less common (eg, Staphylococcus aureus and Gram-negative bacteria) are now accounting for an increased proportion of pneumonia deaths; as in such cases, the addition of gentamicin might provide effective treatment for a broader spectrum of pathogens. Tackling this question is of importance as WHO has recently changed indrawing pneumonia treatment guidance based on trials that suggest equivalence of oral amoxicillin and injectable penicillin.11–14 New guidance recommends outpatient oral treatment for a population of children previously admitted to hospital.10 However, mortality from pneumonia has been reported to be higher in African settings15 16 despite the increasing use of multiple vaccines spanning measles, pertussis, HiB and pneumococcal conjugate vaccines. It remains possible therefore that, for a small number of children, a broader-spectrum antibiotic regimen might be of benefit. This study addresses this question that has not been the subject of prior community and pragmatic clinical trials.
Objectives
Primary
Experiment 1: To compare the effectiveness of injectable penicillin versus penicillin plus gentamicin (both injectable) in treatment of indrawing pneumonia; where severity level is constructed (imputed) using data recorded on each child’s clinical signs (hospitals use a structured record form that supports recording of signs highlighted in guidelines) such that severity classification is consistent with guideline recommendations.
Secondary
Experiment 2: To compare effectiveness of injectable penicillin versus penicillin plus gentamicin in treatment of indrawing pneumonia; where we use clinician assigned severity level.
Experiment 3: To compare effectiveness of injectable penicillin versus penicillin plus gentamicin in treatment of all cases of pneumonia admitted to hospital.
Experiment 1 will be primary as it most approximates a typical randomised trial where recruitment would be based on specified clinical signs. This scenario will provide an evaluation of alternative therapies within a guideline class (where children have very similar clinical signs) and thus is the best mimic of a prospectively designed comparative evaluation in which clinicians stick to the rules of severity classification (see Agweyu et al 18 for an example of a randomised controlled trial in Kenya that this would be similar to—where classification is based on clinical signs). Recommended treatment for this disease classification was penicillin alone; treatment with combination therapy may therefore represent overtreatment. Alternatively, the combination treatment that provides broader antimicrobial cover could provide an advantage in a small proportion of cases that would only be detected in moderately large studies—where the addition of gentamicin offers improved treatment for specific organisms not susceptible to penicillin alone.
Experiment 2 will provide a test of alternative therapies among those where clinicians used their own judgement (possibly including gut feeling) to classify and treat19 and have on occasions (potentially) over-ridden or ignored the guideline recommendations. In this case, although the same label of indrawing pneumonia is given to all, the treatment selected may be an indicator of perceived severity and there may be a potential bias as a result, and the Propensity Score (PS) distributions (see below) may help demonstrate this and in theory may overcome this potential bias. Here if there is no clinically relevant difference between treatments within a group of patients that reflects clinicians’ actual classification decisions, this could reassure them that monotherapy with penicillin (or amoxicillin) would be acceptable.
Lastly, experiment 3 is an extension of the logic of experiment 2. To date, there have been no pragmatic trials of penicillin alone compared with alternative combination therapies for all forms of inpatient pneumonia, and addressing this question may be relevant for two reasons. First, the population of children admitted with severe forms of pneumonia is now largely one that has received Haemophilus influenzae Type B and pneumococcal conjugate vaccines that have likely changed the aetiology of this illness. Second, if clinicians are poorly trained and unable to classify illness severity—resulting in non-adherence to guidelines—it would be useful to explore the potential impact of this across all levels of severity of pneumonia. This analysis has the largest numbers of subjects.
Methods and analysis
To answer these three questions, we will use the Kenyan Clinical Information Network (CIN) dataset that provides observational data on all admissions to 13 Kenyan county hospitals (box 2).
Clinical Information Network
Clinical Information Network (CIN) was initiated to improve data availability from secondary care in paediatrics and as a model for demonstrating the value of routine data in improving quality of care in the county (formerly district) hospitals. These hospitals typically have a single paediatrician leading services predominantly provided by junior clinical teams. Data in these hospitals are collected prospectively postdischarge by trained data clerks, guided by well-defined standard operating procedures, under close supervision by the hospital medical records department and the research team. It is worth noting that the research team has no personnel checking quality of clinical process and whether clinicians correctly document what they do. However, the patient record is the formal (and legal) document describing clinical condition and management. These documents are used for data abstraction, and they include patient files with standardised Paediatric Admission Record forms, treatment sheets, discharge summary forms, laboratory reports and clinician notes. The collected data are used to assess documentation of history, physical examination, diagnosis, laboratory investigations, treatment and discharge plans. Feedback to hospitals as part of the CIN activities has helped improve the quality of clinical data.21 The description of hospital selection and their populations of patients is detailed in Ayieko et al.6
The analysis will proceed in two stages: design and outcome analysis as suggested by Rubin20 as an objective way for analysing observational datasets.
Study design
This will be an observational study conducting secondary analyses of data routinely collected from hospital paediatric wards in Kenya’s CIN. The design process for the three experimental scenarios will be similar and broadly consists of the following steps suggested by Rubin:20
Definition of inclusion and exclusion criteria.
Understanding the pneumonia diagnosis and treatment assignment processes. This is to help understand key and auxiliary variables required for analysis.
Verification of sample size if sufficient for any meaningful analyses.
Creation of comparable treatment arms—which will be addressed analytically aiming to overcome non-random treatment assignment and deal with missing data.
Outcome analysis follows after conceptualisation of design in steps 1–4.
Inclusion and exclusion
This analysis will include all children aged 2–59 months and will exclude children with any comorbidity of HIV, meningitis, tuberculosis and/or acute malnutrition as there are specific antibiotic treatment rules for these children that supersede those for pneumonia. Specifically, Kenyan guidelines for the inpatient treatment of pneumonia in children who are HIV infected recommend only combination therapy. Importantly therefore, children with other comorbidities such as mild anaemia, diarrhoea and malaria are not necessarily excluded from the analysis.
Understanding the diagnosis and treatment assignment rules for paediatric patients with pneumonia
Clinicians are supposed to use guidelines widely disseminated as the ‘Basic Paediatric Protocols’ in Kenya9 that are adapted from WHO guidance, based on available evidence and developed by consensus by a national guideline panel.21–23 In standard practice, the process of treatment assignment happens in three steps. First, there is assessment and documentation of each clinical sign. Step two involves integration of clinical information into severity classification, and in step three, severity classification is translated into a treatment assignment (see box 1). In Kenya, as in many low-income and middle-income countries, these recommendations reflect the absence of access to further diagnostic tests. Thus, pulse oximetry, blood culture or tests for inflammatory markers are not routinely available.6 As indicated above clinicians, may fail to adhere to guideline recommendations by making errors or over-riding recommendations at any of the three steps of assessment, severity classification and treatment assignment. However, based on the clinical symptoms and signs recorded, it is possible to assign a severity classification (and thus expected treatment) based on the data. It is a data-informed and investigator-assigned classification as indrawing pneumonia that is used in the primary analysis (experiment 1).
Variables to be used in analysis
Outcome variable
Mortality will be used as the outcome variable in all the three experiments.
Independent variables
These variables are grouped into key and auxiliary. Key variables are defined as those that should influence pneumonia severity classification and hence treatment based on the treatment protocol9 (box 1). Auxiliary variables are defined as those that might, a priori, be expected to influence treatment assignment based on clinical reasoning (eg, they might make a clinician concerned for severe illness), although according to the formal rules (the guidelines) they are not considered reasons to alter treatment assignment. Such auxiliary variables were identified from those clinical symptoms and signs that are routinely collected within CIN. See table 1 for a summary of key and auxiliary variables that will be used in the analyses.
Summary of key and auxiliary independent variables for experiments 1, 2 and 3
Sample size verification
Here, sample size verification uses the formula cited by Wittes24:
where ns is the size of smaller group, k is the ratio of larger group to smaller group, p1−p2
is the clinical difference in proportions of the outcome, Zβ
corresponds to power of 80%, Z1
−α/2
corresponds to two-tailed significance level (1.96 for α = .05), 
corresponds to average of outcome proportions in two groups.
The value for 
is estimated from studies—two of which formed evidence for earlier WHO indrawing pneumonia treatment guidelines. Table 2 shows the number of deaths per treatment arm reported in these studies.
Summary of some of pneumonia studies that informed previous WHO guidelines
For assessment of sample size for indrawing pneumonia experiments, a weighted† 
of 0.041 from these studies is used. The ratio r is varied between 1 and 3. Figure 1 was generated by fixing power and significance level at 80% and 5%, respectively. Estimates of 
derived from WHO studies were substituted in the sample size formula and data simulated in order to see what detectable differences would be achieved by different sample sizes. A total sample size of about 4000 would be sufficient to detect a minimum difference of 1.5% (absolute difference, eg, a reduction of mortality from X% to X−1.5%) in any of these experiments.‡
Sample size verification.
Statistical and outcome analysis
Statistical analysis will proceed in the following four steps:
Step 1
Subset of patients of interest for the experiments will be obtained.
Experiment 1: First, missing clinical signs data will be multiply imputed§ (excluding outcome data), and then key clinical signs data will be used to impute (construct) a pneumonia severity level for all patients based on the algorithms in the pneumonia treatment protocol.9 Thereafter, a subset of patients with guideline-defined indrawing pneumonia (for each of the imputed datasets) will be obtained for further analyses.
Experiment 2: A subset of patients with indrawing pneumonia (with severity as indicated by the clinicians) will be obtained from the raw dataset, and clinical signs data will be imputed using multiple imputation (without the outcome data).
Experiment 3: The raw dataset containing all the patients with all forms of pneumonia severity will be used, and clinical signs data will be imputed using multiple imputation (without the outcome data).
Step 2
Patients in the alternative treatment arms will be matched using PS methods to overcome non-random treatment allocation. Standardised mean differences (and where necessary density plots) will be used as diagnostic checks for covariate balance and overlap25 26 between penicillin and penicillin plus gentamicin treatment groups. PS methods that use all the data (PS optimal full matching, weighting and subclassification) will be examined in experiments 1 and 2 (on each imputed dataset), and the method that results in the minimum average absolute standardised mean differences for the majority of the variables and retains the largest number of patients in the analysis will be considered appropriate.27 Meanwhile, only PS subclassification will be used for experiment 3. As experiment 3 aims to investigate comparative effectiveness in all cases of pneumonia, PS will be used as a proxy for disease severity. Thus, patients with lower PSs will be considered less ill, while those with higher PSs will be considered more ill (grouped in PS subclasses for analysis).
Step 3
Conducting outcome analysis.
For each imputed dataset (per experiment), outcome analysis will aim to investigate treatment causal effects across all the hospitals. Bayesian log binomial regression models28 will be used to estimate overall treatment effects.¶ A hospital variable will be modelled as a fixed effect in the log binomial regression that measures treatment effects on pooled data. These models will be fitted on each imputed dataset (adjusting for other variables used in PS models), and results will be pooled using Rubin rules.29
Step 4
Sensitivity analysis will be conducted to investigate the effects of unmeasured confounders and validity of estimates obtained through multiple imputation. PS methods generate matched treated and (active) control patients whose distribution of measured covariates are as similar as possible. However, two patients with similar covariate distribution may differ in terms of unmeasured variables—and this may introduce bias in estimated treatment effects.30 On the other hand, if outcome and explanatory variables have missing data, then inclusion of outcome data in multiple imputation may contribute minor information in the substantive (outcome) model.31
Exploring effects of unmeasured confounders
Sensitivity analysis for unmeasured confounders will involve the use of an instrumental variable (IV)32—weekend admission and PS trimming.33 A few IV sources in health studies have been described by Baiocchi et al.34 These include distance to specialty, genes, insurance plan, timing of admission, calendar time and preference-based IVs. Of relevance to this analysis would be timing of admission IVs. A study conducted by Berkley et al 35 in a Kenyan hospital demonstrated that children who were admitted during the weekend experienced higher mortality compared with those admitted during the weekdays—which is a possible indication of poor quality of care and treatment during the weekend. In other words, it is anticipated that children admitted during the weekdays would have better health outcomes. This, in theory, implies that the type of treatment and care received depend on the day of admission—and which later determines the type of health outcome of the patient.
Examining validity of multiple imputation
Analysis steps 1–3 above will exclude outcome data in the imputation model; however, sensitivity analysis will include models in which the outcome variable is included in the imputation approach. This will aim to investigate if including outcome data in the imputation model has an influence.**
Ethics and dissemination
The primary data are held by hospitals participating in the Kenyan CIN project with de-identifed data shared with the Kenya Medical Research Institute (KEMRI)-Wellcome Trust Research Programme for agreed analyses. The analyses described in this protocol are part of this larger project (CIN), which was approved by the KEMRI scientific and ethical review committee (protocol number 2465). This committee agreed the use of de-identified patient data derived from retrospective case record review without gaining individual patient consent as is common practice in service evaluation research. The findings will be useful in understanding the external validity of current treatments—and will provide a platform for doing more similar analyses for different (combinations of) treatments. The results of this analysis will be shared with the Kenyan Ministry of Health and will inform discussions on national pneumonia treatment guidelines to which the research team have made major prior contributions. The work will also be submitted for publication.
References
Footnotes
↵* The fact that inadequate knowledge in handling childhood pneumonia may result in inconsistent treatment allocation is supported by a survey conducted in seven developing countries showing that 56% of nurses and doctors had inadequate knowledge in managing pneumonia in children.17
↵ † Weighting was done using the total sample sizes per experiment.
↵ ‡ A sample size of at least 4000 would be required for experiment 3 as this is the minimum sample for experiments 1 and 2 which are nested in experiment 3.
↵ § For the three experiments, 20 datasets will be multiply imputed using chained equations.
↵ ¶ Bayesian models will be used to overcome any bias due to sparsity of data as PS subclassification in itself reduces the effective sample size.
↵** The primary interpretations will consider results of multiple imputations without outcome if results differ from those of MI with outcome—as is the standard recommendation to analysis of observational datasets by Rubin.20
Contributors LM did an initial draft of this manuscript with the support of RP, EM and ME. Thereafter, all authors edited subsequent versions and approved the final copy.
Funding We are grateful for the funds from the Wellcome Trust (no 097170) that support ME through a fellowship and additional funds from a Wellcome Trust core grant awarded to the KEMRI-Wellcome Trust Research Programme (no 092654) that supported this work. LM is supported by a Nuffield Department of Medicine Prize DPhil Studentship and Clarendon Scholarship (Oxford University). The funders had no role in drafting or submitting this manuscript.
Competing interests None declared.
Ethics approval Kenya Medical Research Institute Scientific and Ethical Review Committee.
Provenance and peer review Not commissioned; externally peer reviewed.