Ethics Statement

Ethical approval was obtained from the Hospital Ethics Committee (Pakistan Institute of Medical Sciences), the Ethical Review Committee of the International Centre for Diarrhoeal Disease Research, Bangladesh, and the WHO Ethics Review Committee.

We employed a two-stage written consent procedure: caregivers were informed about the study and permission was obtained first at the time of diagnosis and enrollment, and then again at the start of the follow-up home interviews.

Setting

DHS and MICS surveys are conducted very widely throughout low- and middle-income countries and in very diverse settings, which range from rural settings served by community services to urban and peri-urban settings, where the first level health services are provided by local hospital outpatient departments. We thus conducted the study in three diverse urban and rural settings.

The study was conducted in two countries: Pakistan and Bangladesh. In Pakistan, study participants were recruited in an urban setting, from the outpatient department of Children's Hospital, Pakistan Institute of Medical Sciences, Islamabad, Pakistan. In Bangladesh, the participants were recruited from both urban (Dhaka Shishu Hospital, a tertiary pediatric hospital) and rural (community-based recruitment in four unions [Mirzapur, Gorai, Bhatgram, and Jamurki] of Mirzapur, 63 km north of Dhaka) settings. The latter setting involved initial identification of pneumonia and “no pneumonia” cases during weekly home visits by Village Health Workers, later reassessed by study physicians according to WHO guidelines and enrolled.

Study Design

The study identified and recruited two groups of children with acute respiratory infections: those who were confirmed to have pneumonia and those who did not have pneumonia. Two to four weeks after recruitment caregivers were surveyed to assess the accuracy of their recall of the diagnosis and treatment given. This enabled an assessment to be made of the degree to which DHS and MICS measures of antibiotic treatment in those with reported symptoms of pneumonia were valid measures of the antibiotic treatment of true pneumonia in a study population.

The study had two phases. Phase 1 involved recruiting large numbers of children with symptoms of acute respiratory infection and having trained study physicians establish whether pneumonia (the reference standard for the test) was present or absent. We purposefully selected hospital outpatient departments as two of the three study sites to increase the probability that we would find sufficient numbers of children presenting with symptoms of acute respiratory infection, and to allow careful training and monitoring of the study physicians to ensure that the reference standard was robust. At the third, rural site in Bangladesh, patients were also identified in the community. Phase 2 involved having trained field workers interview the caregiver of each child in their home using DHS and MICS algorithms and alternative tools, either 2 or 4 wk after their recruitment.

Participants and Selection Criteria

The study participants were children 0–59 mo with physician-diagnosed clinical pneumonia or “no pneumonia” and their caregivers. A single WHO technical expert (S. A. Q.) oversaw staff training and study monitoring to ensure comparability of case definitions across sites.

Enrollment and Follow-Up

Data Management and Analysis

After double data entry and data cleaning, descriptive statistics were used to assess the socio-demographic and clinical characteristics of the study children. Sensitivity and specificity were calculated to assess the discriminative power of each measurement tool (DHS, MICS, pneumonia score questionnaire, pneumonia video) and for various combinations of tools.

For the pneumonia score, a composite score was developed by adding one point for each symptom or sign present. Sensitivity and specificity were calculated for each of the composite scores (ranging from lowest to highest). The intent was to identify a cutoff that can be used to reach a decision on the presence or absence of pneumonia. SPSS 11.0 (SPSS Inc.) and STATA 10 (StataCorp) were used in Pakistan and Bangladesh, respectively.

Sample Size Calculation

Based on baseline estimates of sensitivity of 60%–70% and specificity of 70%–90% of mother/caregiver recall of symptoms of acute respiratory infection or suspected pneumonia in predicting true pneumonia, it was estimated that 300 children under 5 y with physician-diagnosed pneumonia and 300 with “no pneumonia” should be enrolled at each site in order to estimate sensitivity and specificity with a precision of ±5%.

The study is reported in line with the STARD statement (checklist in Text S4). The protocol is provided in Text S5.

Enrollment and Follow-Up Status

At the Pakistan site, 752 children were enrolled in the study—361 with pneumonia and 391 with “no pneumonia”—between October 2010 and February 2011 (Figure 1). Follow-up of 329 pneumonia and 343 “no pneumonia” cases was successfully carried out; 456 at 2 wk and 216 at 4 wk.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Pakistan urban site—flowchart of selection of pneumonia and no-pneumonia cases.

https://doi.org/10.1371/journal.pmed.1001422.g001

In Bangladesh, between March and August 2011, 1,178 children—589 each in the pneumonia and “no pneumonia” groups—were enrolled, of which 700 were from Dhaka Shishu Hospital (Figure 2) and 478 from rural Mirzapur (Figure 3). “No pneumonia” cases were enrolled using recruitment procedures similar to those used in Pakistan.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. Bangladesh urban site—flowchart of selection of pneumonia and no-pneumonia cases in Dhaka Shishu Hospital, Bangladesh.

https://doi.org/10.1371/journal.pmed.1001422.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. Bangladesh rural site—flowchart of selection of pneumonia and no-pneumonia cases in the rural setting.

https://doi.org/10.1371/journal.pmed.1001422.g003

Baseline Characteristics of the Study Children

The socio-demographic characteristics and the clinical features of the study children are presented in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Baseline characteristics of the study children.

https://doi.org/10.1371/journal.pmed.1001422.t001

Discriminative Power of Survey Tools

Sensitivity, specificity, and 95% CIs for the various instruments that were studied are presented in Tables 2 and 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Discriminative power of DHS/MICS questions about suspected pneumonia and of video for identifying childhood pneumonia (based on 2-wk and 4-wk recall).

https://doi.org/10.1371/journal.pmed.1001422.t002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Sensitivity and specificity of different tools (individually and in combination).

https://doi.org/10.1371/journal.pmed.1001422.t003

Data are given as percent (95% CI).

Pneumonia score.

As expected, specificity increased and sensitivity declined with increasing pneumonia score, i.e., the number of positive symptoms from among the pneumonia score questions. An area-under-the-curve analysis was performed for the pneumonia score (see below) and is presented in Figure 4. Thresholds could be chosen so that the performance was comparable to DHS/MICS questions (e.g., >9 in Bangladesh) or to selectively increase test specificity.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. Validity of composite pneumonia score.

https://doi.org/10.1371/journal.pmed.1001422.g004

Recall of Antibiotic Use

In children with pneumonia, antibiotic treatment was recalled correctly by 66.0% and 66.8% of caregivers using DHS or MICS questions in Pakistan and Bangladesh, respectively, with no significant difference between 2- and 4-wk recall. Correct recall increased to 72.0% using a drug flip chart/photo album and to 71.3% using a computer-based drug chart in Pakistan, and to 78.8% and 81.2%, respectively, in Bangladesh.

Association of Caregiver's Recall with Socio-Demographic Characteristics

In both Bangladesh study sites, but not in Pakistan, test specificities were higher for mothers who had higher levels of education. At the Bangladesh rural site the test specificity was 59% for mothers who had secondary or higher education, compared to a specificity of 37% for mothers whose education level was below primary. However, at the Bangladesh urban site these figures were 86% and 73%, respectively.

Importance of Monitoring Antibiotic Treatment of Pneumonia

Despite recent falls in pneumonia mortality over the past ten years, it remains the largest single cause of death in children [8]. It is therefore essential that global and national programs track the coverage of effective interventions against pneumonia if Millennium Development Goal 4 is to be achieved. The antibiotic treatment of pneumonia is highly effective and, together with immunization, is one of the two main control strategies [9]. DHS and MICS household surveys are the primary tools used to measure intervention coverage in low- and middle-income countries where health information systems are weak, and are the primary source of information on common childhood illnesses and treatment coverage. We report here on the first study evaluating the validity of the estimates of pneumonia treatment coverage produced by the DHS and MICS survey instruments and discuss the implications of these findings for program monitoring.

Validity of DHS/MICS Data for Estimation of Pneumonia Prevalence

Community-based epidemiological studies have estimated pneumonia incidence to be about 0.3 episodes per child per year in children under 5 y in low- and middle-income countries, which is some 12–18 times lower than the reported incidence of upper respiratory infections in this age group [10]. This low prevalence of pneumonia among all children with cough who are surveyed by DHS and MICS requires that the survey instruments must have very high specificity to identify pneumonia, otherwise the great majority of cases identified in the survey will not represent true cases of pneumonia (as reviewed by Campbell and colleagues in this PLOS Medicine Collection [11]). An example of this effect can be seen in a recent Pakistan DHS survey, which reported in 2006–2007 that 2,508/8,367 (29.9%) children under 5 y had cough and 1,178/8,367 (14.1%) had symptoms consistent with pneumonia [12]. This ratio of children with reported symptoms and signs of “cough and cold only” versus “suspected pneumonia” of 2∶1 is in marked contrast to that reported in community studies, where this ratio is typically very much higher, as noted above [13].

DHS and MICS reports clearly caution that these data (denoted as “suspected pneumonia” in MICS surveys) should not be used as a proxy measure of the prevalence of pneumonia in the community. However, many pneumonia control programs in the developing world, faced with a lack of data on this important parameter for planning purposes, use the information in this way. The results of our study show that the specificity of these survey tools is well below the very high levels required for this proxy to give an accurate estimation of pneumonia prevalence, and reinforce that these data should not be used for this purpose as they will lead to large overestimations of pneumonia prevalence [11].

Validity of DHS/MICS Data for Estimation of Proportion of Children with Pneumonia Who Receive Antibiotic Treatment

DHS and MICS surveys identify children whose caregivers report that they had symptoms and signs consistent with pneumonia, and then ask whether these children were treated with an antibiotic. The validity of this proxy indicator of the proportion of children with pneumonia who receive antibiotic treatment is therefore entirely dependent on the validity of DHS/MICS-reported symptoms and signs of pneumonia as a measure of true pneumonia. For this to represent a valid denominator for this important indicator, it is important that a high proportion of children with “symptoms of acute respiratory infection” (DHS)/“suspected pneumonia” (MICS) actually have “true” pneumonia. If this proportion is low, then results of monitoring antibiotic coverage among these children will not only be inaccurate but also misleading. The results of this study show that DHS and MICS surveys at both sites have poor discriminative power for the identification of episodes of pneumonia in young children. Previous studies have shown that fever is a significant predictor of pneumonia and that adding fever to WHO criteria increased the specificity of pneumonia diagnosis [14],[15]. Our findings support this observation, as specificity increased when a question on the presence of fever was included (Table 3). Since DHS/MICS surveys currently collect information on the presence of fever in the last 2 wk, these data could be readily incorporated into the definition of “suspected pneumonia.” However, the addition of questions on the presence of fever made only modest improvements to the discriminative power. Thus, most children with “suspected pneumonia” (MICS) or “symptoms of acute respiratory infection” (DHS) who form the denominator for the “antibiotic treatment rate” do not truly have pneumonia. We conclude, therefore, that the addition of questions on the presence of these symptoms and signs would not make sufficiently large improvements in instrument validity for these to be adopted at present.

We found no significant difference in caregivers' recall at 2 and 4 wk, which is contrary to the findings in some previous studies. The recall sensitivity of caregivers' reports dropped when the interview took place after more than 2 wk in a respiratory questionnaire validation study conducted in Peru [6]. Some previous studies on maternal recall of breastfeeding duration showed a drop in mothers' recall accuracy with time [16],[17]. A more recent study conducted in Kenya also assessed the accuracy of caregivers' recall over time and found that it decreased when the recall period was increased to 2 wk from 3–4 d [18]. In our study, the lack of fall in performance with the longer recall period suggests that DHS/MICS surveys may be able to adopt a longer recall period for this question, thus increasing the period prevalence of true pneumonia cases detected in the survey. This possibility should be studied further, because since increasing the recall period, and thus the true period prevalence, can be expected to improve the validity of the indicator [11].

The sensitivities and specificities of the DHS/MICS survey instruments varied substantially by study site and by urban and rural setting (Table 2), reflecting different cultural factors, such as caregiver education, and making comparisons across countries or time periods difficult to interpret. These results have important implications for monitoring programs because these instruments are used to track progress over time (typically with serial surveys in different populations within a country) and to identify countries that have low intervention coverage in comparison to others. These findings urge caution in using DHS/MICS data for these purposes.

We show that the use of video material shows some potential to complement current DHS/MICS surveys and result in increased specificity for identification of episodes of pneumonia. However, substantial improvements in test specificity were found only in Pakistan, where the video was made, and were not found in Bangladesh. This finding is not unexpected and may relate to the fact that Pakistani children look somewhat different from Bangladeshi children, which is likely to have influenced caregiver responses. Moreover, we elected a priori to focus the video on showing fast breathing and lower chest indrawing. Ethnographic studies from this region have shown that mothers have varying perceptions and concepts regarding childhood pneumonia, and many caregivers do not associate fast breathing with pneumonia [19]–[21]. The fixed sequence of clips might also have influenced caregiver responses. The potential use of this video tool presents practical and logistical challenges, not least of which is that each individual country may be required to develop their own video. The improved discriminative power with the use of the video in Pakistan is promising, and we recommend that video presentation be explored further in an attempt to further improve its performance.

In children with pneumonia, antibiotic treatment was recalled correctly by two-thirds of caregivers using DHS or MICS questions in Pakistan and Bangladesh, with no significant difference between 2- and 4-wk recall. Correct recall increased with a drug flip chart/photo album or a computer-based drug chart. Thus, the validity of the DHS and MICS questions in correctly identifying antibiotic treatment for pneumonia improved with specific and more structured questions about the medicines or with the use of illustrations of common treatments. Although drug charts performed better than DHS/MICS questions at all three sites, using them in periodic household surveys may be challenging. There may be a very large number of available medicines in any given area, and new medicine brands are introduced often, so frequent revisions would be required.

Study Limitations

The findings of the survey may have been biased by some aspects of the study design. DHS and MICS surveys are conducted in very diverse urban and rural settings. We attempted to reflect this through the urban and rural study sites selected for study. However, it is possible that the rural site in Bangladesh is not typical because of the ongoing research studies in that population. Furthermore, the urban site in Bangladesh included patients from an expanded urban and peri-urban catchment area, resulting in a relatively high rate of loss to follow-up and delayed follow-up. Thus, our findings may not be generalizable to some settings in low- and middle-income countries. Further studies, in particular in settings with high Plasmodium falciparum malaria transmission to explore the generalizability and to check the consistency of these findings, are recommended on this important issue. Nevertheless, the fact that our main study conclusions were confirmed in all three study settings suggests that the main findings may be widely relevant.

We selected study sites with investigators highly experienced in similar studies and employed a single technical expert to conduct the clinical training and oversee the monitoring of research staff in an attempt to reduce the potential for artifactual differences in study implementation across sites. Nevertheless, some of the differences shown between study sites may reflect some variation in application of methods across sites.

Mothers who take their children to a secondary or tertiary care center do not represent a random sample of all mothers who take part in DHS and MICS surveys. This is likely to influence results. We expect that these mothers will be more educated and show better recognition of signs of pneumonia in their children. To the extent that this is true, we may have overestimated the sensitivity and specificity of DHS/MICS questions. Furthermore, it is also likely that attendance at a hospital makes the episodes more memorable (and thus subject to better recall by the caregiver) and thus may once again lead to overestimation of specificity.

It is possible that the “case mix” in the hospital setting is different from that found in community surveys. We attempted to minimize this by excluding cases of severe pneumonia (rarely found in community surveys due to very low 2-wk period prevalence) and by adopting a matched group who had a respiratory illness (cough and cold) rather than studying healthy children. Nevertheless, this may have influenced our estimates of sensitivity and specificity. Thus, study procedures may have influenced caregiver recall. However, our main finding is that specificity levels associated with DHS and MICS questions are much too low to provide a robust basis for a pneumonia treatment rate indicator. We believe that the biases noted above tend to lead to an overestimation of specificity, and so we consider that our main conclusions about poor discriminative ability should remain valid. In addition, the level of specificity that we found is much lower than would be required, suggesting moderate effects of bias.

Conclusions

Monitoring antibiotic treatment of pneumonia is essential for national and global programs. Despite this, we believe this is the first study to assess the validity of current surveys to measure this program indicator. Given the very large investment of human and financial resources in these surveys and their importance to child health programs, it is remarkable that there has been so little published data from research on this key issue. In the context of a low true prevalence of pneumonia, the DHS/MICS questions to define children with “suspected pneumonia” (MICS term) has a low yield for pneumonia, and thus most of these children do not have true pneumonia. This finding reinforces DHS and MICS recommendations that these measures not be used as a proxy for pneumonia prevalence, as using these measures will result in substantial overestimates.

We found the discriminative power (in particular the specificity) of a construct of caregiver-reported symptoms and signs of acute respiratory infection to be low for true pneumonia, as measured by DHS and MICS questionnaires; therefore, the use of such data from these surveys can be misleading for measuring antibiotic treatment rates. Using such data could lead to incorrect policy decisions having programmatic implications. Although the alternative tools evaluated in this study did not perform markedly better than DHS and MICS questions, we have presented some options for improving their specificity. Furthermore, the sensitivities and specificities of DHS and MICS survey instruments varied substantially by site, reflecting different cultural factors, such as caregiver education, and making comparisons across countries or time periods difficult to interpret. These results suggest that data from these surveys should not be used for the purpose of monitoring antibiotic treatment rates in children with pneumonia at the present time.