Article Text

Download PDFPDF

Measuring open defecation in India using survey questions: evidence from a randomised survey experiment
  1. Sangita Vyas1,2,
  2. Nikhil Srivastav2,3,
  3. Divya Mary4,
  4. Neeta Goel5,
  5. Sujatha Srinivasan4,
  6. Ajaykumar Tannirkulam4,
  7. Radu Ban6,
  8. Dean Spears1,2,7,
  9. Diane Coffey2,7,8
  1. 1 Economics and Population Research Center, University of Texas at Austin, Austin, Texas, USA
  2. 2 r.i.c.e, India
  3. 3 LBJ School of Public Affairs, University of Texas at Austin, Austin, Texas, USA
  4. 4 IFMR LEAD, Institute for Financial Management and Research, Chennai, India
  5. 5 International Initiative for Impact Evaluation, New Delhi, India
  6. 6 Evidence and Measurement, WSH Program, Bill and Melinda Gates Foundation, Seattle, Washington, USA
  7. 7 Indian Statistical Institute, New Delhi, India
  8. 8 Sociology and Population Research Center, University of Texas at Austin, Austin, Texas, USA
  1. Correspondence to Sangita Vyas; sangita.vyas{at}utexas.edu

Abstract

Objectives To investigate differences in reported open defecation between a question about latrine use or open defecation for every household member and a household-level question.

Setting Rural India is home to most of the world’s open defecation. India’s Demographic and Health Survey (DHS) 2015–2016 estimates that 54% of households in rural India defecate in the open. This measure is based on a question asking about the behaviour of all household members in one question. Yet, studies in rural India find substantial open defecation among individuals living in households with latrines, suggesting that household-level questions underestimate true open defecation.

Participants In 2018, we randomly assigned latrine-owning households in rural parts of four Indian states to receive one of two survey modules measuring sanitation behaviour. 1215 households were asked about latrine use or open defecation individually for every household member. 1216 households were asked the household-level question used in India’s DHS: what type of facility do members of the household usually use?

Results We compare reported open defecation between households asked the individual-level questions and those asked the household-level question. Using two methods for comparing open defecation by question type, the individual-level question found 20–21 (95% CI 16 to 25 for both estimates) percentage points more open defecation than the household-level question, among all households, and 28–29 (95% CI 22 to 35 for both estimates) percentage points more open defecation among households that received assistance to construct their latrines.

Conclusions We provide the first evidence that individual-level questions find more open defecation than household-level questions. Because reducing open defecation in India is essential to meeting the Sustainable Development Goals, and exposure to open defecation has consequences for child mortality and development, it is essential to accurately monitor its progress.

Trial registration number Registry for International Development Impact Evaluations (5b55458ca54d1).

  • India
  • open defecation
  • sanitation
  • survey methods
  • experiment

This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.

View Full Text

Statistics from Altmetric.com

Strengths and limitations of this study

  • This is the first study that experimentally tests the difference in open defecation estimated from different survey questions.

  • This study provides evidence that India’s Demographic and Health Survey, which asks about the defecation behaviour of everyone in the household in one question, substantially underestimates open defecation in India.

  • The study shows that measuring open defecation at the individual level is feasible in a large household survey, and finds more open defecation than household-level questions.

  • Monitoring open defecation in India is important for understanding progress towards the Sustainable Development Goals.

  • Because the households participating in this study are not representative of the rural parts of the states they are in, or rural India, the estimates presented here should not be considered as estimates of open defecation for any of the states, or the country as a whole.

Introduction

Rural India is home to more than half of the world’s open defecation.1 Because the persistence of open defecation threatens gains in child health, the Sustainable Development Goals (SDGs) call for its elimination by 2030. Progress towards eliminating open defecation in rural India will be essential to meeting this goal. India’s most recent Demographic and Health Survey (DHS), conducted between January 2015 and December 2016, estimates that 54% of households in rural India defecated in the open, down from 75% in the 2005 to 2006 DHS.2 3 This measure is based on a household-level question that asks about the behaviour of everyone in the household in the same question.

Recent evidence from studies carried out in India suggests, however, that it is common for individuals living in households with latrines to nevertheless defecate in the open. In rural parts of five north Indian states, Coffey et al found that 21% of individuals defecated in the open, despite owning a latrine.4 In rural Tamil Nadu, Yogananth and Bhatnagar report that 54% of respondents defecated in the open despite having a household latrine.5 In Odisha, Barnard et al found that less than half of members of households with latrines reported using their latrines at all times.6

Experimental studies of sanitation interventions have found similar results. Clasen et al report on a sanitation intervention in Odisha and note open defecation among individuals living in households with latrines as a reason for not observing impacts on child health outcomes.7 Patil et al conducted a sanitation intervention in Madhya Pradesh and experienced a similar problem: modest increases in latrine coverage, and even more modest reductions in open defecation.8 These findings suggest that open defecation among latrine-owning households is substantial. Since it is probable that latrine use is the socially desirable response to questions on sanitation behaviour, measures based on household-level questions, such as those from the DHS, will likely underestimate true open defecation in rural India, particularly among households with latrines.

Because open defecation is an individual behaviour, an individual-level survey question may be able to more accurately measure it compared with a household-level question, particularly among households with latrines. We designed this study to experimentally test this hypothesis in rural India. We aimed to investigate whether a balanced question about latrine use or open defecation for every member of a household finds different levels of open defecation compared with a household-level question.

This is the first study to experimentally vary survey methodology to improve on the measurement of open defecation currently being used. Jenkins et al study sanitation survey methods, and report on an index they develop for quantifying household excreta disposal.9 Their study focuses on developing and piloting a new tool rather than comparing different measures. Sinha et al compare answers to survey questions on latrine use behaviour to measures of actual behaviour generated from passive latrine use monitors that were set up in the latrines of respondents, and find poor to moderate agreement between the two measures.10 Our study contributes to this literature by comparing estimates of open defecation obtained from questions that can be administered in a large household survey, and highlighting potential sources of error in open defecation measurement.

Four years ago, the Government of India launched the Swachh Bharat Mission (SBM), a national sanitation campaign, which aims to eliminate open defecation in India by 2019. Many latrines have been constructed in rural India as a result of this campaign. Yet, the effect the SBM has had on reducing open defecation is still unknown. Because large reductions in open defecation in India are essential to meeting the SDGs, and because exposure to open defecation has serious consequences for child mortality, health and human capital development, it is essential to monitor its progress as accurately as possible.

Methods

The study is registered in the Registry for International Development Impact Evaluations.

Sample: mostly latrine-owning households in rural parts of four states

This study uses as its sampling frame the study areas of 3ie’s Promoting Latrine Use in Rural India Thematic Window. This window has funded four independent research teams to conduct randomised control trials of distinct behavioural campaigns to promote the use of pit latrines in rural parts of Bihar, Gujarat, Karnata and Odisha. The study areas are spread across India, representing different contexts and varying levels of rural open defecation.

Because these trials focus on behavioural strategies rather than latrine construction, they are being carried out in villages that had high levels of coverage of pit latrines at baseline, relative to other rural parts of the same states. The households that comprise the sampling frame for this study are those that were identified as having a functional latrine in a census conducted by the research teams in the villages in which they were working. In all states except for Odisha, only households that had been excluded from the research teams’ samples could be selected for this study. In Odisha, the sample selected for this study overlaps with the research teams’ sample. We aimed to survey households that own latrines because we expect that an important source of misreporting of open defecation comes from individuals who do not use the functional latrines that their households own, and that other members of their households use.

Figure 1 describes the sample selection. The villages visited in each state were randomly selected from the full set of villages included in the 3ie research teams’ studies. The full set of villages were selected by the research teams in collaboration with the implementation agencies they were working with. The research team led by Oxford Policy Management worked with World Vision in Bihar, the team led by London School of Hygiene and Tropical Medicine worked with Coastal Salinity Prevention Cell in Gujarat, Eawag worked with Wateraid in Karnataka, and Emory University worked with the Rural Welfare Institute in Odisha. Data for our study were collected in 22–25 villages in each of the four study areas. In most areas, we sampled more villages than we actually visited in order to facilitate coordination with the research teams. Ninety-five villages were visited in total.

Figure 1

Sample selection. Households that were started but not completed refers to households that refused at the beginning or part of the way through and households in which a suitable respondent was not available.

Up to 40 households in each village were randomly assigned to receive the household or individual questions. In some villages, fewer than 40 households were assigned because fewer than 40 households met the eligibility criteria. The survey team visited as many assigned households as it could in these villages, given time constraints, and availability of household members. On average, the survey team interviewed 25 households per village. Data collection took place between March and July 2018.

In each state, data collection took place after the 3ie research teams had conducted their censuses and baselines, but before they had started their interventions. Since in all states, the households visited in this study were also visited at the time of the census, response bias may be a concern. This would not, however, impact the internal validity of this study since randomisation generates equal response bias, in expectation, across treatment arms.

Randomisation and masking: random variation in latrine use questions at the household level

We randomly assigned the type of latrine use question administered in the survey at the household level. Roughly half of the households were assigned individual-level questions on latrine use. The other half were assigned a household-level question. One of the authors who was not involved in data collection carried out the randomisation using a random number generator in Stata. Because of the nature of the study, it was not possible to blind the respondents or surveyors to the type of survey question administered in the survey. However, in the interest of data quality, respondents were not explicitly told that the primary purpose of the survey, which took ~25 min to complete, was to measure open defecation. Additionally, surveyors did not know which survey question had been assigned to a household until starting the survey with the household. This was facilitated through SurveyCTO, the mobile data collection platform used in the study, which was programmed to store the randomisation assignment for each household ID prior to the commencement of data collection. In the field, surveyors were only given a list of households to interview. When a surveyor had correctly identified a household and was ready to start the survey, she would enter the household ID into SurveyCTO, and SurveyCTO would automatically start the questionnaire type assigned to the household.

The individual-level questions asked for every household member age 5 or older whether the individual defecated in the open or used the latrine. The preface to this series of questions was: ‘I have seen that some people defecate in the open, and some people use the latrine. Now I want to ask about where you and your family members defecate’. Then, the surveyor asked the following question for each individual in the household, and coded the answer in a household roster: ‘The last time [name of household member] defecated, did [name of household member] defecate in the open or use the latrine?’. The answer options included latrine, open and somewhere else. Surveyors used the last option, which meant that the household member defecated in a bedpan, cloth or other place, in <0.5% of cases. Because the priming statement and the behaviour question include both open defecation and latrine use, they are balanced between the two different behaviours and could reduce social desirability bias. The surveyor asked household members who were participating in the interview directly about their behaviour, and asked the main respondent, in most cases an adult female member of the household, to report on the behaviour of their family members who were not participating in the interview.

The rest of the households were assigned the household-level question used in India’s DHS: ‘What kind of toilet facility do members of your household usually use?’.11 The answer codes were also the same as those used in India’s DHS: flush to piped sewer system, flush to septic tank, flush to pit latrine, flush to somewhere else, flush to don’t know where, ventilated improved pit or biogas latrine, pit latrine with slab, pit latrine without slab or open pit, twin pit or composting toilet, dry toilet and no facility or uses open space or field. We also included an individual-level question on mobile ownership or preferring vegetarian food versus non-vegetarian food in the surveys that asked the household-level question so that both types of surveys would take approximately the same amount of time to complete.

There are three main factors that differ between the two types of latrine use survey modules: the level of aggregation, the reference period and the presence of a priming statement. Therefore, the differences in reported open defecation that we observe reflect the fact that the two sets of questions vary on all of these factors combined.

Statistical analyses

The primary outcome of interest is reported open defecation. For the household-level questions, we created a dummy variable that is equal to one if the response was ‘no facility or uses open space or field’, and zero otherwise. The unit of observation for households assigned the individual-level question is the individual, while the unit of observation for those assigned the household-level question is the household. Therefore, in order to directly compare and test the significance of differences in reported open defecation between the two question types, we construct estimates that use the same unit of observation. We impute individual-level open defecation from responses to the household-level questions, and household-level open defecation rates from responses to the individual-level questions. To construct individual-level open defecation using the household-level questions, we assign the answer from the household question to each individual in the household. Similarly, to construct household-level open defecation using the individual-level questions, we average the responses among individuals in the household, and assign this average as the household value. Our main analysis tests differences in measured open defecation by question type. We show pooled results, as well as results by study area.

We also conduct subgroup analyses. These analyses investigate differences in the same primary outcome measure, reported open defecation, but look at differences by question type among different subgroups. First, we investigate whether the difference in reported open defecation by question type depends on whether the latrine was constructed privately, or with assistance from the government or a non-governmental organization (NGO). In practice, assistance to construct latrines often comes from the government, but sometimes NGOs get involved in facilitating the implementation of the government programme. As part of the SBM, the Government of India assists rural households to construct latrines either by providing financial assistance directly to households so they can construct their own latrines or by local government officials constructing latrines for households. In the discussion that follows, we will describe a household as having ‘received help’ if it received financial assistance or a partially or completely constructed latrine from the government or an NGO.

The Indian government promotes and constructs latrines with pits that are ~60 ft3.12 However, many rural Indians aspire to construct latrines with pits that are much larger, so that they can avoid emptying the pit, a task that is associated with ritual pollution.13–16 Compared with latrines constructed privately, those constructed with government help are less likely to be used due to concerns over purity and pit emptying. Since a large fraction of rural households are likely to receive latrines with help from the government as a result of the SBM, it is important to explore how much open defecation different types of latrine use questions measure, based on having received help to construct the latrine.

The second subgroup analysis investigates whether the difference in measured open defecation between the two question types is statistically different for males compared with females. Sex differences are an important aspect to explore because observational studies have found consistently higher open defecation among latrine owners for males compared with females.4 This observation could reflect greater demand for latrine use among females due to, for instance, greater psychosocial stress experienced when defecating in the open,17 or it could be because of cultural norms that keep females in their reproductive years inside the home.

Means and differences in reported open defecation, by question type, are calculated using ordinary least squares regression with cluster robust SEs, clustered by village. Statistical analyses were conducted using Stata V.11.

Public involvement

The individual-level questions used in this study are the product of a deliberative process between the authors and research teams from Oxford Policy Management, London School of Hygiene and Tropical Medicine, Indian Institute of Public Health Gandhinagar, Eawag and Emory University. The findings of this study contribute to our understanding of the scale of an important public health problem.

Results

Table 1 shows summary statistics for households assigned the two types of latrine use questions. The total sample consisted of 2431 households, which were approximately equally divided across question type in each of the study areas. There were no significant differences on measures relevant for latrine use between households assigned different types of latrine use questions. Households in both groups had approximately the same number of household members, fraction female, fraction Hindu, educational attainment of the household head, and asset ownership of 13 assets, including mobile phone, electricity, radio, television, fan, mosquito net, bicycle, motorcycle, car, chair, gas stove, pressure cooker and shoes for everyone in the family. It is important for the validity of the results that the sample is balanced on religious composition, since studies have documented an association between household religion and latrine use.18 19

Table 1

Randomisation balance: no significant differences in observed means between households assigned different latrine use questions

As the study design intended, most households in both groups had a latrine. Conditional on having a latrine, 64% of households had gotten help from the government or an NGO to build the latrine, and the average pit size was 180 ft3. It is important that the sample is balanced on these two characteristics because, compared with latrines constructed privately, those constructed with government help are less likely to be used because of concerns over purity and pit emptying.13–16 Finally, among households with latrines, ~80% of them appeared to the surveyor to be in use on observation.

Figure 2 presents the main results of the study; it shows means and 95% CIs from the individual-level (shown as dashed red bars) and the household-level (shown as solid blue bars) questions, for the full dataset and for different subsamples. Observations are individuals for the individual-level estimates, and households for the household-level estimates. In the full sample, and in all subsamples analysed in this figure, the individual questions find more open defecation. Moreover, the size of the difference in reported open defecation between question types is large and consistent. The first set of estimates shown in the figure uses the full sample. The second, third, fourth and fifth sets of estimates break the sample up by project area. The sixth set of estimates uses only households with latrines. No matter how the data are broken up, the individual-level, balanced latrine use questions find significantly higher rates of open defecation than the household-level question.

Figure 2

Individual-level, balanced latrine use questions find significantly higher rates of open defecation than the household-level question. Figure shows means and CIs. Red, dashed lines indicate responses to individual-level, balanced latrine use questions, and blue solid lines indicate responses to the household-level question. Unit of observation is individuals for individual-level questions and households for household-level questions. CIs are computed using SEs clustered by village.

Table 2 shows actual and imputed open defecation rates, measured at the individual and household levels. Columns 1 and 3 are the estimates shown in figure 2. Columns 2 and 4 show imputed open defecation rates, at the individual and household level, respectively. Imputed values are calculated based on the method described in the Statistical analyses section. Column 5 shows the difference in the measured rate of open defecation between the individual and household questions when observations are individuals, and column 6 shows the same difference when observations are households.

Table 2

Individual-level, balanced latrine use questions find significantly higher rates of open defecation than household level questions

In the full sample, the individual-level, balanced questions find 21 (95% CI 16 to 25) percentage points more open defecation than the household-level question when observations are individuals, and 20 (95% CI 16 to 25) percentage points more open defecation when observations are households. Notably, the individual-level questions measure consistently higher levels of open defecation in the full sample and in all subsamples, irrespective of how the difference is calculated. All differences are significant at the 1% level.

The seventh set of estimates shown in figure 2 show reported open defecation from individual-level and household-level questions among households that received help to construct their latrines. Comparing these estimates to the sixth set of estimates in the figure, which include all households with latrines, suggests that the household-level question underestimates open defecation by more among those that received help to construct their latrines, compared with those that did not. The seventh row in table 2 shows that, using different methods for computing differences, individual-level questions measure 28–29 (95% CI 22 to 35 for both estimates) percentage points more open defecation than the household-level question, among those that received help to construct their latrines, compared with those that did not.

The first two columns in table 3 test whether the difference in measured open defecation between the two question types is statistically different among households that received help to construct their latrines compared with households that did not receive help to construct their latrines. The first column in table 3 uses individuals as observations, and the second column uses households. The coefficients in the third row represent the difference-in-differences estimate.

Table 3

Subgroup analyses

The difference in measured open defecation between the two question types is 18 (95% CI 11 to 25) to 19 (95% CI 11 to 26) percentage points larger among households that received help to construct their latrines, compared with households that did not, depending on the method used to compute the difference. Among households that did not receive help to construct their latrines, the individual-level questions find 10 (95% CI 5 to 15) percentage points more open defecation than the household-level question.

The last column in table 3 investigates whether the difference in measured open defecation between the two question types is statistically different among males compared with females. Using the full sample, individual questions find 4 (95% CI 2 to 7) percentage points more open defecation than the household question among males compared with females. Breaking the sample up by sex, the individual questions find 23 (95% CI 18 to 28) percentage points more open defecation than the household question among males, and 19 (95% CI 14 to 24) percentage points more among females. The household question underestimates open defecation by more among males compared with females.

Discussion

Our findings show that in our sample, individual-level, balanced questions find 20–21 (95% CIs 16 to 25 for both estimates) percentage points more open defecation than the household-level question. This is both a statistically significant and practically important difference. This study presents compelling evidence that India’s DHS, which provides the most recent nationally representative estimates of open defecation for rural India, and other surveys that ask household-level questions, greatly underestimate open defecation among households with latrines.

We also found that the difference in reported open defecation between the two question types is significantly greater for households that received help to construct their latrines compared with households that did not. Among households that received help to construct their latrines, the individual-level questions find 28–29 (95% CIs 22 to 35 for both estimates) percentage points more open defecation than the household-level question. This suggests that as more and more households receive government assistance for a latrine through the SBM, household-level questions will become even less accurate at estimating open defecation.

The larger difference in measured open defecation between the two question types among households that received help compared with those that did not is likely arising from higher rates of open defecation among households that received help to construct their latrines. There are several reasons that could explain why households receiving assistance may be less likely to use their latrines. First, these households are likely to have lower demand for latrine use, compared with households that built latrines on their own. Second, households that received help have latrines with pits that are on average 150 ft3 smaller than the pits of latrines in households that did not receive help. Because of concerns over ritual purity, rural Indians are less likely to use latrines with pits that need to be emptied manually every few years, like the latrines that are promoted and constructed by the government.13–16 Whether only one, both or other factors are leading to more open defecation among households that received help, the individual-level questions are better able to capture this open defecation than the household-level question.

We also find a statistically significant difference in reported open defecation between the two question types for males compared with females. The difference between the individual-level, balanced questions and the household-level question is 4 (95% CI 2 to 7) percentage points more for males compared with females. This supports evidence that, conditional on latrine ownership, males are more likely to defecate in the open compared with females. Individual-level questions understate the difference in open defecation between the two sexes by less than the household-level questions. Compared with the difference in reported open defecation by receiving help to construct the latrine, differences by sex are not as large.

Measuring open defecation at the individual level is feasible. Our survey team’s experience suggests that adding the balanced, individual-level questions on use to a survey that already contains a household roster increases survey time by about 2 min, on average. Of course, the amount of time required to ask the individual-level questions depends on the number of individuals in the household.

The household-level question asked in the DHS also collects information on the types of latrines that households own, data that are still of great interest to researchers and practitioners. Therefore, individual questions on use, combined with a separate question on the types of latrines that households own, would satisfy both goals: evaluating latrine infrastructure, and measuring open defecation as accurately as possible.

A limitation of our study is that the samples from these project areas are not representative of the rural parts of the states they are in, nor are they collectively representative of rural India. The households in this study are much more likely to have a latrine than the average rural Indian household, and therefore, the individuals in this study are more likely to use a latrine. For this reason, the estimates presented here should not be considered as estimates of open defecation for any of the states, or the country as a whole. Rather, they show a large and significant difference in reported open defecation based on the type of question asked.

Measuring open defecation at the individual-level in a large household survey is doable and will provide a more accurate estimate of open defecation in rural India. Since reducing open defecation in India is important for meeting the SDGs, and since open defecation is an important factor contributing to poor health among children in India, it is important to measure its progress as accurately as possible.

Acknowledgments

We are very grateful for the collaboration and contributions of the 3ie research teams in formulating the individual-level questions. The questions are the product of a deliberative process between the authors and the research teams. We are also grateful for the research teams’ cooperation on this study. In particular, we thank researchers from Oxford Policy Management, London School of Hygiene and Tropical Medicine, Indian Institute of Public Health Gandhinagar, Eawag, and Emory University, and the implementing agencies with which they are working, including World Vision, Coastal Salinity Prevention Cell, Wateraid, and Rural Welfare Institute, for sharing their census data with us, coordinating field work and supporting this endeavor. This study would not have been possible without their collaboration and assistance.

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
View Abstract

Footnotes

  • Contributors SV, NS, DS and DC contributed to the study design. SV, NS, DC and DS designed the survey instruments. NS, DM, SS and AT oversaw data collection. SV, NS and NG coordinated between the 3ie research teams and the survey team for this study. SV, RB, DC and DS contributed to analysis. All authors contributed to drafting the report.

  • Funding International Initiative for Impact Evaluation, Bill & Melinda Gates Foundation. RB, who is employed on the WSH programme of BMGF, contributed to the analysis of the results. NG contributed to coordinating data collection. Both contributed inputs to drafting the report.

  • Competing interests None declared.

  • Patient consent for publication Not required.

  • Ethics approval The study received ethical approval for research involving human subjects from the Institute for Financial Management and Research’s Institutional Review Board in India, Approval # IRB00007107.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data availability statement Data are publicly available here: https://riceinstitute.org/data/measuring-open-defecation-in-india-using-survey-questions-a-randomized-survey-experiment/

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.