Sound policy and program decisions require timely information based on valid and relevant measures. Recent findings suggest that despite the availability of effective and affordable guidelines for the management of sick children in first-level health facilities in developing countries, the quality and coverage of these services remains low. We report on the development and evaluation of a set of summary indices reflecting the quality of care received by sick children in first-level facilities. The indices were first developed through a consultative process to achieve face validity by involving technical experts and policymakers. The definition of evaluation measures for many public health programs stops at this point. We added a second phase in which standard statistical techniques were used to evaluate the content and construct validity of the indices and their reliability, drawing on data sets from the multi-country evaluation of integrated management of childhood illness (MCE) in Brazil, Tanzania and Uganda. The statistical evaluation identified important conceptual errors in the indices arising from the theory-driven expert review. The experts had combined items into inappropriate indicators resulting in summary indices that were difficult to interpret and had limited validity for program decision making. We propose a revised set of summary indices for the measurement of child health care in developing countries that is supported by both expert and statistical reviews and that led to similar programmatic insights across the three countries. We advocate increased cross-disciplinary research within public health to improve measurement approaches. Child survival policymakers, program planners and implementers can use these tools to improve their monitoring and so increase the health impact of investments in health facility care.