Many of the almost 10 million annual deaths worldwide of children under the age of 5 are preventable.¹ Despite an overall fall in mortality rates there is evidence that in many countries in Sub-Saharan Africa the incidence is rising again.² WHO and UNICEF have developed the Integrated Management of Childhood Illness (IMCI) approach in order to more effectively and efficiently deal with the main causes of this mortality, both in the community and at the primary level health facility.³ The primary care component of IMCI is centred around case management algorithms and has to date been applied in more than one hundred countries.

Most studies so far have concentrated on establishing the efficacy of these algorithms in research settings. The technical merit of the algorithms has been established through comparisons with a gold standard (usually experienced physicians).^4,5 However, there are relatively few studies that have evaluated the effectiveness of IMCI implementation in the context of a functioning health system.^6,7

Cape Town is one of the wealthiest parts of South Africa with an estimated infant mortality rate (IMR) of 30 per thousand. However, this average disguises the significant variations within the population and in many areas the rates are over 60 per thousand.⁸ There is also concern regarding the quality of child health services in these areas as reflected by studies documenting missed vaccine opportunities and high TB defaulter rates.^9,10 For these reasons the local health authority committed itself to the IMCI intervention. However, this has significant financial and human resource implications and they therefore requested that a rapid evaluation of the initial implementation of IMCI be conducted. We report the findings of an intervention study to evaluate the effect of the routine implementation of IMCI on quality of care of sick children in Cape Town.

METHODS

Study sites

IMCI implementation started in January 2001 in health districts within Cape Town.

Adaptations of the generic IMCI were made to suit the local epidemiological profile by the removal of measles and malaria and the addition of HIV/AIDS and asthma guidelines.¹¹ Four of the eight districts in Cape Town were initially selected for IMCI, two because there was local demand (Tygerberg East and West) and two because of their high IMRs (Khayelitsha and Nyanga). Together these four districts account for 70% of all infant mortality in the city. Table 1 outlines the socioeconomic status and health indicators.

The primary healthcare clinics are predominantly nurse led with medical officers conducting occasional sessions. Senior nursing staff have prescribing powers and there is a good referral system to the local tertiary hospitals. Health services to children below the age of 5 are free of charge. All the nurses who attended sick children, before the introduction of IMCI, had some form of paediatric training. IMCI case management training started in April 2001, with monthly 11 day courses for up to 20 nurses at a time. All the training was administered and facilitated by the local health services using the WHO/UNICEF teaching and assessment modules. By May 2002 all professional nurses seeing young children had been IMCI trained in the four districts.

Study design and sampling

This was designed as a plausibility study^12,13 with a prospective, before and after study of the effect of routine IMCI implementation on compliance with IMCI standards. Pre- and post-implementation assessments of nurses were conducted during February 2001 and September 2002 respectively. Clinics were randomly selected by assigning numbers to clinics and a random numbers table. The number selected in each district was determined by the time available for assessment (see table 1). In the pre-implementation phase one nurse who was attending to sick children on the day of the visit was conveniently sampled at each of the selected clinics. In the post-intervention visit a nurse who had been trained in IMCI and attending to sick children was selected. The first five sick children in the queue to see the selected health worker and whose caregivers agreed to participate in the study were chosen. If this was not possible due to low attendance a minimum of three children were observed per health worker.

Data collection

The WHO generic Health Facility Survey instruments¹⁴ were adapted, taking into account the South African IMCI guidelines. Three main instruments were used: a structured observation checklist of the case management of sick children, exit interview with caregivers of sick children, and a structured inventory of equipment, drugs, and supplies in the facility. No independent re-examination of the sick child was performed but all the observers were trained in IMCI and had clinical experience.

Similar training procedures and data collection methods were used for both assessments. Survey teams comprised of three people with a total of four teams. Each team visited one clinic per day. All the surveyors were experienced nurses and supervisors who had been trained on the IMCI course. The surveyors were trained until they had achieved over 90% inter-rater reliability for at least two observations consecutively. A definition list and detailed observation and interview rules were developed. Data collection sheets were collected daily after completion of fieldwork and checked by the field work supervisor. The ethical committee of the University of the Western Cape approved the protocol for this study. Oral informed consent was obtained from the facility managers and nurses, and written consent from caregivers.

Main outcome measures

Data were entered, cleaned, and analysed in EPI-INFO 2002 (CDC, Atlanta, USA). The primary and secondary indicators as identified by the WHO analysis plan¹⁴ were calculated for each sick child consultation. This included an index of integrated assessment which gives a score of 1 for each of the following 10 tasks performed on a child: checking for three danger signs, diarrhoea, cough, fever, palmar pallor, vaccination status, child’s weight, and whether the weight was plotted on the growth card. Each consultation could therefore score a maximum of 10.

Using consultations as the unit of analysis, 95% Wald confidence limits were calculated for the differences between the two assessments and the Kruskal-Wallis test was used for differences in means.

RESULTS

Twenty one nurses in 21 clinics were observed before and after the IMCI intervention. These nurses were not necessarily the same but there was no difference in mean time since qualification as a nurse between the two groups (7.5 years versus 8.5 years, p = 0.625). Ninety sick child observations were conducted before IMCI and 70 after IMCI. Seven and five children before and after the IMCI, respectively, were identified as requiring urgent referral and excluded during the analysis. Three observations pre-IMCI and one observation during the post-IMCI assessment were incomplete and discarded for analysis. The age distribution of the children was similar in the two groups, median age 21 months versus 23 months (range 2–55 months). The median length of time since the nurses’ completion of IMCI training was 9.5 months (range 4–18 months).

Assessment and classification

Assessment of danger signs (lethargic/unconscious, not able to drink or breast feed, vomiting everything, convulsions in this illness) was not done well before IMCI, with only 6.6% of sick children being assessed for these crucial signs. This improved to 72% after IMCI (p < 0.001). During both assessments the most commonly assessed danger sign was the ability of the child to drink or breast feed and the least commonly asked was the presence of convulsions in this illness.

The intervention succeeded in getting nurses to assess not only the presenting symptoms but also other symptoms relating to major childhood illnesses. There was also an improvement in the mean score of the integrated index (table 2).

Treatment

There were also significant improvements in the appropriate prescribing and demonstration of antibiotics (table 3). However, there was no change in the treatment of anaemia or the prescribing of vitamin A to sick children. In the case of anaemia all the children were prescribed iron tablets or syrup, but only a few were also given anthelmintic medication. Even under IMCI, more than one third of children did not receive their first dose of treatment in the clinic.

Counselling

The counselling given to caregivers remained poor. Only one third of caregivers were informed of at least three signs that would necessitate immediate return to the clinic (table 4). There was some improvement in the counselling about the sick child’s need for extra fluids and food after IMCI.

Exit interview with caregivers

There was no improvement (80% versus 76%) of the caregivers’ understanding on how to give the prescribed medication; also similar proportions (55% versus 45%) of caregivers knew at least two signs that the child was becoming sicker before and after IMCI.

Facility support

The clinics were well equipped and supplied during both assessments. Only one clinic in the first assessment was short of vaccines and two of the essential IMCI medicines, though this was a short term shortage. In all cases the nurse had received one supervisory visit in the last six months, and in the second assessment all IMCI nurses had received at least one supervisory visit which had covered IMCI, including observation of case management. However, the recommended IMCI observation checklist had not been used.

DISCUSSION

There are some limitations to this study. The lack of randomisation and the use of historical controls make discounting of confounding impossible. The striking improvement in performance across all intervention sites, the relatively short time between measurements, no change in drug supply, simultaneous improvements across the intervention sites, and the lack of any other quality of care intervention in the study areas suggest that the improvements were due to the IMCI intervention.

There is also the possibility of reporting bias as it was difficult to blind observers to whether the assessment was before or after the IMCI training. It is impossible to estimate the magnitude of this bias.

The selection of only IMCI trained nurses in the follow up survey may overestimate the overall effect of the intervention in health services. However, all of the paediatric nurses working in the districts before the introduction of IMCI were subsequently trained. During the follow up non-trained nurses were either newly qualified or had migrated into the districts since the introduction of IMCI.

We were not able to perform an independent re-examination of the sick child to establish the gold standard classification, but deduced it from observations during the health worker’s assessment of the patient. While not the ideal we do not believe this affected our findings very much. Finally, there is some over-estimation of the precision of findings as district clustering has not been taken into account.

Standard of care

Overall, the standard of care after IMCI documented in this study is towards the high end of assessments from other provinces in South Africa. The integrated assessment score of 8.2 compares favourably with the National Department of Health/WHO review that found assessment scores ranging from 6.1 in Kwazulu/Natal to 8.4 in Mpumulanga and Northern Cape.^15,16

We found a large improvement in the correct assessment of danger signs in sick children and the overall assessment of children after IMCI. This is an encouraging development since recognition of danger signs can be critical. An outpatient study in rural Kenya showed that the mortality risk associated with having at least one danger sign was 6.5 times higher than that for children without any signs.¹⁷ Numerous studies in developing countries have also established the presence of co-morbidity in many sick children and the benefits of taking an integrated approach towards assessment.^18,19

There were also encouraging signs of increased efficiency of care. In this study the IMCI approach resulted in a significant reduction of inappropriate antibiotic use, mostly through the more rigorous definitions of upper and lower respiratory respirations. Similar improvements in the case management of sick children following an IMCI intervention have been reported from Tanzania.⁷ After personnel, drug costs are the largest item on the health budget. This study supports other studies which have found that IMCI can lead to significant reductions in drug costs,²⁰ thus helping to offset the cost of training on IMCI.

The poor level of counselling and the absence of improvement in caregiver understanding of key practices in this study is cause for concern. The counselling component of IMCI training focuses on key messages to the caregiver about when the child should return immediately to the clinic, feeding of the sick child, follow up of the sick child, and checking the caregiver’s understanding.²¹ The lack of improvement in counselling may be because nurses were focusing more on the assessment, examination, and treatment of the sick child (as confirmed by improvements in these aspects after training) or that the counselling component of the training itself may have limited impact on nurses’ practice. The increasing incidence of HIV makes counselling even more important.

Success factors

The Western Cape health authority has invested heavily in ensuring that the health system supports the IMCI intervention through orientating doctors and nursing supervisors in IMCI and adapting drug and health management information systems.²² This approach has support in the literature where reviews of interventions to improve quality of care have underscored the ineffectiveness of just developing and disseminating clinical guidelines or conducting training alone.^23–25 Sustained improvements in quality of care are most likely to result from multiple types of interventions to strengthen the health system and sustain improved case management.²⁶

Community level actions, both early detection, local treatment, and referral, as well as preventive and promotive interventions (for example, nutrition promotion, improved water, sanitation, and hygiene) will be essential to enhance the impact of improved child care in clinics and health centres.

In conclusion, this study has shown that under normal operating conditions and in a context of good facility infrastructure and management support, IMCI introduction is associated with improvements in some important aspects of care. Further strengthening of supervisory support with an emphasis on structured observations of clinical encounters and use of IMCI specific supervisory guidelines could improve practice further.