Methods

Evaluation of the community based injury prevention project, the WCIPP, was conducted over a three year period (1995–97) with funding external to the project. The overall goal of this evaluation was to ascertain the process and impact of the WCIPP model and the progress towards a reduction in injuries within Waitakere. The evaluation was also designed to provide information relevant to policy development.

Process evaluation activities conducted throughout the entire three year pilot phase included analysis of project documentation to obtain information relating to implementation; participant observation at monthly management group meetings and official presentations organised by the WCIPP; regular telephone communication and six monthly key informant interviews with each of the three coordinators; and, six monthly key informant interviews with either management group members and key council staff or representatives from community organisations in Waitakere (n=100 interviews in total). In addition, two post-implementation case studies were undertaken. One was conducted at the Waitakere City Council where the general and Pacific components of the WCIPP were based. The other was centred on the pan-tribal urban Marae where the Maori component of the WCIPP was based. Both case studies evolved when, after the review of the key informant interviews and project document, it became apparent that more in-depth knowledge of the institutionalisation of the WCIPP within these organisations would be helpful for future policy development. For both case studies, in-depth key informant interviews were conducted with individuals responsible for promoting wellbeing and safety within their organisations.

For the outcome evaluation a quasiexperimental research design was used. A comparison population (147 000) was matched to the intervention population (155 000) on a number of relevant variables (demographic characteristics, new housing developments, road safety, and safer community (crime prevention in New Zealand) coordinator positions in both councils. Three primary sources of data were collected: injury statistics; a pre-post intervention telephone survey of Waitakere and the comparison population (n=4000); and a pre-post survey of Waitakere and the comparison population organisations (n=144). Additional routinely collected data were also accessed from Land Transport Safety Authority and the Auckland Fire Service.

Morbidity data for admissions to public and private hospitals were obtained from the New Zealand Health Information Service National Minimum Dataset for the years 1989–98. Records for patients who had been hospitalised overnight as a result of receiving an injury were extracted for the areas of Waitakere, greater Auckland, and the comparison community. Injury hospitalisation rates were calculated from 1991 and 1996 census figures (for years not in a census year, population figures were estimated).

Analysis of trends was conducted for all age groups. As childhood injuries were a particular focus for the WCIPP, a separate analysis was carried out for children aged 0–14.

Logistic regression models were used to model the trends in the rates of hospitalisation over time. For children aged 0–14, ethnicity (Maori v non-Maori); gender; and community were included as variables in the model. For the analysis of all age groups, another variable was included for age group (0–14, 15–24, 25–64, 65+). Additionally, a scale factor was used to allow for over dispersion in the model. The logistic regression model was used to investigate whether hospitalisation rates for the three communities responded differently to the intervention. Separate regression lines were fitted for the pre-intervention period (1989–95), and the intervention and post-intervention periods (1996–98). A test was undertaken to investigate any differences between the communities in the change in slopes pre-intervention and intervention/post-intervention, thereby testing whether any intervention/post-intervention change in hospitalisation rates for Waitakere were significantly different from any changes in the comparison communities.

As mortality data are only available up to 1996, no analysis was undertaken on fatal injuries.

For the telephone survey a 15 minute questionnaire was designed to provide information on awareness of injury and injury prevention, knowledge of and involvement in the WCIPP, self reported injury and treatment sought, risk taking behaviours associated with road related injuries, and use of safety items.

Baseline data were obtained from a random sample of residents (16 years+) in August/September 1995. Post-intervention survey data were obtained two years later using the same interview criteria and questionnaire. Households were randomly selected and respondents who reported being over 16 years of age and having the most recent birthday were selected for interview. Eight callbacks were made to selected households before replacement by an alternative household. The same criteria and questionnaire were used to survey the comparison community. A total of 4000 residents were interviewed in this manner. To investigate pre-post differences within each of the communities χ² tests were undertaken of telephone surveys. Only p values ≤0.05 are quoted as significant.

To assess the impact of the WCIPP on organisations, a questionnaire was developed to cover awareness of the project and injury prevention and to assess changes in safety policies and practices. A total of 144 organisations were contacted in Waitakere and in the comparison community in 1995 and 1997. Organisations were matched and included the local council, preschool centres, schools, alcohol and drug prevention agencies, police, fire service, and rest homes. Analysis of organisational surveys were descriptive and no weighting or tests of significance were conducted. Routinely collected data were also accessed from the Land Transport Safety Authority and the Auckland Fire Service.

Results

PROCESS EVALUATION

WCIPP operation

There were two distinct phases to this three year pilot. First, a development phase covering the first nine months of operation, which involved the establishment of the management group, employment of coordinators, identification of priority areas for the pilot, and the development of a strategic plan. Details of this phase have been described elsewhere.¹³ The implementation phase covered the remainder of the pilot.

The management group was responsible for the development and implementation of objectives and strategies of the WCIPP and developed three major components to the project (Maori, Pacific, and a general population project) and appointed a coordinator for each. All coordinators had limited previous knowledge of injury prevention. However, they all had community development experience, knowledge of appropriate cultural processes, established community networks, and quickly developed knowledge and expertise in injury prevention. The general and Pacific components of this project were based within the council. The Maori component was based at the local pan-tribal Marae. All components came under the umbrella of the council. Analysis of all key informant interviews indicated a high level of satisfaction with the conduct of the WCIPP. They also consistently reported that the management group, coordinators, and council staff were well respected for their skills and ability to work effectively together.

After community consultation, seven priority areas (Maori, Pacific, children, young people, older people, alcohol, and road) were established, which provided an effective focus for activities. Working parties (including representatives from local community agencies and community representatives) were established from each of the priority areas. Analysis of key informant interviews indicated that this was an effective means of taking the project out into the community.

Implementation strategies

A balance of strategies was developed focusing on three broad areas of injury prevention: promotion; education and training; and advocacy and action for hazard reduction and environmental change. Analysis of project documentation, key informant interviews, and participant observation indicated that strategies operating throughout the implementation phase primarily focused on child safety. Examples included promoting correct child restraint use; burns and scalds education; safety practices while playing sport; preventing falls; correct use of cycle helmets; and the reduction of hazards around the home and school. Other strategies focused on the use of smoke alarms and home safety for older people. Findings from key informant interviews illustrate high levels of satisfaction with the quality of resources developed. Additional information on implementation strategies is provided elsewhere.¹⁴

Marae and Waitakere City Cuncil case studies

Key informant interviews showed evidence of increased awareness about preventability of injury at the Marae and case study findings also highlighted a strong network of support for the Maori project and its role in injury prevention. Improvements in lifestyles and the provision of safer environments for whanau (family) were also found. Examples included the installation of smoke alarms; improvements in pedestrian access; covering of drains; promotion of child safety days; and safe sports promotion.¹⁴

Raised awareness of injury prevention requirements by council and recognition that public buildings need to incorporate safety design criteria was highlighted in the council case study. Recognition of the need to provide information on safety issues was also found. Council now requires all leaders of major projects and programmes to state how their project meets, or furthers, safety. There was also recognition that the WCIPP provided an avenue for council to interact with the voluntary sectors of the community and thereby contribute to the social structure of Waitakere. The case study also identified that council had initiated environmental changes in Waitakere, specifically regarding safety for older people, playground safety, safe surfaces for sports fields, and town centre revitalisation.¹⁴

OUTCOME EVALUATION

Injury statistics

All age groups—

Waitakere was the only community to show a decrease in injury hospitalisation rates in both 1997 and 1998 (fig 1). Rates for the comparison community increased in 1996/1997 but decreased slightly in 1998. Rates for the rest of Auckland increased in 1996, decreased slightly in 1997, and then increased again in 1998. There was no significant difference between the communities in the change in slopes during the intervention/post-intervention period.

Children aged 0–14—

Waitakere was the only community to show a decrease in injury hospitalisation rates in the intervention/post-intervention period (fig 2). Rates for the comparison community dropped in 1996 but then continued to rise in 1997 and 1998. Rates for the rest of Auckland dropped slightly in 1996 but continued to increase in subsequent years. The change in slopes during the intervention/post intervention period was significantly different between the communities (p<0.05).

Telephone survey

A 63% response rate was achieved. Compared with pre-intervention and comparison data, a highly significant increase in awareness of injury prevention was found (p=0.001). No reductions in self reported injury for children or adults in the last month or 12 months were found in Waitakere, compared with pre-intervention and comparison data. However, significantly fewer Waitakere respondents, who reported being injured, required medical treatment compared with pre-intervention and comparison data (p=0.0001).

Compared with pre-intervention and comparison data significantly more Waitakere respondents acquired the appropriate type/number of child restraints (p=0.0001); fencing for swimming pools (p=0.0001); fire guards (p=0.0002); stair gates/guards for children (p=0.0001); and protective sports equipment (p=0.0001).

Organisational survey

Increases in awareness of the WCIPP and injury prevention among Waitakere organisations were also evident. Eighty five per cent of Waitakere organisations had heard of an injury prevention project compared with 25% in the comparison community. Half of the organisations had been contacted by someone from the WCIPP in the pre-post intervention surveys and 75% found the contact to be very useful. Changes to Waitakere schools' practices were found compared with pre-intervention and comparison data. Apart from council itself, minimal changes to other Waitakere organisations' safety policies or procedures were found.

Additional information/data

Land Transport Safety Authority data for 1998 showed that 98% of adults in Waitakere were appropriately restrained in the front seat, representing an annual increase of 7%. A 7% increase was also seen in children using appropriate child restraints (78%). Auckland Fire Service 1998 data found that Waitakere had the highest level of smoke alarm ownership in the greater Auckland region (71%), but unfortunately no comparison data with previous years were available.

Discussion

In 1998 Waitakere was accredited as the first WHO Safe Community in New Zealand. In this first evaluation of the WHO model in New Zealand, process and outcome measures demonstrated the powerful effect that the WCIPP had with regard to the institutionalisation of injury prevention within the local authority and Marae. There were also positive results in changes in awareness of injury prevention and positive changes in safety related behaviour. Evaluation findings highlight the appropriateness of adopting a community injury prevention model as a basis for addressing injury prevention. This finding has been reported elsewhere.^11,15,16

In accordance with good process evaluation practice, strategies were designed to gather information on project operation and reach, levels of satisfaction, and quality of resources produced. Process evaluation findings indicated that the role of the project coordinators was pivotal to the success of the WCIPP and that the three components of the WCIPP provided an important cross cultural structure. However, there are at least seven different Pacific nations within the Pacific community in Waitakere, making the task of implementing injury prevention strategies within the migrant community challenging. It is important that the flexibility required to establish appropriate cultural procedures and practices for all cultures represented in the community is recognised.

The establishment of working groups in the priority areas was successful and good intersectoral collaboration occurred. A balance of injury prevention strategies for the three broad areas of promotion, education and training, and advocacy and action for hazard reduction and environmental change was achieved. As expected, the scope of the strategies developed varied. Most of the child safety initiatives operated for the entire implementation phase, while others, such as the Firesafe Waitakere Campaign, were one off events. However, although the objectives of the WCIPP were achieved by the strategies implemented, not all age groups or injury types were covered. While few process evaluations of WHO Safe Community projects have been reported, those that have included information on implementations strategies, also state that not all injury types could be addressed.^18,19 The expectation that this could occur may be unrealistic given the level of resources provided and highlights the possibility that the WHO criteria for Safe Communities may not be entirely appropriate in settings where contracted outputs specify priorities.

Case study findings indicated that the Maori coordinator built a strong network of support for the Maori project at the Marae. Pivotal to the success of this project was the development of a Maori perspective on injury prevention which supported Maori protocol and encompassed a holistic view of health and wellbeing at the Marae. This component of the WCIPP was an excellent model of a diverse injury prevention project, aspects of which could be transferred to other indigenous communities.

Within Waitakere City Council, especially in those sections most likely to consider safety in their policies, there was good support for the WCIPP and the part it played in the creation of a safety culture within the council and the community. The synergy created through the collaboration of the WCIPP coordinators and council representatives resulted in safety being given a much higher priority. This is on-going, as illustrated by the current safety requirements of council. It is also significant that council values the community links that the WCIPP has achieved and the long term benefits that are likely to occur as the community develops a greater sense of trust in local government.

In this first outcome evaluation of the WHO Safe Communities model in New Zealand, significant reductions in the rate of injury hospitalisations for children were achieved in Waitakere, whereas the comparison community showed an increase. This is an important finding as child safety was one of the major campaigns conducted by the WCIPP. Reductions in serious injuries have not yet been able to be demonstrated in similar projects conducted in Australia.^12,18

The WCIPP was not able to demonstrate significant reductions in injury hospitalisations rates for all age groups. However, it is very encouraging that Waitakere was the only community to show a decrease in injury hospitalisation rates for two subsequent years (1997/98). Evaluation of the effects of the WCIPP on fatal injury statistics is on-going.

The WCIPP was unable to replicate findings of significant reductions in self reported injuries for either adults or children.¹² It is unclear why this has occurred. One explanation could be that the activities of the WCIPP have resulted in more Waitakere residents being aware of injury as a public health issue and this knowledge resulted in a more accurate recall of self reported injury.¹⁹

The WCIPP was successful in raising awareness of injury prevention among individuals and community organisations. Positive increases in adoption of appropriate behaviours and practices in the area of child safety were also found. Significantly more Waitakere respondents had obtained fire guards; stair gates/guards for young children; appropriate type and number of child restraints for motor vehicles; and protective equipment for sports, post-intervention and compared with the comparison community. The combination of an increase in awareness of the WCIPP and environmental change are important steps for the reduction of injuries.²⁰

Summary

The community injury prevention model appears to be an effective strategy for injury prevention. This pilot was implemented within a large urban multicultural community. The findings suggest that the WHO Safe Communities model worked well under the umbrella of a local government authority. Given the many areas of influence that local authorities have on safety issues and structures, findings of this evaluation add support to the placement of future community injury prevention projects within local government. In addition, the positive findings in the child priority areas adds weight to the view that community based injury prevention projects should consider fewer, more targeted, interventions.