Many of the determinants of health lie outside the medical realm. Public health protagonists seek to influence policy making outside the healthcare sector in favour of health. Derived from environmental impact assessment, an emerging tool for this is health impact assessment (HIA). HIA is defined as “a combination of procedures, methods and tools by which a policy, programme or project may be judged as to its potential effects on the health of a population, and the distribution of those effects within the population” (Gothenburg consensus paper, 1999).¹ In practice, the emphasis is often placed on community consultation and on formulation of recommendations for a health maximising implementation of the policy or project at stake, with less attention for the actual health consequences that might result.

Quantification of health effects in HIA has a number of advantages. Firstly, knowing the size of an effect helps decision makers to distinguish between the details and the main issues that need to be addressed and facilitates decision making by clarifying the trade offs that may be entailed.² Secondly, adding up all positive and negative health effects into a net effect permits the use of economic instruments such as cost effectiveness analysis, which further aids decision making.³

However, there are two difficulties in quantification: the availability of valid data, and the availability of methods to analyse the data and translate them into information on the health effect of the proposal under scrutiny. In this contribution we focus on the second problem and analyse reports of HIAs performed to date, using a framework similar to that proposed by Joffe and Mindell⁴ in which policy decisions influence health via its determinants (see fig 1). This divides the HIA process in two steps, which for brevity we will refer to as “exposure impact assessment” and “outcome assessment” respectively. This paper addresses two questions: Firstly, what methods are used in quantitative exposure impact assessment, and secondly, what methods are used for quantitative outcome assessment?

METHODS

Search strategy

HIA case studies were obtained by searching (1) electronic sources (http://www.who.int/hia and the links it contains, accessed 20 Jul 2004), (2) the reviews analysed in Taylor and Quigley’s “HIA review of reviews”,⁵ and (3) the complete collection of Dutch national level HIA cases provided by the national coordinating agency for intersectoral health policy (Ondersteuningsfunctie Integraal Gezondheidsbeleid). (4) To include recently published cases we searched PubMed for the period 2002 until 1 August 2004. Strings used were “health AND impact AND assessment”, and “impact AND (assess* OR eval*) AND (policy OR policies)”. Resulting articles were first judged by title and abstract, and obtained in case of possible reference to case studies. Promising reports were requested from the authors.

Selection

We included reports of primary studies that were prospective and assessed the impact of non-health sector policy decisions. Descriptions of case studies in published articles and reviews were used if they contained sufficient information, but in most cases the original reports were obtained. We excluded reports of studies that had not been completed at the time the report was written, reports that were very restrictive in the health outcomes they presented (such as studies on the effect of bicycle helmet use on fatal injuries), and studies that only screened whether a particular proposal was health relevant. We also excluded reports in languages other than English, French, German, Spanish, or Dutch.

Analysis

We distinguish between methods used to estimate the effect of policy on determinants, and methods that calculate health impact given those changes in the exposure to determinants. The estimates of exposure to determinants and of health outcomes were assessed for quantification. Quantification was defined as the expression in numerical terms of the change in health status of a specific population that can be attributed to a specific policy decision.

RESULTS

Figure 2 shows the results. The electronic sources yielded 83 cases of prospective HIA; 12 possible cases could not be obtained. This partly overlapped with the 25 studies obtained from the reviews, of which two possible cases could not be obtained. Out of the collection of Dutch HIA cases 12 studies were included. The search in PubMed added eight cases; two possible cases were not obtained. Of the total of 98 cases, 72 were from the UK, and 12 from the Netherlands (table 1). The Dutch HIA studies focus on national level policy, while all but one of the other studies assess local or regional level projects or programmes.

Of the 98 studies, 17 gave quantified estimates of the effect on determinants of health. Table 1 gives a description of the studies. Ten of the studies deal with physical infrastructure for industry or transport, the remaining seven focus on a variety of projects and policies. Sixteen studies proceed to present health outcomes.

Table 2 provides an overview of the determinants that were quantified, and on what sources of information the estimates were based. We will now discuss the methods used for the assessment of exposure impact and outcome for the 10 (categories of) determinants for which quantified health outcomes were presented.

Carcinogens

For carcinogens and other environmental factors computerised dispersal models were used to predict spread and subsequently the exposure of human populations was estimated.^6,10,20 In the outcome assessment phase, existing computer programs and toxicological data were used.^6,10

Particulate matter

Particulate matter with a diameter less than 10 μm in size (PM10) was used as proxy for air pollution.^14,18,21,22 Models similar to those for carcinogens were used to estimate future exposure. For the outcome assessment authors made use of epidemiological data.²⁶

Road transport: vehicle kilometres

The common determinant used in most predictions of traffic injuries and fatalities is vehicle kilometres—that is, the total number of kilometres driven by vehicles. In four of nine cases no independent exposure impact assessment was done: the estimates are obtained from the project planner without mention of the methods used. The methods to estimate health effects given the number of vehicle kilometres differ. In three studies the expected increase in vehicle kilometres is simply multiplied by the local or national accidents rates per kilometres^14,15,17 while others use more complex methods and take into account road type and mode of transport²⁵ or traffic flows and types of intersections.^7,18 Jobin uses data from the US and multiplies the accident rate by 10 to estimate the number of truck related casualties in Cameroon.¹⁹

Employment

Five studies provide estimates of the employment effects of plans. In four cases no independent assessment was done and the estimates were obtained from documents pertaining to the project plan.^15–17 In one case it was estimated on the basis of financial investments.¹³ Only one author provides an estimate of the effect on health (in the form of premature mortality) based on extrapolation from a longitudinal study.^18,27 Others cite this study but conclude that the size of the effect is difficult to estimate.

Income

Kemm estimates the effect of the cash injection into the local economy represented by the creation of a botanical garden and divides this by the number of residents in the population. In the outcome assessment phase data on the relation between income deciles in the UK and mortality are used to estimate the reduction in mortality.¹⁷

Alcohol

Zwart equates the alcohol sales decrease predicted in an independent economic analysis with alcohol consumption decrease on a population level.⁸ In the outcome assessment phase he applies the Ledermann formula to estimate the number of excessive drinkers. This formula supposes a log normal distribution of alcohol consumption in a population.

Smoking

Mooy and Gunning-Schepers use economic literature on the price elasticity of cigarettes (that is, the relation between price and sales) to estimate the change in the number of smokers resulting from increased tobacco taxing. For the outcome assessment the Prevent model is applied, a macro-simulation model with a dynamic population that incorporates epidemiological data.⁹

Physical activity

Pitches and Kemm estimate the effect of changes in transport infrastructure on the annual distance cycled and walked. This is translated into the number of people that would move from being sedentary to physically active. For the outcome assessment phase they constructed a simple model using survey data on physical activity, mortality and morbidity statistics, and relative risk estimates from meta-analyses and estimate the number of lives (or cases of disease) saved if 1000 persons would increase their activity level.²⁵

Housing

Kemm et al estimate the effect of home insulation on the “comfort range”, the lowest outside temperatures at which the in-house temperatures can be kept at or above 16°C. In the outcome assessment this estimate is linked to ecological data on the effect of outside temperature on overall mortality to arrive at the number of deaths avoided.¹³

Infectious disease

To estimate the burden of malaria attributable to the construction of an oil pipeline in Cameroon, Jobin used Nigerian data in combination with assumptions on the effectiveness of preventive and curative measures. For HIV a computerised transmission model and epidemiological data on prevalence were used.¹⁹ Finally, chances of causing variant Creutzfeldt-Jakob disease by burning cattle carcasses were estimated using a dispersal model.²¹

Type of evidence

Table 2 shows that in exposure impact assessment most assessors used the results of other reports pertaining to the policy under scrutiny, such as environmental impact assessments for exposure to chemical substances or project plans for the expected amount of employment generated. Other sources of evidence are (routine) data, measurements by the researchers, and pre-existing models (which contain data and assumptions). In some cases the evidence is supplemented with the author’s own assumptions, and in two studies expert opinion was sought. A review of the literature was used explicitly in only one study. In contrast, outcome assessment was commonly based on literature reviews, as well as on routine data and pre-existing models. Expert opinion (other than that of the authors) was not used in outcome assessment.

Analysis of methods

Table 3 gives a further analysis of the 16 studies that presented health outcomes. This shows that the types of health outcomes used differ greatly between studies. Furthermore, 14 of the assessments are limited to the effects of proximal determinants, while two studies also include effects of determinants that may be considered distal (defined as exerting their influence via intermediate factors): increased income and employment. The methods used for outcome assessment vary, but are generally similar for comparable determinants. In most studies, the risk measures were the result of epidemiological research, while in three studies toxicological risk measures (derived from animal experiments) were used. The time horizon is unclear in most assessments. In three of the studies in which it is clear, it is determined by the risk measure used: toxicological risk assessment assumes lifetime exposure (70 years by convention), while in one study the properties of the simulation model limit the time horizon to 50 years. Uncertainty in outcomes is seldom made explicit by more than qualitative terms like “about”.

DISCUSSION

From policy plan to determinants of health: exposure impact assessment

The methods used to estimate effects on determinants of health are quite diverse, which is not surprising considering the diversity in factors that influence health. For physical and chemical factors methods are well developed, and also for traffic flows and accident rates models are available.

As a consequence of a narrow evidence base no such models are available for many other determinants. In the cases we reviewed, estimates were commonly made on the basis of (unpublished) data or information provided by project developers. The latter source may introduce systematic bias. Author’s assumptions and expert opinion are options of last resort. In the absence of standardised, validated methods and readily applicable data, some authors display substantial creativity in quantifying socioeconomic determinants. These efforts should be critically evaluated so that they may contribute to the development of a more uniform and robust approach.

From determinants to health outcomes: outcome assessment

From the 17 studies that quantify the effects of the policy decision on determinants of health, 16 proceed to give estimates of the effect on health outcomes, although only five do so for all the determinants they identified as relevant. This compares favourably with the findings of a study on the inclusion of health in environmental impact assessments, which concluded that most studies quantified up to the level of determinants (or pollutants) and compared these with limit values, thus not extending the analysis to health aspects.²⁹ In the case of non-carcinogenic pollutants that do not reach the limit value, this is justified by the generally accepted assumption that for these substances there is a threshold, below which there is no health effect.³⁰ However, in case this threshold is exceeded an estimate of the health effects would be desirable.

Few socioeconomic and behavioural determinants are quantified up to the level of health outcomes. One of the problems may be that a stable evidence base is lacking. Unlike physical and chemical substances, socioeconomic and behavioural determinants are context dependent. For example, being unemployed in Russia is not the same as being unemployed in Germany. This means that the evidence is only to a limited extent generalisable across time and space, and that the degree of standardisation achieved in environmental HIA will be hard to match in HIA that focuses on other policy areas.

Outcome assessment is often done for different risk factors separately as in Hallenbeck and Fehr’s assessments of waste facilities, and in other models such as Mindell and Joffe’s instrument for predicting the health consequences of air pollution.^6,10,31 However, the separate health effects resulting from a policy cannot always simply be added up as this may result in double counting.³² A possible method for the integration of different effects is used in the HIA on tobacco policy (the PREVENT model)⁹ and in McCarthy’s ARMADA model for environmental HIA.²⁴ Both use simulation models that combine epidemiology and demography to assess various effects of a proposal on the health of a population. Differences by age and sex and competing risks can be accounted for. Further development of such models that can be used “off the shelf” could do much to improve quantitative HIA.⁴

Indicators for health outcomes

The measures of health outcome used in the different studies are quite diverse, ranging from numbers of deaths in a specified population, to hospitalisations for asthma and injury only accidents. This diversity is justified by differences in the research questions that need to be answered, but it hinders comparison of effects. It would be useful to additionally express health outcomes in a summary measure of public health such as the disability adjusted life year (DALY).^{24,33,34,37,38} DALYs combine life years lost (or gained) and time spent with disease, adjusted for the severity of that particular disease, into a single indicator. The concept has been criticised^35,36 and should not replace more conventional health outcomes, but for decision making on population level it can be a useful tool. A limitation for use in HIA is that disability weights are only available for diseases as distinguished in the international classification of diseases (ICD), so that, for example, annoyance effects attributable to noise or odour cannot at present be expressed in DALY. Ideally, the aggregate health impact is subsequently differentiated for (vulnerable) subgroups: health inequalities impact assessment.

Data requirements

Whatever shape it takes, quantification in HIA will be limited by the availability of relevant and reliable data. The more detailed the techniques, the higher the information requirements. For example, demographic computer simulation models can cope with differences by age and sex, but the model has to be filled with data that specify these differences. Taking into account health inequalities also increases information needs. Our review confirms Joffe and Mindell’s finding that the evidence base is especially narrow when it comes to linking policy options to health determinants. The creation of databases containing evidence for both the exposure assessment and outcome assessment phases of quantitative HIA would greatly facilitate HIA practice.⁴

What to do when the data do not permit quantification?

Data problems commonly hamper attempts at quantification. If that is the case, robust qualitative work may be the best option. However, before concluding that quantification is not possible it may be worthwhile to bear in mind that the perspective of a decision maker differs from that of an epidemiologist. Not taking any decision is not an option for a decision maker, while the cautious epidemiologist may conclude that further study is needed. An expert’s guess may still be better than no guess at all. The use of a structured process to obtain expert opinion can improve its validity and credibility. A suitable method for obtaining the collective opinion of experts is the Delphi process. Characteristics of this method are anonymity, iteration, controlled feedback, and statistical summarisation of the group responses. It has been used in “future studies”.³⁹ The outcome of a Delphi study may serve as input for outcome assessment, together with the epidemiological evidence and local data.

In HIA of policies that are broadly formulated, or where there is much uncertainty over trends and future developments, the analysis of a number of scenarios might be more informative than a single estimate of the most probable impact. This permits various assumptions to be made without losing scientific credibility, and may convey to decision makers an understanding of the dynamics of the mechanics described by the model.

Validity and reliability

Little is known about the validity and reliability in HIA. For some of the methods used in HIA, validity and reliability have been assessed,^40,41 but for many methods no such research has been done. Likewise, methods to assess the validity and reliability of complete HIA studies are yet to be developed, and even agreed upon definitions suitable for HIA are lacking. We would tentatively define the validity of HIA studies as the degree to which the predicted health effects are confirmed by empirical research. This implies a need for outcome evaluation, notwithstanding the difficulties this will entail.

Standardisation

Once methods for quantitative HIA have been developed and their validity is becoming clear, the need for standardisation will arise. Similar to developments in the field of health economics (for example cost effectiveness analysis), guidelines will be needed to determine what effects to include, what time horizons are appropriate, how to deal with uncertainty, and what are suitable indicators of health outcomes. Standardisation will increase comparability among studies and promote HIA as a reliable and credible instrument for intersectoral health policy making.

Limitations of this study

The HIA reports included in this review do not give more than an indication of what is done in the field. Reports of HIA studies are difficult to obtain as they seldom appear in peer reviewed literature and are not always made public. In Canada for example, HIAs are performed by proponents of projects and incorporated into environmental impact assessment (EIA) reports. Several other countries are probably underrepresented in our sample for similar reasons. In contrast, in the UK there is a tradition of local level HIA separate from EIA and of making studies available via the internet. This results in the inclusion of many small scale studies with little emphasis on quantification. As we did not exclude cases on the basis of thoroughness or amount of resources invested, this partly explains the preponderance of studies from the UK in our sample, and the low proportion of HIA reports with quantified effect estimates.

Conclusion

We conclude that quantification in HIA is useful but not often achieved and that validity is often uncertain. Quantitative HIA would benefit from research and development of (1) methods to quantify the effect of socioeconomic and behavioural determinants; (2) the development of user friendly simulation models for outcome assessment in HIA; (3) the additional use of summary measures of public health; (4) the use of expert opinion and scenario building in HIA; (5) empirical research into the validity and reliability of methods for HIA, and of complete HIA studies.