Remarkable variations in the care provided by individual physicians are widespread and well documented, and reflect the extent of clinical uncertainty about treatment.¹ Researchers at RAND and UCLA developed a method for combining the best available research evidence with expert opinion to assess the appropriateness of treatment.^2,3 Modifications of this method have been widely used to develop quality indicators for health care.

Quality indicators have been used to show that effective health care is delivered only about half as often as it could be in the USA,^4,5 and as many or perhaps even more opportunities for effective care may be being missed in British general practice.^6,7 However, we still know surprisingly little about how many people with common medical conditions in the UK receive effective interventions when needed. An independent review published at the end of 2003 concluded that we do not have enough data to know whether the NHS is getting better or worse, particularly for conditions not covered by national initiatives.⁸ Better measurement of quality at health system level is essential for quality improvement.

Quality measures can be classified into three types. Structure measures consider factors such as the availability of facilities and staff, process measures consider whether treatment adheres to agreed good practice, and outcome measures consider the resulting changes in health status. There are advantages in using measures of process: (1) processes are a more efficient measure of quality;⁹ (2) for most conditions there is insufficient information to adequately adjust outcomes for differences in case mix between providers; and (3) processes of care are amenable to direct action by providers.¹⁰ Where the evidence exists, the healthcare processes should be linked to an improved health outcome.

The thoughtful use of the right quality indicators has the potential to improve health outcomes, at least in specific disease areas.^11,12 However, concerns about the increased use of quality indicators remain. The desire to quantify may lead to neglect of other areas of health care where quality is particularly difficult to quantify. This is particularly relevant for older people⁵ who are major recipients of health care, often for several conditions at once,¹³ delivered across the boundary between primary and secondary care. There have also been problems with the implementation of quality indicators, and in England the Department of Health has been criticised for using indicators that are not based on evidence of effectiveness¹² and for using indicators where the NHS lacks capacity to produce valid data.⁸

The possible neglect of some care may be reduced by more engagement of the public and patients in the implementation of quality indicators. One approach to public involvement is to assess quality through interviews with patients, an underused source of data about healthcare quality which avoids the problems of extracting data from clinical records.¹⁴ The risk of excessive focus on a few specific areas may be less if indicators describe a standard which, if not met, would almost certainly identify poor quality care rather than the optimal care described by practice guidelines.

This paper presents an independently developed set of evidence based indicators capable of measuring the quality of health care of older adults at the level of the health system by covering a broad range of conditions simultaneously, across primary and secondary care, and suitable for use in an interview survey of patients. They are intended for use in interview surveys, including the English Longitudinal Study of Ageing (ELSA), a new national cohort study that offers the opportunity to measure quality of care systematically from a population perspective in 2004.¹⁵

METHODS

The development of evidence based quality indicators de novo is resource intensive so we adapted a comprehensive set of evidence based indicators of care quality for older people published by RAND in the USA in October 2001. The Assessing the Care of Vulnerable Elders (ACOVE) project developed 236 quality indicators in 22 clinical areas for the health care of community dwelling people aged over 75 years.¹⁶ The ACOVE conditions were chosen according to their importance in older people and the potential for quality improvement. For inclusion in our study, clinical areas were selected on the basis of both prevalence and importance as a cause of disability for people aged 65 years and older in the Health Survey for England 2000. For these clinical areas we selected all 93 ACOVE quality indicators that could be used in an interview survey without the use of clinical records. Cross checks by the ACOVE researchers between interview data and medical notes where feasible had found that the ACOVE interview data were comparable with notes data, and for some indicators it appeared that respondents remembered higher rates of appropriate interventions than had been documented in the notes.

There are many similarities in the evidence base for medical practice in England and the USA, but considerable differences in the way that services are organised and financed. Previous research has suggested that quality indicators cannot simply be transferred directly between countries without an intermediate process to allow for variation in professional culture or clinical practice.¹⁷ We therefore recruited a panel of 10 clinical experts to rate the 93 ACOVE indicators for appropriateness for use in England, alongside another 26 indicators suggested by the panel using a modified version of the RAND/UCLA appropriateness method.² The method used two rounds of anonymous ratings on a scale ranging from 1 to 9, with a face to face discussion between rounds. Panel members were supplied with literature reviews summarising the evidence base for each quality indicator.

The 16 clinical areas covered by the indicators are: continuity of care, dementia, depression, diabetes mellitus, falls and mobility disorders, hearing loss, hypertension, ischaemic heart disease, medication use, osteoarthritis, osteoporosis, pain management, screening and prevention, stroke and atrial fibrillation, urinary incontinence, and vision. The indicators were also classified into five domains of care: continuity and follow up, diagnosis, prevention, screening, and treatment. The domains of prevention and screening include the indicators in the clinical area of screening and prevention as well as those indicators in other clinical areas that refer to either screening or prevention. For example, the indicator “All persons aged 65 or older should be offered a hearing screen at least once” is in the “hearing” clinical area as well as the “screening” domain.

The panel was recruited through national organisations in order to reach respected clinicians with access to established networks of colleagues. Members of the British Geriatrics Society’s Special Interest Groups, the Royal College of General Practitioner’s clinical network of general practitioners with a special interest in care of the elderly, and the Royal College of Nursing’s Gerontological Nursing Programme were invited to review the set of proposed indicators. Six consultant physicians in geriatric medicine or medicine for the elderly, three general practitioners, and one nurse made up the final panel. The pack containing the indicator set and rating forms was also sent for review to a senior consultant in old age psychiatry, a senior public health doctor, and the policy officers from Help the Aged and Age Concern, UK charitable organisations for older people. Their comments were considered at the meeting.

RESULTS

A total of 119 quality indicators were presented to the clinical panel which accepted 102 (86%) as being valid for use in England; 17 (14%) were rejected as invalid. Seventy nine of the 93 ACOVE indicators (85%) were approved with no or minor amendments, and 23 of the 26 new indicators suggested by panel members (88%) were approved.

Examples from each clinical area of the indicators rated as valid are given in box 1. The full set of quality indicators rated as valid is shown in Appendix 1 (available online at www.qshc.com/supplemental). Examples of indicators not rated as valid are given in box 2 and the full set of rejected indicators is shown in Appendix 2 (available online at www.qshc.com/supplemental). Only one indicator, which recommended advice about the pharmacological prevention of osteoporosis, was rejected due to disagreement.

Table 1 shows the number of indicators proposed and accepted in the 16 clinical areas: the number of accepted indicators varied from 3 (for continuity) to 12 (for medication use). Table 2 shows the number of indicators proposed and accepted in each of the following five domains of care: continuity and follow up, diagnosis, prevention, screening, and treatment. All of the 58 indicators in the treatment or continuity and follow up domains were accepted, but only just over half (13 of 24) of those in the screening domain were accepted.

DISCUSSION

This research reports on quality indicators developed in the USA for older people that have also been judged as valid for use in England. One hundred and two indicators of the quality of health care for people aged 65 years and older in England were rated as valid by an expert clinical panel and 79 of these are essentially unchanged from the ACOVE indicators from the USA. All 58 indicators about treatment or continuity and follow up were accepted, but only half (13 of 24) of the indicators about screening were accepted. This more conservative approach by the English panel than the USA panel to indicators about screening perhaps reflects the experience of general practitioners in England with the implications of implementing existing screening programmes—for example, breast and cervical screening. The panel discussion reflected concern about a requirement for general practitioners in England to annually seek out all older adults registered with them to ask about, for example, urinary incontinence. In the USA primary care physicians do not generally have responsibility for a population.

Further work is required to operationalise these quality indicators before use. In general, each indicator requires a minimum of two survey questions, at least one to determine whether the respondent has been diagnosed with the relevant condition, and then at least one more question to determine whether the respondent has received the intervention specified by the indicator. When applying quality indicators it is important to consider what aspects of health care are being measured.¹⁸ Quality can be measured across the following six generally accepted dimensions: effectiveness, efficiency, patient centredness, access, equity, and safety. The quality indicators presented in this paper are designed primarily to measure effectiveness, although they can contribute to some extent to the measurement of other dimensions of quality. Other approaches to quality assessment should be used as well to better capture other dimensions—for example, economic tools to measure efficiency.

Lending convergent validity to these indicators are the results from a study in the UK in 1997 which used the RAND/UCLA method to develop review criteria for clinical notes in general practice for angina, asthma, and non-insulin dependent diabetes mellitus.¹⁹ Our study concerned quality indicators for use in interview surveys of older adults, but the indicators in areas of clinical overlap are very similar to the 1997 study. For angina both panels recommended treatment of raised serum cholesterol levels, use of aspirin or antiplatelet therapy, and advice about smoking cessation. For diabetes both panels recommended annual measurement of glycosylated haemoglobin and of proteinuria, annual examination of the feet and of the fundi through dilated pupils (although this panel recommended that the eye examination be done by a specialist rather than by any doctor), monitoring of blood pressure, treatment of raised serum cholesterol levels (although this panel set a level of 5 mmol/l as opposed to 7.8 in 1997). This panel also recommended the use of angiotensin converting enzyme inhibitors or angiotensin receptor blockers and daily aspirin therapy which were not recommended in 1997. The development of new indicators by similar processes conducted 5 years apart shows that quality indicators need regular updating to allow for advances in medical knowledge.

The adaptation of this US set of indicators for use in the UK has two main advantages. Firstly, since the development of quality indicators is resource intensive, it is desirable for developed indicators to be shared internationally where possible rather than developed separately in different countries. Secondly, use of the same indicators in the US and UK will allow international comparisons of the quality of healthcare processes between the two countries. These indicators are designed to measure the quality of health care at the population level. They are not designed to measure the quality of health care received by an individual patient, or to underpin appraisal, assessment and revalidation of individual professionals. These are important tasks, but they require a different approach to standard setting.^20,21

Use of these indicators in a population survey to measure health system performance will allow comparisons to be made about the quality of health care received by different population groups, whether defined by socioeconomic status, wealth, health status, geography, age, or sex. Two studies using RAND quality indicators have recently shown large gaps in the quality of care in the US,^4,5 but the paucity of data about the quality of health care in the UK is currently hindering quality improvement.²¹ The use of these indicators will provide strong signals about where quality improvement is needed, and will act as a stimulus to target quality improvement efforts at the areas with the greatest potential for improved patient care.