METHODS

Selection of conditions

The RAND quality indicators (termed the “QA tools”^9–12) address 58 clinical areas and were developed between 1995 and 1999 using a modification of the RAND/UCLA method of systematically combining evidence with expert opinion.¹³ This process involves the development of a draft set of indicators following a comprehensive review of the literature and then the rating of these indicators by experts for their validity as measures of quality. The process is described in detail in box 1.

In this study we wanted to develop quality indicators for the most common conditions seen in British general practice. We chose 19 of the 58 US conditions, concentrating on those conditions with the highest consultation rates in general practice based on data from the UK National Morbidity Survey¹⁴ (table 1). The UK indicator set was developed between September 1999 and April 2001.

Literature reviews

At the time of the UK study the US literature reviews from which the indicators (and the rating of the indicators) are derived were 3–5 years old. We therefore decided to update the reviews, using leading UK primary care researchers with a specific interest in each condition. In general the UK reviews differed from those from the US through the inclusion of additional papers judged relevant to UK primary care rather than through the inclusion of more up to date evidence. The UK reviewers were asked to examine the US quality indicator set and, where appropriate on the basis of their experience and the modified literature reviews, to remove or suggest additional quality indicators.

Selection of expert panels

Two panels of UK general practitioners were recruited by inviting participation from all 196 doctors who had been awarded the Fellowship by Assessment (FBA) of the Royal College of General Practitioners, the highest quality award for general practitioners in the UK; 75% agreed to take part. From these we purposefully selected nine members for each of two panels on the basis of sex, time since qualification, practice characteristics, and geographical location.

Rating the indicators

Reproducing the RAND/UCLA process used to develop the US indicators, the panel members were sent the draft UK set of indicators and their supporting literature reviews by post. Panellists were asked to rate each indicator on a continuous 9-point scale in terms of its validity as a quality indicator and the “necessity to record” the information in a patient record (box 2).¹⁰ One of the panels rated the indicators for ten of the conditions and the other for nine of the conditions.

The first round scores for validity and necessity to record were fed back to each of the panels at two-day face to face meetings chaired by PS/MM and MM/SC. All the indicators were discussed, first round scores for validity and necessity to record were presented, the wording of the indicators was modified where necessary, and each indicator was then re-rated.

Comparison of US and UK indicators

The second round ratings were used to select the set of UK indicators for comparison with the US parent indicators. To ensure comparability with the US set for the analysis presented in this paper, we used the same cut off points for US and UK sets (validity scores ≥7, necessity to record scores ≥4) without disagreement within the panel (three or more of the nine ratings for an indicator being in both the top and bottom third of scores¹⁰). The UK indicators that we finally published¹⁵ were selected on the basis of more stringent cut offs for validity (≥8) and necessity to record (≥6). Although acute diarrhoea in children was included in both indicator sets, we excluded it from this comparison as the use of different age cut offs by the panels made valid comparison impossible. We also excluded US indicators that related to hospital based procedures such as during admission after acute myocardial infarction as UK primary care physicians do not normally provide inpatient care. The comparisons reported in this paper are therefore based on 18 conditions. For each indicator in the US set we identified whether there was an exact or near equivalent indicator in the UK set. Examples of indicators which were classified as “near equivalents” are shown in box 3. Table 2 gives examples of indicators which were different.

RESULTS

Ninety eight of the 174 US indicators (56.3%) had near or exact equivalents in the final UK set (table 1). US indicators could have been discarded either by the UK reviewers if they were clearly not relevant to UK general practice, or during the panel process, or as a result of the panel scores; we did not attempt to distinguish between these as the purpose of this analysis was to compare the overall outcome of the two processes.

For the 159 indicators in the US set for which it was possible to classify strength of evidence, there was a slight but non-significant relationship between the strength of evidence for an indicator and the probability of the indicator having a near or exact equivalent in the final UK set: level 1 evidence, mainly randomised controlled trials, 64.3% (18/28); level 2, mixed evidence, 58.9% (10/17); level 3 evidence, mainly expert opinion, 54.4% (62/114), χ² 0.96 df=2; test for linear trend not significant (p=0.34).

A few of the conditions are described in more detail below to illustrate the main reasons for the differences between the US and the UK indicators.

Asthma

Of the 17 indicators in the US set there were exact or near equivalents for five in the UK set. This discrepancy appeared in part to be related to the approach the panels had taken to the indicators rather than to fundamental differences in management. In particular, the US panel had eight indicators relating to care for acute exacerbations in the physician’s office compared with only four in the UK set. Two US indicators related to theophylline which is rarely used for asthma in the UK.

Cervical screening

Of the seven indicators in the US set there were exact or near equivalents for three in the UK set. The main reasons for the discrepancy were lower thresholds for action in the US set. Examples of these included shorter routine smear interval (3 years in US v 5 years in UK) and lower threshold for colposcopy (two moderately abnormal smears in the US v three in the UK). However, in one instance the UK panel recommended earlier action (repeat smear of colposcopy after moderately abnormal smear within 1 year in US v 6 months in UK).

Coronary artery disease

Of the 18 indicators in the US set, 13 related to hospitalised patients and so were not appropriate to UK primary care. The remaining five had exact or near equivalents in the UK set.

Depression

Of the 17 indicators in the US set there were exact or near equivalents for nine in the UK set. Two of the differences appeared to relate to expected standards of documentation. For example, the US panel specified enquiry about current medication when depression was diagnosed. The UK panel rated this as valid, but not necessary to record, probably because this information would already have been in the record of the UK primary care practitioner. In one case (see table 2) resources do not exist in the UK to provide the care recommended by the US indicator. UK panelists would therefore be unlikely to regard admission under these circumstances as necessary. Most of the other discrepancies related to a higher level of detail in the US set.

Diabetes

Of the 12 indicators in the US set there were exact or near equivalents for six in the UK set. The most common reason for this discrepancy was more frequent monitoring specified by the US panel (table 2).

Dyspepsia

Of the 10 indicators in the US set there were exact or near equivalents for five in the UK set. All but one of the discrepancies related to actions to be taken following endoscopy. The UK panel therefore appears to have concentrated more on initial diagnosis and treatment.

Headache

Of the 20 indicators in the US set there were exact or near equivalents for 13 in the UK set. The UK set indicates a generally more conservative approach—for example, no indication for MRI based solely on the severity of headache, shorter list of drugs for first line prophylaxis of migraine, and no trial of tricyclic drugs required for moderate to severe tension headache (table 2). However, the UK indicators did require documentation of psychosocial history for patients with new headache (which the US indicators did not). This is consistent with the hypothesis that UK primary care physicians take a less biomedical (as opposed to biopsychosocial) approach to the problem.

Hypertension

Of the 12 indicators in the US set there were exact or near equivalents for 10 in the UK set. For the other two, one related to more frequent screening required by the US panel (table 2) and, in the other (physical examinations to be carried out on new hypertensive patients), the UK panel gave lower validity scores than did the US panel.

Osteoarthritis

Of the seven indicators in the US set there were exact or near equivalents for four in the UK set. There was a major difference in the interpretation of the benefits of radiological investigation (table 2)

Otitis media

Of the three indicators in the US set there were exact or near equivalents for two in the UK set. There was no equivalent in the UK set for the US indicator that required at least 10 days of treatment with antibiotics for acute otitis media in children aged 1–3. Indeed, the UK set contained an indicator advising that, for children over 2 years, antibiotics should not be prescribed unless there were complications or persistent pain or discharge (over 72 hours).

Respiratory tract infection

Of the 11 indicators in the US set there were exact or near equivalents for three in the UK set. There appear to be major differences in approach of the two countries to treatment and management of upper respiratory tract infection (table 2). In particular, UK physicians appear less likely to believe that there is an evidence base for aggressive treatment of streptococcal disease.

Urinary tract infection

Of the 18 indicators in the US set there were exact or near equivalents for 10 in the UK set. In the US set there was generally more emphasis on pretreatment urine culture (required for a wider range of existing conditions) and on follow up investigation. The US panel also generally recommended longer courses of treatment (maximum 7 days for lower tract infection and minimum 10 days for upper tract infection v 5 and 7 days, respectively, in the UK). There was also a recommendation for a drug by the US panel (trimethoprim/sulfamethoxazole combination) which is regarded as obsolete in the UK (where trimethoprim alone is the recommended drug).

Further details are included in table 1 for allergic rhinitis, influenza immunisation, acne, low back pain, contraceptive treatment, and hormone replacement therapy. In these conditions there was generally close agreement between the UK and US set of indicators.

DISCUSSION

There were considerable benefits in using US indicators as a starting point for developing a set of quality indicators for the UK, despite the need to replicate the US development process in order to produce contextually valid indicators for the UK. Collaboration between the UK and the US research teams resulted in new insights for researchers from both countries into the different purposes of quality indicators and into the impact of cultural and organisational factors on quality indicators. Fifty six percent of indicators in the US set had exact or near equivalents in the UK set. These indicators could be used as a basis for comparing quality of care in the two countries although, as noted above, in the final set of UK indicators more stringent cut offs for validity and necessity to record were chosen than in the comparative analysis reported in this paper.

Although we have focused on the presence or absence of US indicators in the UK indicator set as a means of assessing the applicability of the former in a second country, there were also indicators which appeared in the UK set alone. Sometimes these were clearly due to differences in the panel process—for example, detailed indicators on the management of hypertension in patients with angina in the ischaemic heart disease set—and sometimes they were related to the different healthcare context—for example, requirement for registers of patients with angina diabetes and hypertension in the UK sets alone.

We have focused on differences in professional practice in the results reported here. However, there are a number of additional reasons for differences between the two sets of indicators which we have not analysed in detail. Firstly, the literature reviews were different, with the UK reviews being more comprehensive and focused on primary care evidence than the US reviews—for example, the UK reviews were on average 14.7 pages long and contained 69 references while the US reviews were on average 8.3 pages long and contained 30 references. In addition, there may have been differences which related to the selection of indicators for scoring by the panels, the composition of the panels, and the conduct of the panel meetings. Finally, the reproducibility of the panel process is not perfect, although the reliability of panels rating the same set of indicators is generally regarded as acceptable.^16,17 In our analysis we have principally concentrated on differences which related to differences in clinical practice in the two countries.

This study has significant implications for other developed countries that plan to use indicators to improve quality and manage performance. We believe that there is considerable scope for countries to collaborate in the development of quality indicators, particularly countries with similar health systems such as the UK and The Netherlands. Nevertheless, there will always be important contextual differences between countries which mean that indicators cannot be transferred from one country to another without going through a process of modification.