Most clinical trials evaluating the relative effectiveness of coronary revascularisation procedures have focused primarily on differences in mortality and morbidity. However, measures of morbidity are often poorly related to subjective accounts of health and well being¹ and do not capture all aspects of outcome that are important to patients. To improve the evaluation of treatments, generic measures of health related quality of life such as the short form 36 (SF-36)² are now increasingly being used alongside clinical measures.^3,4 Generic measures are useful for comparing diseases but measure the health status of the patient in general and do not address the condition under evaluation. Disease specific measures are more responsive in detecting treatment effects.⁵ However, few clinical trials have used disease specific measures to evaluate outcomes in different coronary revascularisation procedures.

Several coronary heart disease specific questionnaires have been developed, but the majority have not been validated against rigorous scientific standards (S Schroter, PhD thesis, London University, 2001). Psychometric methods provide well established scientific techniques for measuring subjective judgements on numerical scales and for evaluating the scientific rigour of measurement scales (that is, reliability, validity, and responsiveness). There are now rigorous criteria for evaluating the scientific robustness of patient reported health outcome measures.⁶ Although several disease specific measures have been developed to meet these criteria to evaluate outcomes in angina^7,8,9,10 and myocardial infarction,¹¹ no validated questionnaires have been developed to evaluate outcomes specific to coronary revascularisation. There is also substantial evidence that there are unique concerns specific to the experience of coronary revascularisation^12,13 such as adverse effects, physical and psychological recovery from the interventions, and satisfaction with the procedures that are not covered by existing coronary heart disease specific measures.

We describe the development and scientific validation of the coronary revascularisation outcome questionnaire (CROQ), a new patient based measure to evaluate health outcomes and health related quality of life after coronary revascularisation (coronary artery bypass grafting (CABG) and percutaneous transluminal coronary angioplasty (PTCA)).

METHODS

Questionnaire development

We used four sources of information to develop the content of the CROQ: firstly, the literature of health outcomes in coronary heart disease; secondly, existing patient based measures of outcome in coronary heart disease; thirdly, the expert opinions of key health care professionals involved in cardiac patient care (cardiac surgeons, cardiologists, cardiac specialist nurses, pain control nurses, and cardiac liaison nurses) about problems commonly reported by patients undergoing coronary revascularisation; and fourthly, qualitative in-depth interviews with 10 patients who had undergone CABG and 10 who had undergone PTCA to develop questionnaire items based on the words used by patients to describe their experience of coronary revascularisation.

On the basis of this information, we developed a conceptual model to guide the development of the preliminary versions of the CROQ that had four core content domains (symptoms, physical functioning, psychosocial functioning, and cognitive functioning) and two additional content domains (adverse effects and satisfaction) in the post-revascularisation versions. We then generated questionnaire items for all content domains through discussions with an expert group of methodologists with expertise in health outcome assessment and questionnaire design. We borrowed some items from existing questionnaires^2,7,11 and made all items specific to the patient’s heart condition.

We pre-tested preliminary versions of the CROQ by face to face interviews with 11 CABG and eight PTCA patients to evaluate content validity, clarity and appropriateness of wording, item sequence, questionnaire format, and instructions. Minor modifications were made to the pre-test questionnaires to produce field test versions of the CROQ.

Field testing

Patients

After we obtained approval from ethics committees, we recruited patients from the Royal Brompton & Harefield Trust Hospitals in London and the Wythenshawe Hospital in Manchester, UK. Patients were sent consent forms, information sheets, and questionnaires by post at two assessment points: before, and three months after coronary revascularisation. The assessment point of three months was selected because it is generally considered that by this time the majority of patients who have had CABG or PTCA will have recovered from the procedures and only a minority will still be experiencing adverse effects from the procedures.

Pre-revascularisation samples

All patients who expected to undergo isolated CABG or PTCA at the three hospitals during the study period were eligible to participate. Patients who were scheduled for elective surgery were recruited by postal survey to their home address after they were given a date for CABG or PTCA. Patients admitted to hospital by emergency were excluded from the pre-revascularisation sample.

Post-revascularisation samples

All patients who underwent isolated CABG or PTCA in the study period were sent the post-revascularisation version of the CROQ three months after revascularisation, even if they had not completed a baseline questionnaire (including patients who were given very short notice of their procedure and emergency cases). This was done to maximise the sample size for the psychometric analyses and to ensure that samples were representative of all patients undergoing CABG and PTCA. A subset of CABG and PTCA patients were sent post-revascularisation versions of the CROQ on two occasions within a two week interval to evaluate test-retest reliability.

Procedures

The psychometric evaluation of the CROQ was carried out in two independent field tests by postal survey. Standard techniques were used to ensure a high response rate, including personalised letters, standardised instructions, stamped addressed return envelopes, and follow up reminder letters.¹⁴ The purpose of the first field test was to produce a shorter version of the CROQ by eliminating items with poor measurement properties and to carry out a preliminary psychometric evaluation of item reduced versions of the questionnaires. The purpose of the second field test was to evaluate the psychometric properties of the shortened questionnaires in independent samples.

Methods for the two field tests and psychometric evaluations were identical, except that subsets of patients in the second field test were randomly assigned to receive a booklet containing only the CROQ, or the CROQ and SF-36,² or the CROQ and Seattle angina questionnaire (SAQ)⁷ to evaluate construct validity. Data on angina and dyspnoea severity, measured by the Canadian Cardiovascular Society (CCS) and New York Heart Association (NYHA) classifications, respectively, were obtained from hospital records for a subsample of patients before CABG.

Statistical analyses

Psychometric methods^6,15,16 were used to produce item reduced versions of the questionnaires: item-total correlations, item redundancy, missing data, maximum and aggregate adjacent endorsement frequencies, item responsiveness, and item test-retest reliability.¹⁷ The most robust items were retained. Preliminary scales were created on the basis of both the a priori conceptual model and empirical criteria (factor analysis and Cronbach’s α). Scale properties were evaluated to confirm that items in the same scale measured the same construct and that items in different scales measured different constructs.¹⁵ Items were eliminated until all pre-specified criteria were satisfied by an approach developed in our previous work.¹⁶ After confirming that the item reduced scales satisfied tests of scaling assumptions, we evaluated acceptability, reliability, validity, and responsiveness with psychometric tests and criteria (table 1).^15–21 Pre- and post-revascularisation versions of the CROQ were analysed separately with the use of CABG and PTCA samples.

RESULTS

First field test (item reduction)

In the first field test, 146 CABG and 128 PTCA patients completed the CROQ before revascularisation, and 289 CABG and 280 PTCA patients completed it three months after revascularisation (table 2).²²

Item reduction analyses produced shortened, final versions of the CROQ each taking about 10 minutes to complete (Appendices A, B, C, D: to view appendices go to http://www.heartjnl.com/supplemental). All four versions (CROQ-CABG_Pre, CROQ-PTCA_Pre, CROQ-CABG_Post, CROQ-PTCA_Post) contain 32 core evaluative items and one descriptive item that is not included in scale scores. The post-revascularisation versions of the CROQ (CROQ-CABG_Post 52 items, CROQ-PTCA_Post 47 items) contain these 33 core items plus additional evaluative items about adverse effects and satisfaction with outcome and two descriptive items.

Table 3 summarises items in the final versions. The CROQ is scored to produce six scale scores: symptoms (seven items), physical functioning (eight items), psychosocial functioning (14 items), cognitive functioning (three items), satisfaction (six items), and adverse effects (11 or six items). Items in each scale are summed and then transformed to a 0–100 scale by the same method as that used in the SF-36,² with 100 representing the best possible outcome. Preliminary evaluation of the psychometric properties of the item reduced CROQ in the first field test showed that all versions satisfied the tests and criteria described in table 1 (data not presented).

Second field test (evaluation of psychometric properties)

This section describes results from the final psychometric evaluation of all four versions. Because of space constraints, it focuses mainly on the three month post-revascularisation versions. The post-revascularisation versions of the CROQ can also be used in the longer term—for example, one year after revascularisation. A full psychometric validation of data at nine months after revascularisation showed that the measurement properties of the instrument are retained (data not presented).

Patients

In the second field test 696 of 916 CABG and 504 of 734 PTCA patients responded (table 2). The CABG sample consisted of 281 of 407 patients before revascularisation (mean (SD) age 63.6 (9.2) years, 85% men) and 415 of 509 after revascularisation (65.0 (8.9) years, 83% men; 76% response rate). The PTCA sample comprised 159 of 270 patients before revascularisation (mean (SD) age 60.6 (9.7) years, 75% men) and 345 of 464 after revascularisation (62.3 (10.2) years, 73% men; 69% response rate). Subsamples of 198 CABG and 107 PTCA patients completed pre- and post-revascularisation questionnaires (responsiveness samples), and 50 CABG and 48 PTCA patients formed the test-retest reliability samples.

Psychometric evaluation

Acceptability

All versions had good acceptability (table 4) with a low proportion of missing data. Although the CROQ had some ceiling effects, this was expected for post-revascularisation scores, as high scores reflect patients’ return to optimal functioning after an effective clinical intervention.

Reliability (internal consistency and test-retest)

Cronbach’s α coefficients¹⁸ for all scales exceeded the criterion of 0.70¹⁹ (table 4). Item-total correlations within scales were similar and exceeded the criterion of 0.20,⁶ indicating that each item was contributing equally to the scale. Intraclass correlation coefficients exceeded the criterion of 0.70²⁰ for all scales, indicating good test-retest reliability.

Tests of scaling assumptions

Item convergent and discriminant validity correlations¹⁵ confirmed the scale structure. The majority of items were scaling successes, a few were probable scaling successes, very few were probable scaling failures, and none were definite scaling failures.

Content validity

Content validity was evaluated during the development of the CROQ. Evidence from interviews and pretesting with patients, expert opinion, and a review of the literature supports the content validity of the CROQ.

Construct validity (within-scale analyses)

Evidence of high internal consistency supports the construct validity of the CROQ. High α coefficients (table 4) and moderately high item-total correlations for scales indicate that a single construct is being measured and that the items can be combined to form scales. Results of principal axis factor analysis and intercorrelations between CROQ scores provide further support for construct validity (data not presented).

Construct validity (analysis against external criteria)

Correlations between the CROQ and the SF-36 and SAQ scores provide further support for construct validity (table 5). Evidence for convergent validity is shown by moderate to high correlations between similar domains of the three measures and, for discriminant validity, low correlations between domains measuring different constructs on the CROQ, SF-36, and SAQ. For example, correlations between the CROQ and the SF-36 physical component summary score (PCS) were higher for symptoms and physical functioning than for psychosocial functioning and cognitive functioning. CROQ symptom and satisfaction scores were highly correlated with SAQ anginal frequency and satisfaction scores, respectively.

The ability of the CROQ to differentiate between known groups is supported by results from analyses of CROQ-CABG pre-revascularisation scores. CROQ-CABG_Pre symptoms scores showed the expected gradient according to CCS angina severity (mean (SD) score CCS class I 80.1 (19.4), n  =  3; CCS II 47.9 (22.2), n  =  35; CCS III 42.1 (22.5), n  =  53; CCS IV 34.0 (15.4), n  =  16; one way analysis of variance p  =  0.005) and NYHA dyspnoea severity (NYHA I 56.7 (21.2), n  =  17; NYHA II 40.6 (21.4), n  =  39; NYHA III 39.2 (22.1), n  =  52; NYHA IV 28.9 (22.3), n  =  16; one way analysis of variance p  =  0.033).

Responsiveness

Table 6 presents mean CROQ scores for subsamples assessed before and after revascularisation. There was significant change in all scales in the CABG and PTCA samples (p < 0.05). There were large effect sizes²¹ in three of the four scales (symptoms, physical functioning, and psychosocial functioning) in the CABG sample and one scale (symptoms) in the PTCA sample. Effect sizes were moderate for cognitive functioning in the CABG sample and for physical functioning and psychosocial functioning scales in the PTCA sample.

Although the intended use of the CROQ is to compare CABG and PTCA procedures, it is not appropriate to make these comparisons with the data presented in this paper. The data presented here were collected for psychometric testing purposes only and have not been adjusted for case mix severity, age, sex, social status, or ethnicity.

DISCUSSION

The CROQ is a practical and scientifically validated patient based measure of outcome for coronary revascularisation that is acceptable to patients and satisfies rigorous psychometric criteria for reliability, validity, and responsiveness. As the only validated instrument developed specifically for use before and after CABG and PTCA, and which is quick and easy to administer, the CROQ provides a rigorous method for improving the evaluation of outcomes in clinical trials and clinical audit. We achieved high response rates to our postal surveys, suggesting that self administration by post is a convenient and feasible method of collecting outcome data.

In the absence of a more appropriate instrument, the SAQ has been widely used in assessing outcomes after CABG and PTCA. The CROQ provides more appropriate content, as it contains items directly addressing the impact of these procedures based on problems that patients reported to be important. The SAQ was originally developed for use with patients given medical treatment and thus has a more limited focus on angina and satisfaction with treatment. The evaluation of construct validity (table 5) showed that the CROQ is on a par with the SAQ in measuring symptoms and satisfaction, but has the advantage of also measuring psychosocial functioning, cognitive functioning, and adverse effects, with little additional patient burden.

Although generic measures such as the SF-36 have been used to measure change in health status after revascularisation, it is widely acknowledged that disease specific measures are more responsive to treatment effects. Given its demonstrated high responsiveness, the CROQ is a promising new tool for use in clinical trials. It may detect important differences between procedures that have previously not been detected by less sensitive generic measures.

This paper reports the development and initial validation of the CROQ in a British sample. Validation of measures is an iterative process. Future research should be undertaken to confirm the psychometric properties of the CROQ and generalisability of findings to other English speaking patient populations, and population norms need to be generated. The CROQ has recently been used in two clinical trials.^23,24 Data from these trials will provide information about the relation between the CROQ and clinical variables and the ability of the CROQ to predict clinical outcomes. The availability of different language versions will enable international comparisons of patient based outcomes in clinical trials; an Italian version is being validated.²⁵

With increasing resource allocation to coronary revascularisation, it is essential to have scientifically robust tools with appropriate content to evaluate effectiveness. Our conclusions about the strong psychometric properties of the CROQ are based on the results of rigorous two stage field testing in large samples of patients. The CROQ exceeded criteria for all psychometric tests.