Discussion

We observed differences between self-reported MI or stroke and the final verified diagnosis obtained from the medical records or death certificates. Almost half of self-reported MIs (48%) and strokes (42%) were not able to be verified, and 43% of verified MIs and 10% of verified strokes were unreported, with a further 11% of verified strokes reported as TIA. Differences were also observed between events obtained solely from hospital discharge codes and final verified events. Ten per cent of MIs and 22% of strokes obtained from discharge codes were not verified, and 19% of verified MIs and 27% of verified strokes were identified from self-reports but not from discharge codes or death certificates. Relying solely on self-reports of MI or stroke, or solely on discharge codes in our clinical trial, would have led to significant numbers of participants being misclassified with regard to having experienced a significant adverse event. Despite the differences between self-reports, hospital discharge codes and verified events, neither the source of the event nor the level of adjudication substantially altered the relationship between treatment allocation and occurrence of either MI or stroke.

In most cases, the final diagnosis for the non-verified, self-reported MIs was related to a disorder of the heart, suggesting that miscommunication or misunderstanding between the participant and their physician led to the error. It has been suggested that the gastrointestinal side effects of calcium supplements might be misclassified as MI,7 but we found no evidence that this was the case. In at least 43% of non-verified cases, MI was specifically considered as a diagnosis and excluded. The most common final diagnosis for non-verified, self-reported strokes was TIA, suggesting that participants did not understand the distinction between these two conditions. This is not surprising as TIA is often described as a ‘mini-stroke’. In approximately one-third of non-verified, self-reported events (both MI and stroke), there were no details of any potentially related event in the medical records. All these events occurred in people who had experienced a previous verified event, suggesting that they reported the same event more than once, or they reported symptoms that were similar to their primary event for which they did not seek medical attention.

There are limited published data that specifically address differences between self-reported events and the medical records. A systematic review identified 15 studies that assessed the accuracy of questionnaires of medical history compared with the medical records.1 The proportion of illnesses reported in questionnaires ranged from 30% to 53% of those listed in the medical record. Conversely, the medical record listed 36–70% of illnesses reported in questionnaires. Reporting of surgical procedures or hospitalisation in questionnaires appeared to be more accurate than reporting of non-surgical illness. The findings of this systematic review have been replicated in more recent studies.2 ,8–10 Further studies have addressed the accuracy of self-reported cardiovascular events. For MI and stroke, approximately 70–80% of self-reported events were confirmed by hospital record review,2 ,8–13 which the authors suggested is sufficiently accurate for use in research studies.8 ,9 ,11 For subtyping of stroke, review of hospital records adds useful data to self-reports and discharge coding.14

A comprehensive analysis of reporting of cardiovascular events that occurred during the Women's Health Initiative clinical and observational studies has been published.10 In these studies, local physician adjudicators reviewed self-reported data and patient medical records for all self-reported cardiovascular events. The local adjudicators were able to verify 68% of self-reported MIs and 72% of strokes. The local adjudicators could only verify MI in 78% of events with a discharge code for MI, and stroke in 81% of events with a discharge code for stroke, suggesting coding data from hospital admissions may not be accurate, a finding consistent with the results of our study. Finally, there was no complete agreement between local and central adjudicators. Eighty-one per cent of locally adjudicated MIs were verified by central adjudicators, and conversely 86% of centrally adjudicated MIs were classified as MI by local adjudicators. The issue of unreported events was not addressed.

Our findings are broadly consistent with this previous research, although it is uncertain whether self-reports in response to open-ended questions in a clinical trial are comparable to responses to questionnaires or medical records based on a mixture of open-ended and closed-ended questions. The lower accuracy of MI self-reports in our study compared with other studies might be because the participants in our trial were older, had more comorbidities, and had more cognitive impairment than participants in previous studies. Another contributing factor might be that although cardiovascular events were prespecified secondary endpoints, many of the participants might not have been aware of this, and may have placed less value on reporting these events than fractures, which were the primary endpoints of the study. In the UK, accuracy of diagnostic coding has improved substantially in recent years,15 although no similar data are available for New Zealand. It is possible that the magnitude of differences between adjudicated events and hospital discharge coding might have changed since our study was undertaken.

Inaccurate self-reports or underreporting of events would not be expected to affect the results of a study unless the study treatment introduced a systematic bias that produced differential rates of inaccurate self-reports or underreporting of events. However, when the event numbers are small, as in many clinical trials, chance differences can substantially impact the study results. In our study, the adjudication process led to marked changes in the degree of statistical significance, even though the relative risks of MI with calcium supplements remained elevated at each stage. The relationship between calcium supplements and occurrence of MI or stroke was similar regardless of whether the events were based solely on hospital discharge coding or self-reports or final verified events, a finding that is supported by the similar increase in vascular risk associated with calcium supplements across randomised trials that used various means of event identification.16 ,17

A common approach in clinical trials is to adjudicate all significant adverse events centrally, although the need to do this remains uncertain.18 The advantages of central adjudication include systematically applying the definition of an event, reducing the possibility of differential misclassification of events, giving greater confidence in the validity of the study results, and by including suitable triggers, potentially identifying events that are missed by local investigators.18 However, central adjudication is unlikely to identify events that are not reported to the local investigators, adds significant cost and complexity to trials, and has not been shown to improve the ability to determine treatment effects in cardiovascular trials.18

In summary, our results suggest that when the accuracy of an event in clinical trials is critical, even for a relatively common and serious medical event such as MI or stroke, self-reports cannot be relied upon and should be independently verified from the medical records. However, even if independent verification occurs, our results also suggest that a substantial number of events will not have been reported. For adverse events of particular interest, additional steps, such as searches of hospital discharge databases, should be considered to attempt to identify unreported events. Relying solely on non-adjudicated hospital discharge coding will lead to similar inaccuracies as using self-reported data because some events will be missed and some events will be included that would not be verified by adjudication.