Original Article
Health care utilization: measurement using primary care records and patient recall both showed bias

https://doi.org/10.1016/j.jclinepi.2005.12.008Get rights and content

Abstract

Objective

To assess reasons for discrepancies between primary care consultation measured from patient self-report and that based on medical records.

Methods

Retrospective comparison of recalled consultation in previous 12 months among 2,414 subjects aged 50+ who reported knee pain in a population survey vs. primary care medical records. Record review included (1) all knee morbidity codes and (2) knee problems mentioned in consultation text. It was then extended to: (3) more than 12 months before survey, and (4) consultations for leg or widespread problems (e.g., generalized osteoarthritis).

Results

In those who reported knee pain, recalled consultation prevalence for knee problems “in past year” was 33% compared with 15% based on medical records. Forty percent of those with a recalled consultation had a recorded knee problem in the same time period (kappa = 0.43). Expanding record search to include leg and widespread problems, and knee problems up to 40 months prior to survey, increased “verified” self-reported consulters to 80%.

Conclusions

Disparity in estimates of consultation prevalence arose from inaccuracy of: (1) recall in survey responders and (2) recording by general practitioners of specific problems and repeat consultations. Perceived importance of problem in a multiproblem contact and whether it leads to an outcome (e.g., prescription) may influence recording. Implications exist for service provision projections and research.

Introduction

Epidemiologic studies (including general population surveys) often ask subjects to self-report health problems and recall related consultations to primary care. In the UK, this self-reported data can be used for the purposes of population-based needs assessment by Primary Care Trusts who plan and commission health services for their local population. Unmet needs in the community may be identified if there are people with health problems who are not seeking health care, and self-reported data can also be used for costing health care related to specific conditions. An alternative source of health care utilization data is the primary care medical record.

Many studies have found variable levels of agreement on diagnoses between self-reported questionnaire data and medical records [1], [2], [3], [4], [5], [6], [7], [8], [9], [10]. Fewer studies have investigated the agreement between recalled consultation for specific problems and the record of that consultation in the paper or computer records of the patient's general practice [11], [12], [13], [14], [15], [16]. In general, these latter studies have found disagreement. However, none have established whether the discrepancies are predominantly a problem of patient recall or of underrecording in the practice. Identifying the reasons for disagreement and estimating the bias in using either source to obtain the prevalence of consultation will assist researchers and policy makers in planning and interpreting research and in the provision of health care services.

We have therefore assessed (1) the agreement between recalled and recorded general practice consultation using knee pain as an example, and (2) reasons for discrepancies between self-report and practice records by investigating whether such discrepancies disappear if the medical record search is expanded in terms of the time period and the diagnoses covered.

Section snippets

Methods

Six thousand seven hundred ninety-two (adjusted response 77%) subjects returned a completed questionnaire from a general population survey of all adults (n = 8,995) aged 50 and over registered at three general practices in North Staffordshire, UK [17]. Three thousand twenty-three responders reported knee pain and, of these, 2,414 (80%) consented to medical record review and responded to the question on consulting a general practitioner (GP) for their knee pain. These 2,414 respondents form the

Part 1—All subjects

For steps 1–3, only knee disorder contacts recorded in the general practice medical records as being to the GP surgery, a home visit, or by telephone were considered as verifying the recalled consultation. This meant entries of hospital letters, for example, were excluded. After each step, the percentage of those with a recalled consultation who had a recorded consultation identified in the search for that step was determined. This was repeated for those who said they had not consulted. To show

Part 2—Subjects without a recorded consultation in Part 1

All subjects who recalled a knee pain consultation but had no record of a knee disorder consultation in steps 1–3 were selected. An equal number of those said they had not consulted (equally divided across practices) and had no record of a knee disorder in steps 1–3 were randomly selected. All searches for steps 4–7 were performed by KJ and CJ, who were blinded as to whether the subject had recalled a consultation or not.

Part 1: Steps 1–3

Two hundred fourteen (27%) of the 807 responders with a recalled consultation for knee pain had a knee-related Read Code in their medical records for the 12 months before the survey (Table 1). Agreement on consultation between the medical records for this 12-month period and self-report was slight based on Shrout's [20] definitions (κ = 0.30; 95% CI 0.27, 0.34). By extending the search to knee problems mentioned in the text, the number of responders who reported a consultation who also had a

Discussion

Measures of health service utilization may be derived from self-reported population data or from general practice medical records. We have compared the prevalence of self-reported consultation to that recorded in general practice and determined that substantial discrepancy exists. Only 40% of self-reported consulters over 1 year could be verified by examining medical records for the identical time period for knee problems. In contrast, the rate of incorrect reporting of nonconsultation in the

Acknowledgments

We are very grateful to all the survey responders, the North Staffordshire and GP Research Network, and administration staff at Primary Care Sciences Research Centre, Keele University. This study was funded by an NHS Executive (West Midlands) New Blood Research Training Fellowship and the Haywood Rheumatism Research and Development Foundation.

References (30)

  • R. Gross et al.

    The validity of self-reports on chronic disease: characteristics of underreporters and implications for the planning of services

    Public Health Rev

    (1996)
  • M.K. Walker et al.

    Validation of patient recall of doctor-diagnosed heart attack and stroke: a postal questionnaire and record review comparison

    Am J Epidemiol

    (1998)
  • M. Ockander et al.

    Patient–doctor concordance in elderly women's self-reported health and medical records

    Methods Inf Med

    (2002)
  • C.F. Simpson et al.

    Agreement between self-report of disease diagnoses and medical record validation in disabled older women: factors that modify agreement

    J Am Geriatr Soc

    (2004)
  • P.D. Cleary et al.

    The validity of self-reported physician utilization measures

    Med Care

    (1984)
  • Cited by (44)

    • Best Evidence Osteoarthritis Care: What Are the Recommendations and What Is Needed to Improve Practice?

      2022, Clinics in Geriatric Medicine
      Citation Excerpt :

      For example, QIs target different joints (eg, hip vs knee OA), whereas others target the type or frequency of treatment,57 or are for use by different health professions (eg, GP or physiotherapist58). QI assessments can be undertaken through an audit of medical records, although QI results based on medical records should be treated with caution as consumer perceptions of their treatment outcome may be different from that of their health care practitioner.59 Therefore, consumer-reported QI sets developed from OA clinical practice guidelines are considered a better option to effectively monitor and evaluate care than medical records.60

    • Situations and mechanisms of non-contact knee injury in adult netball: A systematic review

      2021, Physical Therapy in Sport
      Citation Excerpt :

      The majority of studies were large scale epidemiological studies designed to investigate the most common injuries in netball (Hopper & Elliott, 1993; Hopper et al., 1995a, 1995b) where identification of the mechanism of injury was a secondary goal and was self-reported. The self-report method risks incomplete or inadequate data because it relies on participants’ memory, or the memory of others, both of which may be inaccurate (Althubaiti, 2016; Gabbe, Finch, Bennell, & Wajswelner, 2003; Jordan, Jinks, & Croft, 2006). It is also important to consider whether there was any bias from the clinicians gathering the data and if they asked leading questions to the participants who may have been unsure how they sustained the injury (Althubaiti, 2016).

    • Simplified risk prediction indices do not accurately predict 30-day death or readmission after discharge following colorectal surgery

      2019, Surgery (United States)
      Citation Excerpt :

      Using the number of visits to an emergency room in the 6 months before the index admission would be difficult to assess based on patient recall without the aid of an EMR or large administrative data sets like the one used here. To address this limitation, we converted this original LACE variable to a binary variable for any number of emergency room visits to simplify assessment and decrease error by recall bias.27,28 The impact of this work can be far-reaching.

    • Methodological challenges collecting parent phone-call healthcare utilization data

      2016, Applied Nursing Research
      Citation Excerpt :

      The Internet is increasingly being used as a means of patient self-report, with good effect (Bhandari & Wagner, 2006). Because systematic errors can occur in both self-report data and provider documentation, a dual source of data collection is likely to yield more robust and accurate data than a single source (Jordan, Jinks, & Croft, 2006; Killeen et al., 2004). Analysis of parent call data to pediatric providers has been shown to be an important indicator of healthcare utilization, as it can identify potential cost savings not otherwise considered in healthcare utilization studies (Jordan et al., 2006; Leibowitz, Day, & Dunt, 2003).

    View all citing articles on Scopus
    View full text