Article Text
PDF
Abstract
Objectives In Japan, an annual health check-up and health promotion guidance programme was established in 2008 in accordance with the Act on Assurance of Medical Care for the Elderly. A self-reported questionnaire on medication use is a required item in this programme and has been used widely, but its validity has not been assessed. The aim of this study was to evaluate the validity of this questionnaire by comparing self-reported usage to pharmacy insurance claims.
Setting This is a population-based validation study. Self-reported medication use for hypertension, diabetes and dyslipidaemia is the evaluated measurement. Data on pharmacy insurance claims are used as a reference standard.
Participants Participants were 54 712 beneficiaries of the National Health Insurance of Chiba City.
Primary and secondary outcome measures Sensitivity, specificity and κ statistics of the self-reported medication-use questionnaire for predicting actual prescriptions during 1 month (that of the check-up) and 3 months (that of the check-up and the previous 2 months) were calculated.
Results Sensitivity and specificity scores of questionnaire data for predicting insurance claims covering 3 months were, respectively, 92.4% (95% CI 91.9 to 92.8) and 86.4% (95% CI 86.0 to 86.7) for hypertension, 82.6% (95% CI 81.1 to 84.0) and 98.5% (95% CI 98.4 to 98.6) for diabetes, and 86.2% (95% CI 85.5 to 86.8) and 91.0% (95% CI 90.8 to 91.3) for dyslipidaemia. Corresponding κ statistics were 70.9% (95% CI 70.1 to 71.7), 77.1% (95% CI 76.2 to 77.9) and 69.8% (95% CI 68.9 to 70.6). The specificity was significantly higher for questionnaire data covering 3 months compared with data covering 1 month for all 3 conditions.
Conclusions Self-reported questionnaire data on medication use had sufficiently high validity for further analyses. Item responses showed close agreement with actual prescriptions, particularly those covering 3 months.
- MEDICAL HISTORY
- EPIDEMIOLOGY
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
We evaluated, for the first time, the validity of a self-reported questionnaire on medication use in the annual health check-up system of Japan by comparing self-reported usage to a log of pharmacy insurance claims, a record that is free of recall bias and regarded as a ‘gold standard’.
A large population-based sample was used.
Specificity might be underestimated due to incomplete data on pharmacy insurance claims.
Introduction
Self-reported questionnaires have been employed to determine drug usage in many epidemiological studies.1–3 However, the accuracy of the information obtained by such questionnaires is limited by recall bias.4–13 A substantial amount of inaccurate data could result in ‘misclassification bias’, leading to incorrect estimates of disease risk and/or prevalence.14 To date, a few studies have evaluated the validity of self-reported medication use but the results have been inconsistent, with some finding high validity4 ,10 ,13 and others finding relatively low validity.5 ,9 ,11 This inconsistency could result from differences in data collection method, type of drug, age and/or nationality of the target population, and healthcare system.
In Japan, an annual health check-up and health promotion guidance programme was started in April 2008 in accordance with the Act on Assurance of Medical Care for the Elderly by the Ministry of Health, Labour and Welfare (MHLW).15 Medical insurers are obliged to provide this programme to all their beneficiaries aged 40–74 years. During the period from 2010 to 2014, a total of around 112 million people used the programme. The programme mainly targets individuals with metabolic syndrome. A self-reported questionnaire on medication use for hypertension, hyperglycaemia and hypercholesterolaemia, is one of the required items, and the collected data are used to identify individuals in need of further guidance. When a recipient of a health check-up answers ‘yes’ to the question on medication use, he or she is automatically excluded from the target population for the health guidance programme. Consequently, misclassification of medication use can lead to recipients with metabolic syndrome losing the opportunity to receive appropriate guidance. In addition, the questionnaire has been used to detect untreated individuals so that they can be advised to see a doctor when their laboratory data strongly indicate hypertension, diabetes and/or dyslipidaemia. If all health insurance claims were to be computerised and integrated with health check-up data, a self-reported questionnaire would no longer be necessary for public health researchers. In fact, the Japan National Database (NDB) project led by the MHLW was started for this purpose. At the moment, however, the linkage rate between health insurance claims and health check-up data in the NDB is very low,16 meaning that researchers must use self-reported questionnaire data as an alternative. Thus, validation of the data is crucial for practical as well as for research applications.
The aim of this study was to evaluate the validity of the self-reported questionnaire on the use of drugs for hypertension, diabetes and dyslipidaemia that is conducted as part of the annual health check-up in Japan's health guidance programme. To do this, we compared self-reported usage to a log of pharmacy insurance claims, a record that is free of recall bias and regarded as a ‘gold standard’.
Methods
Participants
The participants of this study were beneficiaries of National Health Insurance (NHI) of Chiba City. Japan has a universal healthcare insurance system that covers all citizens.17 There are two types of coverage for individuals younger than 75 years of age, Employees’ Health Insurance and NHI; the latter is managed by municipalities and covers the self-employed, farmers, retirees and the unemployed. The participants in this study consisted of 54 760 beneficiaries aged 40−74 years who received a health check-up from 1 May 2012 to 28 February 2013. Of these individuals, 48 with missing data were excluded, for a final total of 54 712 beneficiaries (22 242 men and 32 470 women). Health check-up data and pharmacy insurance claims data were integrated for comparison using the values of household number, birth month and sex.
Ethics statement
Consent was not obtained from participants because this study was performed using only existing data. To ensure anonymity, personal identifiers (eg, name, address and telephone number) were removed from the records, date of birth was changed to the first of the month, and personal number and household number administered by Chiba City Hall were converted to random numbers prior to release. The study was conducted in accordance with the Declaration of Helsinki.
Definition of self-reported medication users
The self-reported questionnaire, which is required in the health check-up, includes the following item.
Are you currently taking the following medications?
Medication for hypertension (yes or no).
Insulin injection or oral medication for hyperglycaemia (yes or no).
Medication for hypercholesterolaemia (yes or no).
A participant who answers ‘yes’ is defined as a self-reported user of drugs to treat hypertension, diabetes and/or dyslipidaemia.
Definition of true medication users
‘True medication users’ were determined by pharmacy claims submitted from April 2012 to March 2013. Unfortunately, pharmacy insurance claims provided by Chiba City for this study include only prescriptions dispensed outside hospitals. Nonetheless, pharmacy insurance claims are available for 71.4% of all prescriptions filled in Chiba Prefecture during fiscal year 2012.18 Although the pharmacy claim data in this study were not perfect, they were the best data currently available for determining medication users in our study. Thus, we used the obtained pharmacy claim data as a tentative ‘gold standard’. Generic names of medications for the three conditions are listed in online supplementary files 1–3. For detecting appropriate drugs, the Database of Drugs in Japan was used.19 Codes of the Anatomical Therapeutic Chemical Classification System (ATC codes) provided by the WHO20 have not been assigned to every drug used in Japan, but we list as many as possible in the online supplementary files.
Initially, we used two different definitions for the true medication users detected by pharmacy insurance claims: one for participants prescribed during the same month as the health check-up (1 month); and the other for participants prescribed during the same month as the check-up or in the previous 2 months (3 months). In Japan, the law limiting prescriptions to 2 weeks was repealed in April 2002 to allow long-term prescriptions (with some exceptions). Thus, even if the participants did not receive a prescribed medication during a survey month, they might have already received one during the previous month. To overcome this possible omission error, we decided to analyse the month of the check-up and the month of the check-up plus the previous 2 months, separately.
Equivalent household income
Individual annual income from 1 January to 31 December 2011 was obtained from tax records at Chiba City Hall. Number of people per household was obtained by counting the persons with the same household number. People per household included persons insured by NHI in Chiba City and other householders regardless of whether they were beneficiaries. Household income was calculated by summing the incomes of all household members, as aforementioned. An equivalent household income was calculated as household income divided by the square of the number of household members.
Statistical analysis
The proportions of the individual medication users as determined by the self-reported questionnaire and by pharmacy insurance claims were compared by McNemar's test. For assessing the validity of self-reported medication use for hypertension, diabetes and dyslipidaemia, medication use as detected by pharmacy insurance claims was assumed to be accurate (as the gold standard). Sensitivity was calculated as the proportion of participants with self-reported medication use among the participants with pharmacy claims, and specificity was calculated as the proportion of participants without self-reported medication use among the participants without pharmacy claims. In addition, κ statistics were also calculated for each medical condition. The κ statistic measure of agreement is scaled to be 0 when the agreement is what would be expected by chance and to 1 when there is perfect agreement. Landis and Koch21 defined values of 0.00−0.20 as slight, 0.21−0.40 as fair, 0.41−0.60 as moderate, 0.61−0.80 as substantial and 0.81−1.00 as almost perfect agreement. All comparisons were planned and all tests were two tailed. A p value <0.05 was considered statistically significant. All statistical analyses were performed using the STATA13 software package (Stata Corp, College Station, Texas, USA).
Results
Participant demographics, clinical characteristics and medication use, as determined by the self-reported questionnaire and insurance claims, are shown in table 1. Means and SDs of age and body mass index were 65.5±7.8 years and 22.9±3.3 kg/m2, respectively. Median equivalent income was 1 170 000 yen (as of 30 June 2015, US$1 was equivalent to 122.72 yen). The proportions of participants prescribed medications for hypertension, diabetes and dyslipidaemia, as indicated by insurance claims during the check-up month (20.3% for hypertension, 4.0% for diabetes and 14.3% for dyslipidaemia) and within 3 months (25.0% for hypertension, 5.0% for diabetes and 17.9% for dyslipidaemia) were all significantly lower than as indicated from self-reports (33.4% for hypertension, 5.6% for diabetes and 22.8% for dyslipidaemia).
Demographic and clinical characteristics, including medication use as determined by the self-reported questionnaire and insurance claims
Table 2 presents the results of the validity analysis of self-reported medication use in all participants. In general, the self-reported questionnaire predicted actual prescriptions (ie, according to pharmacy insurance claims) with high sensitivity and specificity for the month of check-up and for 3 months. Specificity was uniformly higher for predicting prescriptions within 3 months for all three drug classes. The κ values were also higher for predicting prescriptions within 3 months compared with 1 month. Thus, the self-reported questionnaire more accurately represented medication use for 3 months than for 1 month, the only exception being sensitivity for hypertension medication use (p=0.001). Only those results for predicting actual prescriptions over 3 months were used for subgroup analyses (table 3).
Validity of self-reported medication use
Validity of self-reported medication, among subgroups, for predicting actual prescriptions during 3 months
Analyses of subgroups divided by sex, age range and income are shown in table 3. In all subgroups, sensitivity and specificity were >80% and κ statistic >60%. Thus, the validity of the self-report questionnaire was high regardless of sex, age and income.
Discussion
The self-reported medication-use questionnaire for the annual health check-up programme overseen by the Japanese MHLW was found to have high validity for predicting actual prescriptions for drugs used to treat hypertension, diabetes and dyslipidaemia.
Although pharmacy insurance claims data inherently include comprehensive prescription information, making the data suitable as a ‘gold standard’, only prescriptions dispensed outside hospitals were available in this study, which accounted for 71.4% of all prescriptions in fiscal year 2012.18 Indeed, the proportions of participants with actual pharmacy claims were significantly lower than the proportions based on self-reports for all three conditions. Accordingly, we should consider how this drawback influences the accuracy of the sensitivity and specificity values calculated. For calculation of sensitivity, we determined the proportion of participants self-reporting use only among those with external (outside of hospital) prescriptions. However, if the responses to the self-reported questionnaire by participants with external prescriptions and those with in-hospital prescriptions are assumed to be the same, the sensitivity should be close to the true figure. On the other hand, for calculation of specificity, we used the number of participants without actual prescription claims as the denominator, which includes those actually prescribed in-hospital, making the value lower than the true data. Despite this influence, however, values of specificity and sensitivity were satisfactory (>80%) for all three diseases.
To date, only two studies, to our knowledge, have conducted a validity assessment of self-reported medication use for hypertension, diabetes and/or dyslipidaemia. One assessed self-reported drug use for hypertension and diabetes in 17 191 participants of different ethnicities in British Columbia, Canada.10 This study found high specificity for hypertension (99−100% in each ethnicity) and diabetes (99−100% in each ethnicity), but relatively low sensitivity for hypertension (60−76% in each ethnicity). Here, the two-step methodology may have influenced the result. In the first step, only those participants who answered ‘yes’ to ‘Do you have this condition?’ were extracted. Then, the selected participants were asked, ‘In the past 12 months, have you taken any medicine for this condition?’. It is known that this two-step method increases specificity but decreases sensitivity.14 In contrast, the self-reported questionnaire item for medication use analysed in the present study asks participants a single question for each condition (‘Are you currently taking the following medications?’), and high validity was observed, indicating that this mode of questioning is appropriate and reliable.
The second study, from Washington State, reported high validity for self-reported medication use for hypertension and dyslipidaemia (statins) in 403 participants of a population-based, case–control study of breast cancer in women aged 65−79 years.4 Sensitivity in cases and controls were 92% and 92% for hypertensive medication, and 83% and 98% for statins, respectively. Specificity was 91% and 93% for hypertensive medication, and 98% and 98% for statins.4 This result is quite similar to ours.
In contrast to medication for hypertension, diabetes and dyslipidaemia, these studies reported lower sensitivity of self-reported medication use for asthma (32−52% in each ethnicity10) and depression (64% and 66% in cases and controls4), suggesting that the validity of self-reported medication use depends on the specific medical condition. So et al10 suggested that the difference in self-report validity among medication types is likely related to the frequency of use. For example, self-report validity for medications used to treat acute symptoms, such as asthma, tends to be lower than that for medications taken routinely for chronic conditions such as hypertension, diabetes and dyslipidaemia. Further studies are needed to confirm whether this is also the case for the health check-up self-report questionnaire.
We found that validity of the self-reported questionnaire was higher (more accurately reflected medication use) during 3 months than during 1 month. Patients in Japan can now obtain longer term prescriptions (after the law limiting prescriptions to 2 weeks was repealed). Thus, we assumed that many patients currently taking medications for hypertension, diabetes and/or dyslipidaemia would have prescriptions filled in previous months before the annual check-up and simultaneous completion of the self-reported questionnaire. In general, prescriptions for chronic diseases are renewed, at most, every 3 months,22 so there should be relatively few respondents, with no recent claims over this period, answering ‘yes’. Even so, to control for the influence of longer term prescriptions, we evaluated the validity for predicting actual prescriptions over 10 months in 20 529 participants who received the health check-up from 1 December 2012 to 28 February 2013 (data not shown). The κ (95% CI) values for hypertension, diabetes and dyslipidaemia medication use were 72.2% (70.9% to 73.6%), 78.4% (77.0% to 79.7%) and 70.5% (69.2% to 71.9%), respectively; similar to or even slightly higher than the values for 3 months. Thus, we suggest that the self-reported questionnaire is better for predicting medium-term and long-term medication use than for predicting short-term use.
We found high validity of the self-reported questionnaire in this study, indicating that healthcare workers and public health researchers can both use these data for practical and research purposes. For example, these data can be used in research toward the NDB's goal of replacing self-reported data once integration of data between annual health check-ups and insurance claims becomes reliable.
Conclusion
We found that the self-reported questionnaire of medication use for hypertension, diabetes and dyslipidaemia that is conducted as part of the annual health check-up in Japan's health guidance programme is a valid measure of true medication use. Accuracy appears better for predicting prescriptions filled within 3 months compared with those filled within 1 month.
Acknowledgments
The authors greatly appreciate the invaluable help and support from the staff of Chiba City Hall: Shinji Tada, Atsushi Nakamura, Yukiko Ishikawa, Isamu Nagata and Shinsuke Mizuma, who provided all data analysed in this study.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
- Data supplement 1 - Online supplement
- Data supplement 2 - Online supplement
- Data supplement 3 - Online supplement
Footnotes
Contributors MF was responsible for the study conception, design, analysis, interpretation and drafting of the manuscript. MF and AH acquired the data. AH, YS, KN and ST assisted with study conception and design. YS, KN and ST assisted with statistical analysis. All the authors contributed to critical revisions of the manuscript and approved the final manuscript.
Funding This work was supported by JSPS KAKENHI grant number 26460826 and the Chiba Foundation for Health Promotion & Disease Prevention.
Competing interests None declared.
Ethics approval The Research Ethics Committee of the Graduate School of Medicine, ChibaUniversity, approved this study (number 1724).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.