Objectives Self-reported approaches that assess opioid usage can be subject to social desirability and recall biases that may underestimate actual pill consumption. Our objective was to determine the accuracy of patient self-reported opioid consumption using a 14-day daily paper or electronic diary.
Design Prospective cohort study.
Setting Multicentre study conducted in four Québec (Canada) emergency departments (ED): three university-affiliated centres, two of them Level I trauma centres and one urban community hospital.
Participants ED patients aged ≥18 years with acute pain (≤2 weeks) who were discharged with an opioid prescription. Patients completed a 14-day daily diary (paper or electronic) assessing the quantity of opioids consumed. On diary completion, a random sample from the main cohort was selected for a follow-up visit to the hospital or a virtual video visit where they had to show and count the remaining pills. Patients were blinded to the main objective of the follow-up visit.
Outcomes Quantity of opioid pills consumed during the 2-week follow-up period self-reported in the 14-day diary (paper or electronic) and calculated from remaining pills counted during the follow-up visit. Intraclass correlation coefficient (ICC) and Bland-Altman plots were used to assess accuracy.
Results A total of 166 participants completed the 14-day diary as well as the in-person or virtual visit; 49.4% were women and median age was 47 years (IQR=21). The self-reported consumed quantity of opioid in the 14-day diary and the one calculated from counting remaining opioid pills during the follow-up visit were very similar (ICC=0.992; 95% CI: 0.989 to 0.994). The mean difference between both measures from Bland-Altman analysis was almost zero (0.048 pills; 95% CI: −3.77 to 3.87).
Conclusion Self-reported prescription opioid use in a 14-day diary is an accurate assessment of the quantity of opioids consumed in ED discharged patients.
Trial registration number NCT03953534.
- PAIN MANAGEMENT
- WOUND MANAGEMENT
- ACCIDENT & EMERGENCY MEDICINE
Data availability statement
Data are available upon reasonable request. Original data set found in the manuscript is available upon request to the corresponding author.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
Strengths and limitations of this study
Multicentre prospective study including emergency department with different trauma levels.
Comparison of paper and electronic diary to assess the quantity of opioid consumed.
Use of a better gold standard of opioid consumption validation: counting remaining opioid pills instead of pharmacy insurance claim.
Relatively high percentage of the patients refused to participate with possible limitation in population representativeness.
Our samples were opioid naïve, without chronic pain and possibly without opioid use disorder, resulting in a possible selection bias.
Opioids are frequently prescribed for the management of moderate to severe acute pain after emergency department (ED) discharge.1 2 Previous studies evaluating the quantity of opioids consumed during an acute pain follow-up for ED discharged patients,3 4 or after surgery5–8 reported that the majority of the prescribed opioids were actually not consumed. These results are often used to determine how much opioid to prescribe for pain management; if more pills than needed are prescribed, there are leftovers available for misuse, while too few pills can compromise pain relief. Furthermore, self-report of opioids consumed is used to predict long-term use and opioid use disorder.9 However, most studies used self-report surveys, interviews or questionnaires to assess opioid usage and these can be subject to social desirability and recall biases that may possibly underestimate the real quantity consumed.10 11 Others have used diaries for self-reporting of their medication usage, which contrary to interviews or surveys, offer the advantage of being recorded closer to events to minimise memory bias.12
Self-reported medication use was strongly associated with pharmacy insurance claims,13–16 while others found lower reliability.17–19 These inconsistencies could result from differences in data collection method, type of drug used, age and/or nationality of the target population, healthcare system, and in gold standard used for validation.13 Self-reported information on current medication use, collected via mailed medication inventory questionnaire, and compared with pharmacy records, was almost in perfect agreement for the use of statins, β-blockers and calcium channel blockers.20 Although another similar study reported significant variation between medication classes; antidepressants and antihypertensives showed very good agreement, but mood stabilisers demonstrated only moderate-poor agreement.21 However, none of these studies reported opioid use agreement.
Few studies have estimated the accuracy of self-reported opioid use. One study on prescribed analgesic medication use (including opioids) by chronic pain patients showed good accuracy of self-report compared with an administrative prescription claims database.22 However, a study performed in hospitalised patients and healthy individuals using urine rapid drug screening showed that both groups under-reported their opioid use when asked by trained interviewers.23
Self-reported illicit drug use, including opioids, could also be biased by social desirability. However, studies have generally shown that self-report of substance use to be valid relative to urine drug testing.24–28 Furthermore, several reviews support the reliability and validity of self-reporting risky behaviours when privacy is assured, assessments are self-administered, and computerised.29–31 In addition, we could hypothesise that patients who received a legitimate opioid prescription from ED physicians for acute pain relief should be less inclined to under-report their opioid use since it is not a socially disapproved behaviour.12
Most studies seeking to validate self-report medications consumption have used pharmacy insurance claim (or similar administrative databases) as a gold standard, which are not as reliable as counting leftover pills. This study’s objective was to determine the accuracy of the quantity of consumed prescription opioids self-reported by patients in a 14-day diary compared with the quantity calculated from counting the number of remaining opioid pills in the prescription bottle through an in-person or virtual video visit.
Materials and methods
Patient and public involvement
This research originated from the rinsing death toll from opioids overdose. A patient partner was involved in the design (including the development of the paper and electronic version of the pain medication diaries) and conduct of the study.
Study design and setting
This was a planned substudy of a large Canadian multicentre prospective cohort study (detailed methods for this study not included) conducted in four Québec (Canada) EDs (three university-affiliated centres, two of them Level I trauma centres and one urban community hospitals). The objective of the main cohort was to evaluate the quantity of opioids consumed for acute pain complaints after ED discharge. We randomly selected a sample from the main cohort to determine the accuracy of the self-reported opioid consumption.
Patients aged 18 years and older, treated in the EDs from November 2019 to November 2021 with an acute painful condition of a duration of less than 2 weeks and who were discharged from the ED with an opioid prescription, were identified by ED physicians 24/7 and recruited by research assistants. We excluded patients who did not speak French or English, reported using opioid medications prior to the ED visit, or suffering from cancer or chronic pain. The recruitment was interrupted from March 2020 to August 2020 due to the COVID-19 pandemic. After the completion of their 14-day analgesic consumption diary, patients from one of the study sites were randomly invited to participate to an in-person follow-up visit. However, after we changed the follow-up format from in-person to virtual video visit due to COVID-19 public health regulations, consecutive patients from all sites were invited to participate. Patients received a $50 compensation for their participation in the study.
ED physicians asked for patients’ verbal consent to be contacted by research assistants. The research assistants explained the study and subsequently obtained informed consent in-person or over the phone. Patient demographic information, pain intensity at triage, arrival mode to the ED, triage priority, length of stay in the ED and opioids received during their ED visit were extracted from the local electronic medical records. Emergency physicians completed the participants’ initial information with the final diagnosis, pain intensity at discharge and prescribed pain medications. Pain intensity was evaluated with a verbal 11-point Numerical Rating Scale (NRS) ranging from 0 to 10, where 0 represents ‘no pain at all’ and 10 represents ‘the worst imaginable pain’. Patients were instructed to complete a 14-day paper diary in which they recorded the quantity, dosage and name of all pain medication consumed daily (online supplemental figure S1): ‘List all the pain medications you are taking, whether or not they were prescribed by a doctor. You must enter the name of the medication, the number of milligrams and the total number of pills taken during the day. The number of milligrams (mg) is written on your medication bottle.’ The 2-week follow-up period was chosen because acute pain usually lasts only for a short time and most patients stop taking opioids during that period (88% in the pilot study).32 Using pre-addressed and pre-stamped envelopes, these diaries were mailed back after completion (or brought back for cases of in-person follow-up). Patients also had the choice of completing an identical online version of the diary (online supplemental figure S2).
After completion of their 14-day analgesic use diary, participants were interviewed in-person at the hospital or during a virtual video visit to answer questions concerning their pain medication use during the same period. Patients were unaware that the live interview would focus on counting remaining opioid pills. In both follow-up formats, patients were asked whether they filled another opioid prescription than the one initially prescribed, and if that initial prescription was entirely consumed (so it could be added to the consumed quantity). When the interviews were conducted after day 14, patients were asked how many pills they consumed after diary completion and this number was added to the consumed quantity of the 14-day diary. For in-person follow-up visit, patients were asked to bring all opioid prescription bottles they received during the 2-week period and research assistants counted the remaining pills. For the virtual video visit, patients were asked to show the content of their opioid prescription bottles, and to count remaining pills under constant supervision. Study data were collected and managed using REDCap (Research Electronic Data Capture), a secure, web-based application tool hosted on the hospital server.33
The primary outcome was the quantity of opioid pills consumed during the 2-week follow-up period self-reported in the 14-day diary (paper or electronic) and calculated from remaining pills counted during the follow-up visit.
Data analysis and statistics
Comparison of baseline characteristics between patients included and those who refused to participate in the validation study was compared using Cohen’s effect sizes.34 Small, medium and large effect sizes for χ2 and Mann-Whitney U tests are <0.3, ≥0.3 and <0.5, ≥0.5, respectively. The agreement between the quantity of opioid consumed during the 2-week follow-up obtained from the diary and quantity calculated from follow-up visits was assessed using intraclass correlation coefficient (ICC) and Bland-Altman plot.35 The ICC estimates and their 95% CI were calculated using absolute-agreement and two-way mixed-effects model.36 Mean difference and ±1.96 SD of bias between both measures were reported from the Bland-Altman analysis. As secondary analysis, the same agreement statistics were performed separately for paper and electronic diary to assess the accuracy of both types of diaries. Alpha level was set at 0.05, and all statistics were performed using SPSS V.26 (IBM, Somers, New York, USA).
The required sample size to determine the accuracy of self-reported opioid use, estimating an ICC of 0.80 (which is considered good concordance) between information from the diary and from the follow-up visits (in-person or virtual video), a total of 162 patients was necessary to achieve a power of 0.90, with an alpha of 0.05. Sample size calculations were performed using PASS V.11.0 (Power Analysis & Sample Size software, NCSS, Kaysville, Utah, USA).
Study cohort description
A total of 292 patients meeting the initial study inclusion/exclusion criteria and who completed the 2-week diary were contacted to participate in the validation substudy. Of these, 43% refused to participate, leaving 166 patients who accepted the follow-up visit, 31 in-person and 135 in virtual video (56 electronic and 110 paper diary; figure 1). Included patients and those who refused to participate were similar in regards to baseline characteristics; between group effect sizes were below 0.3, which is considered small according to Cohen’s definition (table 1).34 Median age for included patients was 47 (IQR=21) years, 49.4% were women, and the median NRS pain intensity at triage was 8, decreasing to 5 at ED discharge. They were prescribed a median of 15 (IQR=10) opioid pills at ED discharge. All patients filled their opioid prescription and consumed a median of 6 opioid pills (IQR=12) from the diary and a median of 7 opioid pills (IRQ=13) calculated from the follow-up visit. Agreement analyses were similar for in-person compared with those with virtual video visit (ICC >0.98 for both groups); therefore, we combined both results together.
The quantity of opioid pills consumed during the 2-week follow-up period as self-reported in the 14-day diary and the one calculated from follow-up visit were very similar (ICC=0.992; 95% CI: 0.989 to 0.994). The mean difference between both measures from Bland-Altman analysis was almost 0 (0.048 pills), indicating no under or over estimation from the diary (65% of the patients had identical values between both measures). Furthermore, the variability of the difference was low as 95% of the patients had a margin of error within 4 opioid pills (95% CI: −3.77 to 3.87; figure 2).
Agreements for paper and electronic diary separately
Results from ICC analysis were also high for both the paper (ICC=0.985; 95% CI: 0.979 to 0.990) and electronic (ICC=0.996; 95% CI: 0.993 to 0.997) version of the diary when compared with the follow-up visit. Mean difference between both measures from Bland-Altman analysis was low for the paper version of the diary (−0.082; 95% CI: −3.97 to 3.80; figure 3 top panel) and low for the electronic version (0.30; 95% CI: −3.36 to 3.96; figure 3 lower panel). Again, the Bland-Altman plots showed that both types of diaries provide no indication of under or overestimation, compared with follow-up data as well as low variability in the difference (both showing that 95% of the patients had a margin of error within 4 opioid pills, figure 3).
This prospective study demonstrated that a self-report pain medication diary is an accurate measure of the quantity of consumed opioids during a 14-day follow-up in ED discharge patients when compared with the quantity calculated from counting the number of remaining opioid pills in the prescription bottle during a follow-up visit. We also established that both types of diaries (paper or electronic) demonstrated similarly high rates of agreement when compared with follow-up visits.
The ICC observed in our results was especially high and parameters (mean difference and deviation of the mean difference) from Bland-Altman analysis were very low, indicating a very strong agreement between diary information and follow-up data. Other studies have also found high agreement between self-report prescribed medication and pharmacy records; as the sensitivity, specificity and positive predictive values for self-reported use of statins, β-blockers and calcium channel blockers were all 95% or greater.20 Hafferty et al also reported high agreement for prescribed antidepressants and antihypertensives self-report use.21 Our agreement results were higher than those observed by Lacasse et al between current self-report opioid use and prescription claims database.22 However, their analyses were the results of two databases linkage performed on a chronic pain population (our cohort was exclusively composed of acute pain patients) and in studies using prescription claims or administrative databases as gold standard, it is impossible to know how much opioids were consumed or if the patients even used them at all. In the present study, 14 patients (8%) filled their prescription but did not consume any opioid pills. Rashidian et al also observed lower agreement between non-prescribed opioid self-report and urine rapid drug screening with a sensitivity of 77% and 69% among hospitalised patients and healthy individuals, respectively.23 However, self-report of opioid use that are not prescribed by a clinician may be subject to greater social desirability bias.
Since several reviews demonstrated the reliability and validity of self-reporting risky behaviours when privacy is assured and computerised,29–31 we anticipated differences in agreement between both diary formats (electronic or paper). This was not what we observed; the ICC and Bland-Altman parameters were similar for both types of measures. It appears that self-reporting legitimate opioid prescription consumption associated with the confidentiality of a study is less influenced by social desirability bias.
Furthermore, the mean difference bias from the Bland-Altman analysis was almost null. This can also suggest that patients receiving a legitimate opioid prescription from ED physicians for an acute pain were not inclined to over or under-report their opioid use since it is probably not perceived as a socially disapproved behaviour.12
Another factor that explain the strong agreement between our two measures may result from the fact that daily diary use is less affected by memory bias contrary to self-report surveys, interviews or questionnaires. Diary information offers the advantage of being recorded closer to events to minimise memory bias.12 It also allows the quantification of the daily opioid consumption over a period, the study of the temporal trend of opioid use, whether co-analgesics were used, the identification of the time when opioids were no longer necessary and the relationship between opioid intake and pain levels. However, diaries are sometimes not adequately completed, require more work and require more motivation, have a lesser return rate than other instruments, and therefore may involve selective bias that limits generalisation.12 Simple diaries with clear explanations and good examples, offered in a paper and electronic version may help decrease these selection biases.
This study has some limitations. A relatively high percentage of the patients (43%) refused to participate in the validation sub-study, limiting the representativeness of the ED population. However, baseline characteristics (table 1) between patients who accepted and those who refused to participate in the substudy were similar. The small number of opioid pills consumed in the present study (median of 6 from diary and 7 from follow-up visits) may contribute to the high ICC and the small mean difference observed in the Bland-Altman results. Nevertheless, the variability in the mean difference did not seem to increase significantly with the number of pills consumed. However, the very high agreement observed in our study could results from a selection bias; our patients were opioid naïve, without chronic pain, and possibly without opioid use disorder (since we exclude those who consumed opioids before the ED visit). Therefore, our results may only be generalisable to this population.
In summary, self-reported prescription opioid use in a 14-day diary (paper or electronic) is an accurate assessment of the opioid quantity consumed compared with counting remaining pills during a follow-up visit. This modality is an acceptable method to measure consumed opioids after an ED visit.
Data availability statement
Data are available upon reasonable request. Original data set found in the manuscript is available upon request to the corresponding author.
Patient consent for publication
This study involves human participants and was approved by Comité d’éthique de la recherche du CIUSSS du Nord-de-l’Île-de-Montréal (ID: MP-32-2019-1684). Participants gave informed consent to participate in the study before taking part.
This work was supported by a Grant from the Canadian Institutes of Health Research Fund (CIHR). We would also like to thank Martin Marquis and Gloria De Buyl for English language editing.
Contributors RD, AC and JM conceived the study and obtained research funding. RD, JP, DW, JJP, MI, VC, JM and AC contributed to the final protocol and data interpretation. JP was responsible for data management and statistical analysis. RD drafted the manuscript. RD, JP, DW, JJP, MI, VC, JM and AC contributed substantially to its revision. RD, JP, DW, JJP, MI, VC, JM and AC approved the final manuscript as submitted and have agreed to be accountable for all aspects of the work. RD accepts full responsibility for the work and/or the conduct of the study, had access to the data, and controlled the decision to publish.
Funding This study was supported by a Grant from the Canadian Institutes of Health Research Fund (CIHR, no 159808).
Competing interests None declared.
Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.