Article Text
PDF
Abstract
Background Shared decision-making (SDM) has become a policy priority, yet its implementation is not routinely assessed. To address this gap we tested the delivery of CollaboRATE, a 3-item patient reported experience measure of SDM, via multiple survey modes.
Objective To assess CollaboRATE response rates and respondent characteristics across different modes of administration, impact of mode and patient characteristics on SDM performance and cost of administration per response in a real-world primary care practice.
Design Observational study design, with repeated assessment of SDM performance using CollaboRATE in a primary care clinic over 15 months of data collection. Different modes of administration were introduced sequentially including paper, patient portal, interactive voice response (IVR) call, text message and tablet computer.
Participants Consecutive patients ≥18 years, or parents/guardians of patients <18 years, visiting participating primary care clinicians.
Main measures CollaboRATE assesses three core SDM tasks: (1) explanation about health issues, (2) elicitation of patient preferences and (3) integration of patient preferences into decisions. Responses to each item range from 0 (no effort was made) to 9 (every effort was made). CollaboRATE scores are calculated as the proportion of participants who report a score of nine on each of the three CollaboRATE questions.
Key results Scores were sensitive to mode effects: the paper mode had the highest average score (81%) and IVR had the lowest (61%). However, relative clinician performance rankings were stable across the different data collection modes used. Tablet computers administered by research staff had the highest response rate (41%), although this approach was costly. Clinic staff giving paper surveys to patients as they left the clinic had the lowest response rate (12%).
Conclusions CollaboRATE can be introduced using multiple modes of survey delivery while producing consistent clinician rankings. This may allow routine assessment and benchmarking of clinician and clinic SDM performance.
- PRIMARY CARE
- shared decision-making
- patient-reported experience measure
- patient-reported measurement
- mode effects
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
- PRIMARY CARE
- shared decision-making
- patient-reported experience measure
- patient-reported measurement
- mode effects
Strengths and limitations of this study
This study deployed a range of survey administration modes within 24 hours of clinic visits.
By consistently measuring performance of the same clinicians, we were able to assess effects of survey administration mode on clinician rank order and CollaboRATE score.
We accounted for patient characteristics in our analysis by including patient demographic and clinical data.
Data collection for 15 months led to respondent and organisational fatigue, impacting response rates.
Cost analysis relied on retrospective cost estimates.
Introduction
Assessing whether patients have experienced shared decision-making (SDM) in clinical settings has become a priority for multiple stakeholders.1 In the US context, the Consumer Assessment of Healthcare Providers and Systems Clinician and Group (CG-CAHPS) survey is a widely used patient reported experience measure (PREM).2 However, the measurement of SDM is tangential to the overall survey goals.3 In addition, recall bias may undermine the validity of results given that patients are asked to assess experiences over the preceding 6 months.4 Furthermore, response rates are significantly below the 40% target set by the Agency for Healthcare Research and Quality. Alternative PREMs of SDM are designed primarily for research and not clinical use.5–7
To address this gap we developed CollaboRATE, a 3-item PREM of SDM.8 ,9 CollaboRATE assesses three core SDM tasks during a clinic visit: (1) information and explanation about health issues, (2) elicitation of patient preferences and (3) integration of patient preferences into decisions.
CollaboRATE has demonstrated psychometric soundness,9 and has also been used in a demonstration project to assess differences in SDM across 34 clinical teams in England.10 In addition, Blue Shield of California is using CollaboRATE to assess SDM as part of a preauthorisation process for joint replacement.11 A research study at a Veterans Administration site found positive correlations between CollaboRATE scores and patient assessments of satisfaction and communication.12
CollaboRATE's brevity minimises respondent burden and facilitates varied data collection modes, including email, interactive voice response (IVR) automated telephone calls, short message service (SMS) text messages, electronic kiosk, tablet computer and paper. Tourangeau identified five administration-mode factors related to survey response: (1) respondent contact method (eg, in person or email); (2) survey medium (eg, paper or electronic); (3) administration method (interviewer or self-completed); (4) sensory input (eg, visual or oral) and (5) response mode (eg, handwritten, keyboard, voice).13 Data analysis needs to account for mode effects and differences in response rates, especially if results are intended as measures of performance. However, to date, there has been limited research on differences in validity, reliability and administration costs by mode in patient-reported measurement.14
Mixed modes of CG-CAHPS survey administration, including options to respond over the phone, have yielded response rates near 40%.15 These mixed-mode response rates are higher than those achieved by mail-back surveys alone15 while collecting comparable data.16 In a systematic review of survey response rates, email delivery was associated with response rates of 15%,17 similar to the 14% response rate for CG-CAHPS delivered by email.16 SMS administration has associated response rates ranging from 49% to 98%,18–21 and data quality comparable to that collected by person-led phone interviews or IVR.21
The cost of PREM administration has significant implications, yet receives little attention.16 ,22 ,23 Achieving a reasonable response rate, at a realistic cost, is a key requirement if PREMs are to become sustainable sources of clinical performance data.
This project was designed to assess feasibility and identify challenges to the routine implementation and real-time delivery of CollaboRATE. In this paper, we aim to evaluate CollaboRATE as a measure of clinician performance with regard to response rates, CollaboRATE SDM scores at the level of individual clinicians, respondent characteristics across modes, impact of mode and patient characteristics on SDM scores, and cost of data collection per patient response. To do so, we conducted a single-site demonstration project where we sequentially introduced different modes of patient-reported data collection including paper, patient portal, IVR calls, SMS and electronic tablet in the clinic. We hypothesise variation in CollaboRATE scores by clinician as well as lower CollaboRATE scores in data collection modes completed outside the clinic setting.
Methods
Participants
Setting
The study was conducted with two clinical teams, consisting of 15 clinicians, in a primary care clinic of a rural academic medical centre in New Hampshire, with 16 000 registered patients.
Inclusion criteria
Consecutive patients 18 years or older visiting the participating clinical teams Monday through Friday beginning in April 2014 were eligible to complete CollaboRATE. Parents or guardians of patients under age 18 were eligible to complete CollaboRATE on behalf of their children.
Measures
CollaboRATE
CollaboRATE consists of three items (box 1).8 ,9
CollaboRATE items
Thinking about the appointment you have just had:
How much effort was made to help you understand your health issues?
How much effort was made to listen to the things that matter most to you about your health issues?
How much effort was made to include what matters most to you in choosing what to do next?
Responses to each item can range from 0 (No effort was made) to 9 (Every effort was made) for a maximum total of 27.
Responses to CollaboRATE were scored in a binary fashion; participants who responded to all questions with a ‘9’ were considered to have experienced SDM and all others were not. Participants with one or more missing responses on CollaboRATE were excluded from analyses. We calculated the proportion of participants who reported a score of 9 on each of the three CollaboRATE questions for each clinician.9 ,24 This scoring technique is a strategy for aiding interpretation while avoiding potential ceiling effects common among clinician performance assessments completed by patients.9 Incomplete CollaboRATE surveys were not included in analysis.
Data collection
Modes of CollaboRATE survey administration
We adopted five modes of CollaboRATE administration. In each mode, eligible patients were given opportunities to complete the survey within 24 hours of their clinic visits. Modes were implemented sequentially beginning in April 2014, each for a 3-month period (table 1). Clinicians were made aware of data collection at the start of the study. At the end of the study, we provided participating clinicians with a summary of the results, their individual scores and a brief presentation on how to achieve SDM in routine clinical settings.
Modes of CollaboRATE administration in chronological order
Participant characteristics
For the first four modes, CollaboRATE responses were linked to patient demographic and clinical data, including patient age, gender, number of health conditions, visit type (annual wellness or other), marital status and socioeconomic status (SES). SES was estimated by the percentage of people living in the patient's zip code with income beneath the federal poverty level using data from the American Community Survey.25 ,26 These data were extracted from patients' medical records, anonymised by medical centre staff and shared with the research team. Data linkage was not undertaken for the tablet mode; clinician, patient age and gender were self-reported in that mode.
Data analysis
Respondent characteristics and response rates
Response rates were based on the number of patients attending the clinic during the data collection periods. Not all patients were registered with a patient portal or had a cellular telephone number in their record. We therefore calculated adjusted response rates for patient portal, IVR and SMS modes, including only patients who could have received CollaboRATE. We compared the demographic characteristics of respondents to non-respondents for each mode. Pearson's χ2 tests and Student's t-tests were used for categorical and continuous variables, respectively. Logistic regression analysis was used to confirm the descriptive findings comparing respondents to non-respondents and to examine whether an interaction between age and number of comorbidities predicted response, where response was the binary outcome variable and independent variables included age, number of comorbidities, age multiplied by number of comorbidities, gender, age, and whether the visit was for a wellness check-up. We also calculated the median time to achieve 25 completed CollaboRATE surveys per clinician per mode, which has been used as a minimum for obtaining stable score estimates.24 Studies which have calculated the number of patient responses needed to obtain reliable information have found that around 25 completed questionnaires are needed.27
Clinician SDM performance by mode
To assess clinician performance across modes, we estimated a logistic regression model including clinician indicators and all available patient characteristics as covariates with the CollaboRATE top score indicator as the dependent variable. We used the model to estimate the CollaboRATE top score (accounting for patient characteristics) for each clinician in each mode and used these estimates to rank clinicians' performance and assessed concordance of the estimated rankings across modes. Clinicians who did not reach 25 patient responses in any of the four modes were excluded from the results.24 ,27
Effects of administration mode and patient demographics on CollaboRATE scores
To evaluate the effects of survey administration mode and patient characteristics on CollaboRATE scores, we conducted mixed effects logistic regression analysis with CollaboRATE score as the dependent variable. Fixed effects included administration mode and patient characteristics. Clinicians were included as a random effect to account for clustering of patients by clinician, allowing inferences to be generalisable beyond the clinicians in our study. The unit of analysis was the CollaboRATE score per individual visit. To address potential clustering of CollaboRATE responses by patients who had more than one visit, we also conducted the mixed effects logistic regression analysis described above including only the first CollaboRATE response from each patient.
Cost analysis by survey administration mode
The costs associated with each administration mode were estimated to calculate a cost per fully completed survey. Data included development costs (programming, testing and planning) and field costs (piloting, supplies and labels, technical and vendor support, management and staff time). We estimated the cost per completed survey by dividing the total cost for each mode by the number of completed surveys. Estimated costs per mode do not include costs associated with an institutional electronic medical record, patient portal or survey collection platform but do include variable costs such as technical support and charges associated with sending SMS messages and IVR calls. To estimate the likely long-term cost per completed survey, we repeated the analysis excluding research and set-up costs. Analysis was conducted in Stata V.13 and SPSS V.21. An α level of ≤0.05 was considered statistically significant.
Results
There were 4421 patients who completed the CollaboRATE survey over the 15-month study period (April 2014–October 2015), resulting in an overall CollaboRATE top score of 68% for the clinic. Only 19 incomplete CollaboRATE responses were recorded. Among eligible patients, the average response rate across modes was 25%, with the highest response rate in the combined tablet and mail mode (41%) followed by the patient portal mode (34%). The administration mode had significant effects on CollaboRATE scores, with the highest score recorded during the paper survey administration (81%), followed by the patient portal (71%), with the lowest overall CollaboRATE score recorded in the IVR survey administration mode (61%) (table 2).
Response rate and CollaboRATE scores across modes
The fastest median time to achieve 25 completions per clinician was accomplished in the tablet/mail-in mode (17 days), followed by paper in-clinic (25 days), patient portal (25 days), IVR (31 days) and SMS (40 days).
Clinician rank
Figure 1 demonstrates that clinician rank order by proportion of CollaboRATE top scores was identical across the four administration modes (adjusted by patient case mix; see online supplementary appendices 1 and 2), though the average top score differed across modes.
Clinician scores by mode, adjusted for patient characteristics^*. ^While 15 clinicians participated in all four data collection modes, only eight reached 25 patient responses in all modes; therefore, eight of 15 clinicians are shown here. *During the electronic tablet/postal mail phase, responses were not linked to the electronic medical record; as a result, patient demographic data were unavailable.
supplementary appendices
Respondent characteristics by administration mode
Respondents versus non-respondents by mode
Demographic characteristics of all patients attending the clinic were similar across modes (table 3). Respondents were slightly older than non-respondents across all modes and represented the overall clinic population with regard to gender. Respondents were slightly more likely than non-respondents to be seen for an annual wellness visit in the patient portal and SMS modes. Respondents also had more comorbid conditions than non-respondents in the paper mode. Logistic regression analysis confirmed these descriptive findings and showed no significant interaction between age and number of comorbid health conditions in predicting response to the CollaboRATE survey (OR 1.00; 95% CI 0.99 to 1.00; full results available on request).
Respondent and non-respondent characteristics by mode
Respondent characteristics by mode
Among respondents, gender (χ2=11.32, p=0.02), age (χ2=49.88, p<0.001), reason for visit (χ2=11.04, p=0.01) and number of comorbidities (χ2=288.39, p<0.001) showed statistically significant variation across the modes. Respondents to the SMS and IVR modes were on average 8–10 years younger than respondents to the paper and patient portal modes. There were differences in respondent health status between modes; in the paper mode ∼46% of respondents had two or more comorbidities compared with 21% in SMS and IVR modes and 7% in the patient portal mode.
Effect of administration mode and patient demographics on scores
The results of a mixed effects logistic regression demonstrate significantly lower CollaboRATE scores in the patient portal (OR 0.60, 95% CI 0.45 to 0.80), IVR (OR 0.45, 95% CI 0.34 to 0.59) and SMS (OR 0.51, 95% CI 0.38 to 0.67) modes when compared with the paper mode. CollaboRATE scores increased slightly with patient age (OR 1.01 per year of age, 95% CI 1.01 to 1.02) but no other patient characteristics were associated with CollaboRATE scores (full results in online supplementary appendix 3). The estimated SD for the clinician random effect was 0.34, implying there was substantial unexplained heterogeneity in the clinicians' ratings even after adjusting for observed differences between the patient cases. Our sensitivity analysis including only the first CollaboRATE response from each unique patient finds very similar results (see online supplementary appendix 4). While the annual visit predictor becomes significant (OR 1.65, 95% CI 1.15 to 2.37) in this analysis limited to the first CollaboRATE response from each patient during the 15-month study period, all other fixed effects retain similar magnitude and significance and the estimated SD for the clinician random effect remains consistent at 0.33.
Cost of data collection
Owing to high personnel involvement in administering the paper and tablet modes, these modes were the most expensive (see online supplementary appendix 5). The cost per unique response was US$20.56 and US$16.71 for the paper and tablet modes respectively. The SMS, IVR and patient portal modes were significantly less expensive at US$10.28, US$8.87 and US$6.39 per unique response, respectively. When estimating the variable cost per completed response (excluding set-up costs and research related costs), these decrease to US$16.38 and US$13.46 for the personnel-intensive modes (tablet and paper modes) and US$3.36, US$2.83 and US$2.20 for the SMS, IVR and patient portal modes.
Discussion
Principal findings
Clinician SDM performance rankings were stable across data collection modes, though they were sensitive to mode effects, as were response rates. We were able to achieve the highest response rate (41%) by using tablet computers administered by research staff in-clinic, although this approach was among the costliest. The lowest response rate involved clinic staff giving paper surveys to patients as they left the clinic (12%).
Mode of administration and respondent characteristics were associated with CollaboRATE scores, with the lowest score in the IVR mode and the highest in the paper mode. We speculate that this finding may be explained, at least in part, by patients' perceived distance from the clinic experience and the perceived anonymity of the response. Across all modes, respondents were slightly older than non-respondents. Younger patients appeared to prefer, or were more comfortable than older patients, completing CollaboRATE using SMS and IVR, where respondents were ∼10 years younger than respondents from other modes. Across all modes, the clinician identity and mode of survey administration had the greatest impact on CollaboRATE scores, while increased patient age was associated with a small but statistically significant increase in CollaboRATE score.
If fixed institutional costs are excluded, the use of existing technological systems such as patient portals or SMS capabilities can reduce the cost of obtaining completed survey responses. The variable costs associated with collecting 25 responses per clinician are US$55 for patient portal, US$71 for IVR, US$84 for SMS, US$337 for paper in-clinic and US$410 for tablet, indicating that the modes requiring the least personnel involvement that maintained reasonable response rates (SMS, IVR, patient portal) cost the least. While the tablet mode had high response rates and was the most time-efficient data collection method, the personnel time required made it the most expensive mode.
Strengths and limitations
We consider the work to have several strengths. First, we deployed a range of modes to deliver a PREM of SDM, daily and within 24 hours of the clinic visit. By consistently measuring the performance of the same clinicians, we were also able to assess mode effects on clinician rank order and CollaboRATE score, and demonstrate the feasibility of this approach to performance measurement. We collected CollaboRATE routinely in a clinic setting and experienced the related practical challenges: competing priorities for administrative, information systems, and clinical staff made survey implementation challenging. We faced significant challenges embedding these survey administration methods in institutional information systems, where programming SMS, patient portal and IVR methods were by necessity secondary to other institutional deadlines and demands. However, we were able to access and link patient demographic and clinical data with CollaboRATE responses, which allowed us to account for patient characteristics.
The study also had limitations. Data collection for 15 months, using sequential administration modes, led to respondent and organisational fatigue, impacting response rates. Clinic staff were burdened by paper survey tasks. We received reports that patients who had previously completed the CollaboRATE survey perceived little value in repeating their evaluation. Our study design also did not account for potential repeated measures; since CollaboRATE's reference period is the patient's most recent clinical encounter, we assumed each CollaboRATE response to be independent and did not account for clustering by patient. Our sensitivity analysis excluding all but a patient's initial CollaboRATE response found results consistent with our original conclusions, suggesting only minimal potential influence of patient-level clustering on our results. Order of mode implementation was not randomised, and we were unable to account for potential order effects in our analysis.
In addition, significant changes in clinic management impeded paper administration of CollaboRATE. SMS and IVR administration modes were hampered because the clinic record system did not consistently distinguish between cellular and other telephone numbers. Selection bias, enabled by low response rates in some data collection modes and the potential for unmeasured demographic differences between respondents and non-respondents, may contribute to variation in scores between modes. However, we believe the risk of selection bias is low given the similar measured demographic characteristics of respondents and non-respondents. Our cost analysis methods also relied on retrospective cost estimates.
Context in existing literature
CollaboRATE provided a consistent estimate of the level of SDM practiced by clinicians, with no difference in rank order of clinicians across modes despite a shift in CollaboRATE top score. These findings are similar to those from the review by Hood et al14 where mode was not related to differences in the precision of 36-Item Short Form Health Survey (SF-36) scores, but survey administration (self-report or interviewer-led) and sensory stimuli (auditory, visual or both) were the factors that impacted the absolute mean scores. Using the Tourangeau13 framework, it appears that collecting data in the clinic setting results in the highest scores, compared with the lower scores obtained using the IVR or SMS modes. Lower scores when using IVR are also found by Rodriguez et al23 who reported primary care patients had significantly lower scores on the Ambulatory Care Experience Survey by IVR, after adjusting for patient characteristics, compared with mail and internet administration. In addition, our findings match others who report lower evaluations when patients are asked to respond outside clinic settings.28–30
Only the in-clinic electronic tablet/mail-in mode achieved the 40% response rate recommended for CAHPS.31 The patient portal response rate of eligible email recipients was 34%, double the average response rate found in a recent review,17 which supports the view that technological methods yield improved response rates, provided respondent burden is avoided.14 The use of SMS was not as successful; unreliable cellular coverage was a factor, and it remains unclear whether this mode can provide representative data given demographic differences in the use of cellphones and text messages.
Our cost analyses correspond with previous reports that the highest cost per completed survey is associated with in-clinic data collection.16 ,22 ,23 However, when set-up and research costs were excluded, all modes of administration were below US$20 per completed response and were <US$5 per completed response for three modes, namely patient portal, SMS and IVR.
Implications
Patient experience measurement is a relatively recent addition to healthcare settings and is widely viewed as a key method of assessing value.32 Nevertheless, there is a significant risk that the measurement of patient experience becomes viewed as a burden, where surveys are lengthy and administered out of context. Findings from our study suggest that CollaboRATE can overcome these challenges as it consists of only three items that can be delivered via multiple modes at a cost comparable to existing survey administration. In addition, clinician ranking was consistent, regardless of data collection mode. The decision of which mode is best for data collection will depend on the extent to which data collectors can harness technological tools and access patient information such as email addresses or telephone numbers to facilitate contact while maximising patient confidentiality. If multiple modes are used, it is important for administrators to account for mode effects when estimating CollaboRATE scores.
We now face the challenge of studying CollaboRATE in more settings to test the generalisability of our findings and establish benchmarks for clinic-level and clinician-level performance. There is also a need to investigate automated collection, analysis and visualisation of the data so that the results can facilitate the core purpose of providing timely feedback to improve clinical practice. Clinicians and clinics who score low on CollaboRATE will want to know what they can do to achieve SDM and in turn provide to patients higher quality healthcare.
Acknowledgments
We acknowledge the following individuals for their invaluable contributions to the project: Erik Olson and Shelley Sanyal for programming data collection and reporting systems for IVR/SMS and online patient portal data collection modes, respectively; Margaret Menkov for data management and medical record data extraction; Jef Hale, Scott Farr and Charles Goff for IT oversight; Chelsea Worthen, Ethan Berke and Michelle L'Heureux for clinic leadership.
References
Footnotes
▸ Additional material is available. To view please visit the journal online (http://dx.doi.org/10.1136/bmjopen-2016-014681).
Twitter Follow Paul Barr @BarrPaulJ
Contributors PJB contributed to design of the work, drafting the article and final approval of the version to be published. RCF contributed to design of the work, data analysis and interpretation, drafting the article and final approval of the version to be published. RT contributed to design of the work, critical revision of the article and final approval of the version to be published. EMO contributed to design of the work, data collection, data analysis and interpretation, critical revision of the article and final approval of the version to be published. RA contributed to the design of the work and final approval of the version to be published. MGC contributed to design of the work, data collection, critical revision of the article and final approval of the version to be published. AJO contributed to design of the work, data interpretation, critical revision of the article and final approval of the version to be published. GE contributed to conception and design of the work, drafting the article and final approval of the version to be published.
Funding This work was funded by the Gordon and Betty Moore Foundation, grant number 3929.
Competing interests GE reports personal fees from Emmi Solutions LLC, personal fees from National Quality Forum, personal fees from Washington State Health Department, personal fees from Shared Decision Making 3rd edition, personal fees from Groups (Radcliffe Press), outside the submitted work; and GE has initiated and led the Option Grid patient decision aids Collaborative, which produces and publishes patient knowledge tools in the form of comparison tables (http://optiongrid.org/). GE has been a member of teams that have developed measures of shared decision-making and care integration. These tools and measures are published and are available for use. For further information see http://www.glynelwyn.com/.
Ethics approval The study received ethics approval by Dartmouth Committee for the Protection of Human Subjects (#24529).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.