Article Text
PDF
Abstract
Objectives This research aimed to understand the barriers and facilitators clinicians face in using a digital clinical decision support tool—UpToDate—around the globe.
Design We used a mixed-methods cohort study design that enrolled 1681 clinicians (physicians, surgeons or physician assistants) who applied for free access to UpToDate through our established donation programme during a 9-week study enrolment period. Eligibility included working outside of the USA for a limited-resource public or non-profit health facility, serving vulnerable populations, having at least intermittent internet access, completing the application in English; and not being otherwise able to afford the subscription.
Intervention After consenting to study participation, clinicians received a 1-year subscription to UpToDate. They completed a series of surveys over the year, and we collected clickstream data tracking their use of the tool.
Primary and secondary outcome measures (1) The variation in use by demographic; (2) the prevalence of barriers and facilitators of use; and (3) the relationship between barriers, facilitators and use.
Results Of 1681 study enrollees, 69% were men and 71% were between 25 and 35 years old, with the plurality practicing general medicine and the majority in sub-Saharan Africa or Southeast Asia. Of the 11 barriers we assessed, fitting the tool into the workflow was a statistically significant barrier, making clinicians 50% less likely to use it. Of the 10 facilitators we assessed, a supportive professional context and utility were significant drivers of use.
Conclusions There are several clear barriers and facilitators to promoting the use of digital clinical decision support tools in practice. We recommend tools like UpToDate be implemented with complementary services. These include generating a supportive professional context, helping clinicians realise the tools’ use and working with health systems to better integrate digital, clinical decision support tools into workflows.
- information technology
- health & safety
- change management
- international health services
- quality in health care
- medical education & training
Data availability statement
Data are available upon reasonable request. Data may be obtained from a third party and are not publicly available. Data were used under a data use agreement for the current study.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- information technology
- health & safety
- change management
- international health services
- quality in health care
- medical education & training
Strengths and limitations of this study
This study combines surveys with clickstream data from the digital tool—data that clinicians directly generate in using the tool—which provides precise and robust data.
We only included clinicians who applied and met criteria for a donated subscription; this included those who were able to complete the application in English and those working in limited-resource settings, limiting generalisability.
Due to time and resource constraints, we could not measure all components of the logic model we built relating the use of UpToDate to patient outcomes.
While we hypothesised that access to UpToDate can improve clinicians’ sense of self-efficacy, the psychometrics of the self-efficacy scale we instituted did not function properly in this study and resulted in null results.
Background
Diagnostic and treatment errors account for a significant amount of harm across high-income, middle-income and low-income settings and represent a serious public health problem. Most people will likely experience a diagnostic error in their lifetime.1 In a high-income country in an outpatient setting, one study found that 5% of the adults experienced diagnostic errors each year. Over half of these errors had the potential for severe harm. The researchers suggested that their findings were likely an underestimate and that the rate of diagnostic errors in low-income countries may be much higher. Other studies analysing mortality data from autopsies have shown that 10–15% of the deaths are due to missed diagnoses.2 Even in cases that are ultimately correctly diagnosed and treated, errors leading to delay may result in poor quality of care, patient safety risks, increased costs and, in some cases, malpractice litigation.3
Diagnostic and treatment errors can happen at any point in the care process, including initial assessment, performing and interpreting diagnostic tests, determining treatment, follow-up visits and tracking. These errors involve the failure to provide a timely, accurate determination of a patient’s health condition or treatment option and/or to communicate necessary, accurate, timely information to a patient.4 They represent missed opportunities to provide quality, effective care based on the best available clinical evidence.
More than half of the cases of diagnostic error are due to cognitive errors. Frontline healthcare workers face a demanding cognitive load. They need to keep up with new evidence and incorporate it into care decisions; more than 950 000 new publications are indexed in MEDLINE every year.5 The COVID-19 pandemic has further increased the speed and volume of clinical evidence, exacerbating the challenges.6
Health information technology or digital tools used at the point of care—clinical decision support tools—can reduce diagnostic errors. In 2019, the WHO acknowledged digital tools as important levers for ensuring effective, high-quality, equitable care.7 They can support clinicians’ decision-making, enabling quick, better informed decisions that lead to better health outcomes.8 Such tools include computerised alerts or reminders; clinical guidelines; focused patient data reports and analyses and contextually-relevant reference information, among other offerings.8
Clinical decision support tools like Merck Manuals, epocrates, UpToDate, DynaMed and VisualDx are applications and websites that bring the most recent medical evidence to the clinician at the bedside. Editors working behind the scenes review scientific literature and integrate it into relevant clinical guidance. At UpToDate, for example, more than 7400 subject matter experts review emerging research related to their topic areas and update the tool’s guidance as relevant to make sure clinicians can easily access the most current evidence when caring for patients. Clinical decision support tools can suggest key follow-up questions or tests to consider, support in weighing diagnostic probabilities, show visual images to help with identification of disease or rashes and more. Previous research has demonstrated a positive connection between evidence-based clinical decision support tools and clinician capacity; the use of UpToDate, for example, was shown to increase performance on standardised examinations among US clinicians5 and, most importantly, to reduce risk-adjusted mortality rates at non-teaching hospitals.6
Despite these proven benefits, uptake and use of digital tools among clinicians around the globe remain inconsistent.9–11 In fact, the World Medical Association recently acknowledged that lack of access to timely, current, evidence-based healthcare information—which such digital tools can provide—is a major contributor to morbidity and mortality in resource-limited settings.12 For some, the cost of a subscription, which can be up to US$580 for an individual, limits access.
In 2009, we started a programme that removed the cost barrier by offering free access to UpToDate for clinicians serving vulnerable communities at resource-limited health facilities, with the goal of improving patient outcomes and health equity. Eliminating the UpToDate subscription cost led to increased use of the tool; however, we observed wide discrepancies in use patterns, suggesting that other barriers to use persisted.13 To better leverage the potential impact of evidence-based clinical decision support tools in limited-resource settings, it is important to understand what factors affect their uptake and use.
Using data from a global sample of clinicians who received UpToDate subscriptions through our online donation programme, we conducted a mixed-methods cohort study. The general objective was to describe and explain the barriers and facilitators to use of the tool. Specifically, we aimed to describe:
The variation in UpToDate use by demographic characteristics of users,
The prevalence of barriers to and facilitators of UpToDate use in clinical practice, and
The relationship between barriers, facilitators and UpToDate use.
Study participants reported barriers and facilitators in repeated surveys over 1 year, and actual use of the tool was measured through clickstream data gathered from Wolters Kluwer/UpToDate.
Methods
Study sample
All clinicians who went to the online donation programme during our 9-week enrolment period (1 March 2018 to 4 May 2018) were invited to participate in and consent to the study electronically before applying. Informed consent covered the collection of the application, survey and clickstream data. Eligibility criteria for the donation programme included being a physician, surgeon or physician assistant outside of the USA; working for a public or non-profit limited-resource health facility; having at least intermittent internet access; being able to complete the application in English; attesting they are serving vulnerable populations (patients with limited-resources); and attesting they are not otherwise able to afford the subscription. Team members looked up health facilities and verified consistency in application information to confirm eligibility . Recruitment for the donation programme relies primarily on word of mouth and occasional communications to beneficiaries suggesting they invite their colleagues to join. No additional recruitment efforts were undertaken for study purposes (application is available at www.better-evidence.org). The decision to limit participation to those able to complete the application in English stems from the fact that the content within UpToDate is only available in English. We acknowledge that language is a barrier to access and did not feel it was necessary to test this hypothesis at that time.
Patient and public involvement
No patients were involved.
Logic model
We built a logic model detailing how access to UpToDate could eventually affect patient outcomes (figure 1). In this model, the inputs were the donation itself and technical supports such as a functioning internet connection. These enabled users to login to UpToDate and learn about it through the included orientation materials. These activities would then enable several short-term outcomes, including actual use of UpToDate, ability to navigate the tool and perceived utility of the tool in practice. Medium-term outcomes included increased medical knowledge, integration of that knowledge into practice and increased self-efficacy. In the longer-term, these elements could lead to faster and more accurate diagnoses and clinical management, which would eventually translate to improved patient outcomes. This overall process would be facilitated by a professional context that supported the use of digital clinical decision support tools in practice.
Logic model showing how UpToDate use among clinicians can impact patient outcomes. UTD, UpToDate.
Due to time and resource constraints, we could not measure all components of this logic model, but we measured several elements through two data streams: surveys and clickstream data.
Surveys
The survey included demographic, quantitative and open-text response fields. We captured respondents’ gender, age, years of experience, country of practice, urban/rural setting, patient load per week and employment type (full-time paid vs other).
We developed survey questions based on seven factors in the logic model downstream of the inputs as delineated in table 1 (see online supplemental materials for survey questions).
Supplemental material
Barriers and facilitators measured in surveys
Factors 1–3 were measured as barriers to use. Factors 4–6 were measured as facilitators of use. Four types of clinical decisions were covered in the survey: treatments, diagnoses, devices and procedures. We measured Factor 7, a sense of self-efficacy, with the 8-item New General Self-Efficacy scale.14 We added a contextualising frame at the start of the scale: ‘When providing clinical care, how true are the following statements for you?’
Twelve prior donation recipients from three continents, ranging in age from 25 to 65, provided feedback on the survey’s clarity, wording, response options and acceptability. The survey was shortened, language was updated and feedback was incorporated after several reviews and circulated to remaining reviewers for further review and refinement.
We integrated the baseline survey and the UpToDate donation application. Following the application approval, survey links were then triggered to be sent by email for the 2-month survey (sent 60 days after approval), 4-month survey (120 days after), 6-month survey (180 days after) and 12-month final survey (350 days after). We excluded survey answers that were completed more than 30 days after the survey link was sent.
The baseline, 6-month and 12-month surveys covered all topics; to reduce respondent burden, the 2-month and 4-month surveys only measured self-efficacy and barriers to use. Participants automatically received a 6-month subscription extension for completing the 6-month survey and another 6-month extension for completing the 12-month survey. In addition, those completing the 12-month survey were entered into a drawing for 10 prizes of US$100. The survey was built and administered in REDCap (Research Electronic Data Capture).15
Clickstream data
We measured the actual use of UpToDate (purple box in figure 1) through the tool’s clickstream data, a machine-generated record of each click from every user. The records identified which pages users visited and when, starting from the day the subscription link was sent out for 365 days, across all mobile and desktop applications as well as during offline use.
We linked the survey data to the clickstream data through a unique identifier. We qualified online use in two ways: first, we created a binary indicator of whether a user ever logged on through the donated link, called ‘ever-users’ and, second, we calculated the total amount of time ever-users spent using UpToDate over the yearlong study period. We estimated the length of specific user sessions as a function of (1) the time between clicks, (2) the content or function clicked on and (3) overall estimates of the amount of time spent reading content, navigating the site and managing user accounts. These methods have been detailed elsewhere.16
Quantitative analysis
We grouped countries into the six geographical regions used by the WHO. We determined the total number of donees in each respondent’s country using historical administrative data from the donation programme. We reported the per cent distributions of all demographic characteristics of the study sample. We collapsed 34 categories of specialties into eight groups (see online supplemental appendix A).
Supplemental material
We then calculated the per cent of each demographic subgroup who were ever-users, and among them, the median number of hours they spent using the tool over the year. We used median hours instead of means due to a highly logged distribution. We presented the proportion of users who experienced each barrier or facilitator once they had the subscription, at the 2-month, 4-month, 6-month or 12-month mark.
Next, we modelled the relationship between barriers, facilitators and use of the tool. The first set of regression models predicted the use of the tool around the time of the survey. For each user, we first identified the date they completed the 2-month, 4-month, 6-month or 12-month survey, and summed up the amount of time they spent using the tool in the 2 weeks around that date (7 days before to 7 days after), using the clickstream data. We fit 21 statistical models, 1 for each barrier or facilitator we measured, of the form:
Where: β0=intercept.
Yi=any use of the tool by subject i in the 2 weeks around survey month m (binary).
m=month of survey (encoded as a continuous variable with values 2, 4, 6 and 12).
BFim=presence of barrier or facilitator for subject i at survey month m (binary).
Χi=vector of demographic characteristics for subject i.
These 21 generalised linear models used a binary link function to the outcome and accounted for repeated measures over each subject.
The second set of models included only ever-users of the tool and predicted the minutes spent using the tool around the time that a barrier or facilitator was reported to be present. Like the first set of models, these accounted for repeated measures over subjects. The dependent variable—the minutes of use around each survey—was logged to bring its distribution closer to normality, and no link function was applied.
To select demographic variables to include in the model, we tested each variable for the strength of its relationship to both outcomes and for collinearity with other demographic variables. This process identified three controls to include in the model: age category, specialty and the total number of UpToDate donation recipients in the user’s country. In order to constrain the risks of multiple testing over the full set of (42) models, we set the alpha level for each coefficient at 0.0012, which is the standard alpha of 0.05 divided by 42. In line with this alpha threshold, we present 99.9% CIs. All analyses were done in SAS V.9.4 (Cary, North Carolina, USA: SAS Institute).
Qualitative analysis
We imported the free-text responses from the surveys into NVivo V.12 (QSR International) for coding and analysis. The coding scheme included high-level themes developed deductively from the research questions and subthemes developed inductively based on the content of the responses. Responses tended to be brief, containing a single idea closely aligned with the theme, so codes were applied with little need for interpretation or subjectivity. We included a sample of 250 surveys for analysis, choosing at random from across the spectrum of tool use. Because of the nature of the responses, one person coded all the responses for consistency under supervision.
Results
We had 1681 study enrollees and collected baseline data on all. Follow-up survey response rates were 67% at month 2, 60% at month 4, 54% at month 6 and 58% at month 12. Eighteen per cent of the respondents answered all four follow-up surveys, and 36% answered none. Based on the clickstream data, 249 (15%) of the enrollees never used the tool at all; although, 245 (98%) of these did respond to at least one follow-up survey.
Demographic characteristics
The vast majority (69%) of study enrollees were men, and most respondents (71%) were between 25 and 35 years old. As is typical, years of experience was highly correlated with age, and most respondents (55%) had four or fewer years of experience. Many subjects (42%) were general practitioners, with 22% in a medical subspecialty. Surgery, paediatrics and other specialties each had under 10% of respondents. Nearly two-thirds of the sample (61%) was in full-time paid work. Patient load fell into rough quartiles: 20% saw under 50 patients per week, 25% saw 50–99 patients, 29% saw 100–199 patients and the remaining 26% saw 200 or more patients. Most subjects (57%) were in urban settings, with 26% in rural settings, and the remainder in mixed areas (figure 2).
Population demographics and use of the tool. Med., medical; OB/GYN, obstetrics and gynaecology; Surg., surgical; SE Asia, Southeast Asia; SS Africa, sub-Saharan Africa; W Pacific, Western Pacific.
Two-thirds of our sample came from countries with 200 or more other donation recipients. A quarter of the respondents came from countries with 50–199 donation recipients, and the remaining 9%, from countries with only 1–49 other donation recipients. Eighteen study participants were the first and sole donation recipients in their entire country. Finally, the study sample included clinicians from all six geographic regions, mainly from Southeast Asia (35%) and sub-Saharan Africa (33%).
Variation in use by demographic characteristics
While 85% of the sample used the tool at least once, per cent of ever-users ranged from 77% to 89% depending on the demographic group (figure 2).
Among ever-users of the tool (N=1432), median time spent with the tool was 5.0 hours over the course of the study year. However, time varied strongly by some demographic groups (figure 2). Variation by specialty was marked, ranging from 1.9 hours for surgical subspecialists to 7.3 hours for medical practitioners. Similarly, variation by geographical region was large, from 3.3 hours in Sub-Saharan Africa to 7.2 hours in Europe.
As for age, the middle age group (25–35 years) used the tool for 5.8 median hours, while the younger users (under 25) used it for 4.2 hours, and the older users (over age 35) used it for 3.2 hours. The lower use among older users was also reflected in the results by years of experience: those with seven or more years of experience used the tool for less time than others (3.9 hours vs 5.4 or more hours).
Those with the highest patient load (200 or more patients per week) used the tool for comparatively longer over the year, 6.2 median hours, compared with the median across other groups, 4.5 to 4.8 hours. Users in countries with many donation recipients (200 or more) used the tool for 5.6 median hours over the year, while those from countries with fewer than 200 recipients used it less, for 3.8 to 4.0 median hours. There was very little variation in median hours of use by gender, employment type or urban/rural setting.
Prevalence of barriers and facilitators to use of UpToDate in clinical practice
The least common technical barrier (figure 3, Factor 1) was lack of a device (6% or less at all time points), and the most common barrier was slow internet speed (reported by about 33% of users at months 6 and 12). The per cent of users reporting difficulties with access to the internet declined over time, from 31% at month 2 to 16% at month 12 (figure 3, Factor 1).
Per cent of users reporting presence of each barrier or facilitator by survey month.
Few users reported barriers to navigating the tool (figure 3, Factor 2). In each follow-up survey in which these questions were asked, 9% or fewer respondents reported that they faced barriers either in knowing what was available or in finding the information they needed in the tool.
Fewer than 20% of users at any time point faced the barriers of lack of time, understanding the medical content or finding it difficult to fit into their workflow. One clinician mentioned in a free-text response, ‘Even though I don’t speak English fluently, I can understand easily because the terms they use are not complicated…it’s very easy when you want to find out something…you get it quickly’. However, enrollees also explained workflow concerns: ‘Patient flow is way too high. So I don’t get time to open UpToDate at that time…’ and ‘[It’s] tough opening UpToDate and checking patients in a crowded and hurr[ied] situation’. Regarding having what was needed to apply the information learnt from the tool in practice, the percentage of users reporting this barrier rose from 13% at month 2 to 32% at month 12 (figure 3, Factor 3).
As for facilitators, approximately 40% of the respondents at months 6 and 12 reported that they had ever referred to the orientation materials (figure 3, Factor 4). The utility of the tool in practice, measured as the percentage of users who reported being able to find answers to questions more readily as compared with before they had the tool, was stable across months 6 and 12: 47% of respondents were better able to find answers to treatment questions, 43% to find answers to diagnostic questions, 34% to procedure questions and 33% to device questions (figure 3, Factor 5). Clinicians shared examples of using the tool:
Let me exemplify a case of pneumothorax. There was a lot of debate regarding the tube thoracostomy. One of the residents read out the contents of UpToDate, and thence the tube thoracostomy was planned.
I have been using UpToDate to make management plans for my patients and to optimize their care. Whenever I am having a problem getting a diagnosis for a patient, I go to UpToDate and read around the topic.
The professional context results were fairly consistent across months 6 and 12. Approximately 80% of the respondents reported that clinicians typically viewed the use of a digital tool like UpToDate positively, and roughly 70% said that most of their clinical colleagues used such tools. About 65% reported using the tools often or very often in front of other clinicians (figure 3, Factor 6). Open text answers related to this factor include responses such as ‘Seniors [attendings] recommend it’ and ‘It is commonly known and most colleagues use it’. One clinician explained, ‘I came to know about the subscription of UpToDate through my colleague. There was an incident when I was working late night duty. I was confused about the latest recommendation, and my colleague helped me with the help of UpToDate’.
Clinicians did not feel patients were as supportive of tool use. Only 30% of the subjects reported that they believed their typical patient viewed the use of a tool like UpToDate during care positively, and about a quarter used the tool often or very often in front of patients during clinical care (figure 3, Factor 6).
Self-efficacy
The self-efficacy results were problematic, including ceiling effects and evidence of straightlining (24% of all administrations of the scale had the same response for all eight questions). Moreover, we found almost no group-level variation where it might be expected: across age, years of experience, specialty, geographical region or any other demographic group. Self-efficacy scores showed no consistent or notable increase or decrease over time, either on the group level or the individual level. By comparison, other survey questions did exhibit these basic features of item validity and functioning. Given it is implausible that the self-efficacy of all clinicians was identical and unchanging, we concluded that the psychometrics of the self-efficacy scale did not function properly in this study. For this reason, we dropped self-efficacy (Factor 7) from our presentation of results.
Relationship between barriers, facilitators and UpToDate use
Results of the statistical models are presented in figure 4. Panel A shows the estimated ORs of using the tool around the time when a barrier or facilitator was present compared with when it was not present, adjusted for age, specialty and number of donation recipients in the subject’s country. For the 11 barriers, most estimates were less than 1, suggesting that the odds of using the tool were lower when the barrier was present. However, only one of these relationships rose to statistical significance under the multiplicity adjusted alpha threshold: when clinicians reported that it was difficult to fit the tool into their workflow, they were 42% less likely to use it (OR 0.56, p=0.0003).
Relationship between barriers, facilitators and use of the tool around the time of the survey. UTD, UpToDate.
For facilitators, most ORs were near or above 1, suggesting that the odds of using the tool may have been higher when the facilitator was present. Of the 10 facilitators, two were statistically significant. First, users were 1.5 times more likely to log on if they reported that using UpToDate increased their ability to find answers to their clinical questions about treatments (OR 1.5, p=0.0001). Second, users were 1.7 times more likely to log on to the tool if their professional context supported using the tool in front of other clinicians (OR 1.7, p<0.0001).
Panel B shows the estimated ratio of minutes using the tool around the time that the barrier or facilitator was present. For the 11 barriers, none of these coefficients was statistically significant, although most were below 1, which was in the expected direction. Among the 10 facilitators, most were above 1, suggesting longer use of the tool at the time that the facilitator was present. One coefficient reached statistical significance: when users felt that they could more easily find answers to questions about diagnoses, they spent 1.4 times as many minutes using the tool, compared with when they did not feel they could answer more questions (ratio of minutes 1.4, p=0.0004).
Discussion
Our results drew attention to three factors relating to clinicians’ uptake and usage of UpToDate. The first factor (Factor 3) highlighted the ability to integrate the digital tool into practice. Of statistical significance, when clinicians reported difficulty fitting the tool into their daily workflow, they were only about half as likely to log on to the tool as when they did not face that difficulty. Although under 20% of the clinicians reported lack of time, difficulty fitting the tool into their workflow or problems understanding the medical content, and not all had statistically significant findings, clinicians who faced such barriers did appear to use the tool less. Interestingly, over the study year, the prevalence of not having what was needed to apply the information in UpToDate (Factor 3) rose from 14% to 33%. This increase over time could demonstrate decreasing resource levels for clinicians or clinicians’ increased knowledge of the resources they lack. In other words, clinicians may have been more aware than previously of newer supplies and tests that were unavailable to them after a year of using UpToDate. Regardless, the presence of this barrier did not deter use: it was not associated with how likely users were to log in to UpToDate nor the number of minutes they spent using the tool.
Second, the facilitator of perceived utility of the tool (Factor 5) seemed to matter for uptake. For example, the percentage of subjects reporting an improved ability to find answers to questions about treatments and diagnoses (as compared with before having access to the tool) was consistently above 40%. Moreover, though not all correlations were statistically significant at the multiplicity adjusted threshold, donees recognising the tool’s utility for treatment and diagnostic decision-making were more likely to log in to the tool and spent more minutes on the tool than those who did not report increased ability to find answers with the tool. In other words, positive perceptions of the tool’s utility for diagnoses and treatment correlated with more use of the tool.
Third, a positive professional context (Factor 6) also seemed to facilitate tool use. Measures of professional context (the belief that colleagues viewed the use of the tool positively, most clinical colleagues used the tool and used the tool in front of other clinicians) were all consistently reported by more than 60% of the participants. When subjects reported feeling comfortable using the tool in front of other clinicians, they were approximately 70% more likely to log in (statistically significant) and spent 30% more minutes on the tool (not statistically significant at multiplicity-adjusted threshold). Study participants in countries with 200 or more donation recipients used the tool for longer over the year compared with those in countries with fewer donation recipients. A professional context in which more clinicians had access to the tool and felt comfortable using it in front of other clinicians was associated with more use of the tool.
Other barriers and facilitators we tested did not show these kinds of relationships. For example, facing technical access barriers did not significantly change the odds of using the tool or of the amount of time spent using it. This result may seem counterintuitive but likely points toward the determination of these motivated users. For example, at months 2 and 4, about one-third of the users reported that access to the internet was a barrier for them, but this proportion fell to about 20% at months 6 and 12, and limited access to the internet was not related to the likelihood of logging on or how long was spent using the tool. This could have resulted from differential dropout—those with worse internet access stopped responding to surveys—or the users may have learnt how to download and use the tool offline or secured better internet connections. These technical considerations were not the barriers to use that we might have expected. Similarly, users did not report high levels of difficulty navigating the tool or finding information on it. About 40% of the clinicians reported using the orientation materials, but reading those materials was not a significant facilitator of tool use.
One final factor related to usage was age. Only 7% of the study participants were in the youngest age group (<25), likely due to the fact that most people do not start practicing medicine until later. Those aged 25–29 represented 42% of all applicants, and, along with those aged 30–34, used the tool more than the oldest participants (aged 35+). This suggests there is a stronger interest in technology among the newest generation of clinicians and provides hope that uptake and use of digital clinical decision support tools may increase with time.
Our study had several limitations. First, while our sample of clinicians was large and diverse, it was non-representative across countries and types of clinicians; we accepted all clinicians who applied and met eligibility criteria for the donation programme during the study period. Eligibility criteria required that clinicians be able to complete the application in English and be working in a limited-resource setting. The sample included only clinicians motivated to apply to the programme, who self-selected to try to improve their practice, making it non-representative of the general clinician population. Thus, external validity and the generalisability of our conclusions may be limited. Second, any of the factors we explored can be framed and measured as either barriers or facilitators; we measured some as barriers and others as facilitators, which may have impacted how participants answered the questions. Finally, we were able to integrate the baseline survey into our application process in order to not alter the application experience dramatically; however, other surveys may have influenced tool use by reminding users about the tool when they normally would receive no such reminder.
Globally, the healthcare workforce faces scarce time and attention, high demand for services, varied patient populations and ever-growing medical literature. As a result, clinicians must remember, apply and integrate a massive volume of information under difficult circumstances. Digital tools can help, but only if clinicians can and do use them in clinical care. We believe that the patterns suggested here can serve as the basis for further implementation work and research to better understand how to best reach diverse, both more and less motivated populations of clinicians.
Conclusion
This study can inform the implementation of digital clinical decision support tools in the future. Findings suggest implementing the use of digital clinical decision support tools like UpToDate in cohorts of clinicians to generate supportive professional contexts, encouraging the use of such tools over time to increase exposure and help clinicians realise the utility of them and working with health systems to promote the use of clinical decision support tools in workflows to promote use.
There is great potential for digital tools to help ensure effective and high-quality care. By learning how to better facilitate use and minimise barriers among clinicians around the globe, we can take an important step toward more effective diagnostic and clinical management leading to better, more equitable health outcomes.
Data availability statement
Data are available upon reasonable request. Data may be obtained from a third party and are not publicly available. Data were used under a data use agreement for the current study.
Ethics statements
Patient consent for publication
Ethics approval
This study received ethical approval from the Partners Human Research Committee (now called Mass General Brigham Institutional Review Board) under protocol 2018P001183. The Harvard T. H. Chan School of Public Health institutional review board ceded review to the Partners Human Research Committee. All participants provided informed consent. The informed consent addressed the collection of both the survey and clickstream data. Participants gave informed consent to participate in the study before taking part.
Acknowledgments
Thank you to Atul Gawande and Lisa Hirschhorn for helping shape and support the vision. Thanks to Becky Hawrusik for analytical support, to Iman Ahmad, Claire Donovan, Shreyas Sirinath, Anup Mankar, Robbie Singal and Davin Eurich for technical, implementation, and project management support. Thank you to all our survey reviewers for the feedback and improvements. Thanks to the UpToDate team for securing the in-kind donations and the clickstream data transfer.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @JDRMPH
Contributors JR conceived the design of the work, contributed to survey design and data collection, to data analysis and interpretation, to drafting and critical revision of the article and to final approval of the version to be published. KM contributed to the design of the work, survey design, data analysis and interpretation, drafting and critical revision of the article and final approval of the version to be published. OP contributed to data collection, data analysis and interpretation, and final approval of the version of the article to be published. NH contributed to survey design and data collection, to data analysis and interpretation, and to revision and approval of the final version of the article to be published. AK contributed to drafting the article, critical revision and final approval of the version to be published. RW conceived of the design of the work, contributed to survey design, to data analysis and interpretation, critical revision of the article, final approval of the version to be published and is responsible for the overall content as guarantor.
Funding This study was conducted with in-kind donations of UpToDate subscriptions from Wolters Kluwer. The publishing company provided no monetary support to the study and was not involved in data analysis or interpretation. Thank you to the Paul G. Allen Foundation, Patrick J. McGovern Family Foundation and the Horace W. Goldsmith Foundation for financial support. Award/grant number: N/A.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.