Objective To investigate the nature of physicians' use of research evidence in experimental conditions of open access to inform training and policy.
Design This qualitative study was a component of a larger mixed-methods initiative that provided 336 physicians with relatively complete access to research literature via PubMed and UpToDate, for 1 year via an online portal, with their usage recorded in web logs. Using a semistructured interview protocol, a subset of 38 physician participants were interviewed about their use of research articles in general and were probed about their reasons for accessing specific articles as identified through their web logs. Transcripts were analysed using a general inductive approach.
Setting Physician participants were recruited from and registered in the USA.
Participants 38 physicians from 16 US states, engaged in 22 medical specialties, possessing more than 1 year of experience postresidency training participated.
Results 26 participants attested to the value of consulting research literature within the context of the study by making reference to their roles as clinicians, educators, researchers, learners, administrators and advocates. The physicians reported previously encountering what they experienced as a prohibitive paywall barrier to the research literature and other frustrations with the nature of information systems, such as the need for passwords.
Conclusions The findings, against the backdrop of growing open access to biomedical research, indicate that a minority of physicians, at least initially, is likely to seek out and use research and do so in a variety of common roles. Physicians' use of research in these roles has not traditionally been part of their training or part of the considerations for open access policies. The findings have implications for educational and policy initiatives directed towards increasing the effectiveness of this access to and use of research in improving the quality of healthcare.
Statistics from Altmetric.com
Strengths and limitations of this study
This study is among the first to simulate universal open access to research for physicians, enabling a study of what research literature they turned to with such access, if any, and for what purposes.
A strength of this study is that participants were queried about their use of the biomedical articles that they selected and viewed.
The use of a semistructured interview protocol enabled researchers to tailor the interviews to each participant's explanation of his or her use.
A limitation of this study is that it was unable to track use of research from outside of the study portal.
This study's findings are also limited by its recruitment of US physicians only.
In today's complex healthcare system, physicians play multiple roles beyond caregiver, many of which are affected by the availability and use of information. Physicians collaborate on interprofessional teams to improve patient safety;1 analyse processes and make recommendations to contain costs;2 contribute to the larger body of knowledge for clinical care; advocate for the health of the public3 and teach the next generation of healthcare providers.4 Yet how they use research in these roles has been studied little, despite growing access to this literature. This lack of knowledge has implications for educators designing physician training and policymakers considering public access mandates for research.
In addition to being clinically focused, the research on physician information use has concentrated on usage of point-of-care resources, which are designed to save physicians time through synthesised evidence summaries.5–7 In light of findings that physicians generally dedicate no more than 5 min in pursuit of evidence,8 this focus has gained prominence. However, the study reported here found, through an initial quantitative analysis (of which the analysis below is the qualitative complement), that 34.2% of physicians (N=336) took time to locate and consult a relevant research article with an average frequency of somewhat more than once a week when they were provided with the relatively complete access to this literature afforded by access to Stanford University Lane Medical Library.9
The advent of open access to biomedical literature is increasing physician and patient opportunities to consult the primary research literature. The National Institutes of Health (NIH) Public Access Policy, for example, requires that the research undertaken with NIH funds be made freely available within 12 months of publication, and a new generation of open access journals have emerged in the biomedical field.10 The proportion of recent literature that is freely available to physicians and the public has crossed the 50% point and will only increase going forward.11
Thus, there is a need to understand the uses that physicians can be expected to make of this growing access to research in the course of their clinical practice and other professional roles. To that end, this paper analyses interviews with 38 physicians about their general use of research, with a subset of 26 physicians about their use of specific research articles accessed during a study, which provided them with full-text online access via PubMed to the near-complete biomedical literature available. Based on these findings implications for education and policy are considered.
This qualitative analysis of physician interviews represents a component of a larger mixed-methods research initiative led by this author team. This study was reviewed by the Stanford University Institutional Review Board and deemed exempt from further review.
Between March 2013 and July 2013, the study enrolled 336 physician participants and 98 public health non-governmental organisation staff in the USA. The initiative provided all participants with relatively complete online access to the research literature indexed in PubMed, as well as access to the research summary service UpToDate, for 1 year via an online portal, with their usage recorded in web logs.9 This unfettered access to research simulated universal open access. Web logs provided details of participants' use of the portal, including search terms used and research articles accessed. The quantitative results, reported earlier, demonstrated that 34.2% of the physicians viewed at least one research article in the course of 11 months, with the average for the physicians who viewed articles amounting to 1.2 articles a week.9
Of the 131 participants contacted for an interview via email, 44 responded, and 38 (38/131, 29%), ∼10% of the total sample, were interviewed between April and June 2014, within 2–4 weeks of the end of their access to the Stanford University Library collection. In this paper, we report on interviews with the 38 physicians who represented a range of article usage levels: extreme (7 physicians had accessed 200 or more articles), high (9 who accessed 40–199 articles), medium (10 had accessed 11–39 articles) and low (6 had accessed 1–10 articles), as well as those (6) who did not use any research articles. In the overall study, 65.8% (n=221) of participants did not access any research articles.9
Of the 38 interviewed, 22 (58%) were male and 16 (42%) female. Of these participants, 12 (31.5%) graduated from medical school prior to 1990, 4 (10.5%) between 1990 and 1999, 14 (37%) between 2000 and 2009 and 8 (21%) from 2010 onward. Among the 38 physicians, there was representation from 16 states across the USA and 22 specialties, including radiation oncology, internal medicine, surgery and paediatrics.
LLM, a female doctoral student at the time with a master's degree in liberal arts, conducted phone interviews that ranged from 30 to 60 min in length. The interviews were audio recorded and transcribed verbatim. Transcripts were not returned to the participants. LLM wrote field notes during the interviews. No one was present besides the participant and researcher on the phone. LLM was trained to undertake the interviews through qualitative methods coursework and had experience interviewing people as a professional journalist. All participants were personally unknown to LLM and the research team.
Participants completed online consent forms at the beginning of the study. The consent forms informed the participants of LLM's role in the study, the study's goals and the research team's interest in physicians' access to the biomedical literature.
LLM asked all participants about their overall information use during the study based on a semistructured interview guide designed by the research team based on the biomedical literature. The interview guide was pilot tested with physicians not enrolled in the study. Participants were not provided the guide prior to the interview.
Despite being aware of their access to research via the study portal, six interview participants did not access an article using the study portal. These participants were therefore asked about their overall information use external to the portal and were questioned about their non-use of the portal.
The 32 participants who viewed articles were presented with the titles of 1–9 of the articles that they had viewed during the study. A total of 122 article prompts were presented to participants. Participants were asked to describe the circumstances and perceived outcomes in which they accessed the mentioned articles and/or topics based on search terms. Six of the participants were unable to recall the circumstances around their search and possible use of an article.
LLM, LAM and JMW, whose expertise is in education, information science, and journalism, analysed the interview transcripts. Operating from a constructivist paradigm,12 we used qualitative description13 and inductive method.14 We identified physician use of research through multiple readings of transcripts. We collectively identified a preliminary coding structure based on uses, with a checklist matrix,15 allowing for participant comparison, simple quantification and verifiability of codes, as well as the identification of major themes and disconfirming cases. The checklist matrix was created using the qualitative analysis software (Dedoose V.7.0.23). Our coding was sensitised by the physician roles identified in the CanMeds Framework,16 which we modified based on our analysis of the transcripts.
Following general coding of all transcripts (n=38), we conducted a secondary analysis of participants' information use roles that focused on the 26 participants' responses to the interviewer probes on their searches run and articles accessed as identified in the web logs. Codes related to information use roles were applied to all 26 transcripts and agreement was reached, leading to thematic saturation in identifying six major information use roles that were prevalent throughout the transcripts and/or that were emphasised by participants. Participants were not asked to provide feedback on our findings.
Among the 38 transcripts, universalities across participants included an awareness of journal article costs, the use of research outside of the clinical setting and frustration with online systems for accessing information generally. Based on collected demographics (eg, specialty, year of graduation, gender), we did not observe notable differences between those who viewed articles and those who did not. The six physicians who did not use research articles within the study provided a variety of reasons for non-use, ranging from forgetting that they had access to simply not having time to use the portal. No non-users mentioned the adequacy of the information provided by UpToDate as a reason for not accessing the research articles.
Where possible, quotes from a variety of participants representing various specialties are presented. Each physician quoted is identified by a number and specialty (eg, MD-147, internal medicine).
When asked about general information use, both within and external to the study portal, physicians offered a range of reasons for low or non-use. Specifically, we noted when describing practices outside of the study, over half of participants (n=22) mentioned the actual costs—with $35 the most common figure identified—involved in purchasing access to individual journal articles. Participants identified article viewing costs as a barrier. An adult pulmonary/critical care specialist said, “I won't pay, because it's so expensive. I'm never going to pay $45 for an article. I just do without” (MD-96). Participants did not identify if any specific level of paywall would be acceptable; however, one internal medicine physician commented in regard to purchasing access: “It's probably something I would end up paying for just to have that access…to do my job well” (MD-147).
Related to costs, physicians presented a nuanced perspective of the time they dedicated to locating and securing research. While busy and typically unable or unwilling to search out answers for all their clinical questions, particularly while at work, most said they would allot a great deal of time (up to several hours) if a situation demanded or their curiosity could not be satiated. Physicians regularly allotted more time to their search in hopes of finding freely accessible research. An obstetrician/gynaecologist explained, “I don't mind the time. The cost seems — I mean, if you were doing it a lot — the cost is kind of ridiculous. They usually want, like, $30, $40…So that would definitely be pretty unusual that I would purchase it” (MD-207).
Participants reported information use outside clinical work settings, whether in the evenings or early mornings from home. This use was generally tied to participants' curiosity, desire to stay up-to-date and for teaching purposes. When related to patient-centered clinical questions, physicians generally attempted to look up information in the moment, if possible. For example, when asked about when they use information, a paediatrician commented:
Usually during work, just for direct clinical care, like looking up doses of medications or treatment questions, things like that. And then if I use it at home, it's mostly either I happen to have to follow up like a lab or something, or some issue that comes up after hours that is for direct clinical care, and then the only other time I would access it is for if I'm doing a presentation. (MD-350)
Although participants were questioned specifically about their use of the Stanford portal during the study period, physicians across the sample voiced frustration with the information systems and lack of information access more generally available to them outside of the study:
A lot of article research papers…have their own little online portfolio that you have to sign into every single time and every single one had different ways to access it from a username and password and certain hospitals or certain universities have certain ways of getting into the site that's very cumbersome sometimes and so it takes 30 minutes to get to one article that should've taken only a minute. (MD-171, emergency medicine)
Frustration was especially pointed in relation to needs for multiple passwords, and two participants referred to the access as ‘cumbersome’ especially when not in their offices (MD-147; MD-162). Several participants also reported that information resources made available through their practices or hospital systems were inadequate. For example, one physician explained:
I'm pulmonary clinical care, and one of the major—like it's not even an obscure journal; it's CHEST. And [my hospital library] made a decision that they will only have access to the archives, so I can only get access to papers that are a year old or older. (MD-96, pulmonology)
Participants also voiced frustration with the timeliness of research access. For example, a rheumatologist said: “Say something big comes out, especially if it hits the news, things like phosphates causing atypical fractures…or hormone replacement therapy actually is worse for women and we need to stop that. You know, if I wait six months before I read the paper and start acting on it, I'm feeling pretty bad” (MD-28). This physician also pointed out, “If I advertise to my patients that I am up to date as of one year ago that doesn't sound very good” (MD-28).
In some cases, this frustration and inability to access desired information led to workaround strategies. For example, emergency medicine physician said:
If there was an article I wanted to read then I would read the abstract and…I would either try and search for it in Google to see if somebody had posted a PDF somewhere or if I really needed to read it and couldn't bear to be without it, I would email a colleague who has university access and ask them to send in the article. (MD-473)
Research use by role
The 26 physicians who were asked about their viewing of articles attested to the value of consulting research literature as contributing to their work within six of the typical roles that physicians play, namely, as clinician, learner, educator, researcher, administrator and advocate (table 1). Participants described information use in each of these roles, with some overlap among them. We did not detect from the interviews differences among the low, medium, high or extreme users of the literature and the roles in which they used the information. Roles are presented in order of frequency observed among participants.
Based on the provided prompts from their web logs, all but 1 of the 26 participants described using the mentioned articles for patient care (table 2). To support patient care, participants often used research to ensure that they were providing care that was safe, up-to-date and that met current standards and guidelines. While the use of review articles was noted in most roles, they were most frequently used in clinical care (n=14). Participants referred to how they used this information to inform care choices, such as selecting a particular therapeutic approach or diagnostic strategy when there is high risk, a need to ‘refresh’ and an interest in the latest findings.
In several instances, participants used articles to facilitate a conversation with their patients and engage in shared decision-making. Several physicians reported developing a file system that allowed them to further share that information with future patients. Physicians also found value in sharing research with their colleagues, occasionally to end a debate but often to update guidelines or challenge established practices. Few participants did not identify use, and there were several counter examples. For instance, one participant who is now more engaged in the management of a start-up healthcare organisation reported not using any of the articles reviewed for patient care (MD-146).
As educators, participants primarily mentioned retrieving articles to educate medical students and residents (table 3). Articles were used in one-on-one training encounters and in group sessions such as noontime resident conferences, journal club activities or undergraduate medical courses. Some participants also used information to prepare for their educational roles. Oftentimes, participants were willing to dig deep for research for education. One physician conducted 57 searches in preparing for a talk for residents.
Several participants highlighted how they relied on research to help educate their patients in interpreting media articles and reports as well as journal articles about health. For example, participants also explained how they searched for articles to learn about so-called hot topics, including medicinal marijuana, the side effects of statins, and soft drinks and obesity to communicate on these topics with their patients.
Participants identified themselves as researchers in addition to clinical care providers (table 4). Generally, these individuals heavily accessed articles through the site. Research efforts ran the gamut, from a paper-writing graduate student (MD-57) to authors writing for mainstream publication (MD-57, MD-36), bloggers (MD-287, MD-57, MD-171) and a grant-writing policy analyst (MD-81).
Participants reported that their clinical care inspired their research and use of information such that a particular patient case, perhaps rare in nature, would be the impetus for a project. Occasionally, the physicians' research needs were specific. More generally, though, these individuals relied on journal articles to keep updated and to explore their ongoing research interests.
As learners, participants accessed research to support their continuing development as professionals (table 4). In these cases, access was self-directed and generally not mentioned within the context of continuing medical education (CME) accreditation or maintenance of certification. Additionally, we noted participants used information to improve their knowledge of conditions common in their practice. Several participants also expressed a desire for information to satisfy their curiosity and to keep current on important or popular topics in their field.
As administrators, participants accessed research articles to help shape guidelines and treatment protocols for their health centres, influencing health centre policy, and responding to cost factors (table 5). This was observed at the unit, departmental and centre levels. For example, a gynaecologist was searching for information on universal screening for methicillin-resistant Staphylococcus aureus in all preoperative patients to inform his health centre's policy (MD-207). Physicians also mentioned searching for information to compare and contrast effectiveness of treatment and diagnostic approaches to justify costs.
Occasionally, physicians took on the role of advocate in a way broader than their care of their specific patient (table 5). In this role, physicians engaged in projects that would impact population health such as working with legislative officials to inform future healthcare laws and policies, informing insurers, and with hospital and community-wide programmes, such as breastfeeding initiatives. In some cases, physicians played advocacy roles across several public health concerns within their communities.
In this study, participants were provided with a simulation of universal open access to the biomedical literature. While earlier studies have addressed physician's clinical information needs and uses,17–19 participants in this study demonstrated that access to research can contribute to the additional roles played by physicians. Beyond their clinical responsibilities, which prompted use of research in this study, those who consulted the research also turned to it in their roles as educators, researchers, learners, administrators and advocates. Our findings point to the need and value of preparing physicians to work with the increasing degree of open access to research resources, as it can serve the spectrum of roles they play, encompassing such topics as administrative cost-control, community-based advocacy and shared decision-making.
In reflecting on their previous experiences with research, participants expressed their awareness of per-article access charges based on a first-hand familiarity with paywalls. This experience presumably arose out of an interest in consulting this literature, while indicating that these charges act as a deterrent rather than an enabler for those without subscription access.
Some physicians engaged in complex workarounds in hopes of accessing paywalled research. This finding contrasts with previous research demonstrating that physicians tend to use readily available information even to the point of sacrificing quality for efficiency.6 ,20 ,21 These results suggest that there is a risk that physicians will not be attuned to the gradual increases in access taking place, as their encounter of costs barriers may have effectively closed off, to their thinking, the availability of such resources. Future research might explore physician awareness of the expansion of access and might test assumptions as to at what price point physicians will purchase articles instead of investing their time attempting workarounds.
The changing state of public access to research needs to figure in 21st century medical education, along with an awareness of the tools, systems and processes that contribute to the professional evaluation, interpretation and usage of these resources. At this point, education is provided on locating and managing research evidence across the continuum of medical education.22 This suggest that the current clinical focus of this information training should be broadened to encompass additional skills and resources that address physicians' varied information needs. For example, researchers have proposed that curriculum be expanded to include training for sharing information with patients.23 We support this suggestion, but recommend medical educators also consider training to support physicians in their broader roles, as identified here. Therefore, training might include incorporating additional information resources, such as databases that include literature on approaches to administration, materials for patient education, or CME opportunities. However, it is worth noting that participants were self-directed in their use of the literature without CME credits being involved. This contrasts with certification organisations' long-standing efforts to offer CME credits for reading articles24 and suggests future research might explore physician motivations for information use to align CME offerings with practice. Finally, it is notable that six interviewed participants and over 200 of the physician participants in the overall study did not access research articles. This suggests a need for further research to understand what impedes such use, including potential gaps or weaknesses in training, ease of access, lack of incentives and need for evidence of value.
Over the last decade, the advent of policies such as the NIH Public Access Policy,10 the Bill & Melinda Gates Foundation Open Access Policy,25 the Research Councils UK Open Access Policy,26 and the Wellcome Trust27 have increased public access to biomedical research. While these policies are contributing to the extent of open access, our findings, on the perception of a universal paywall barrier, might suggest a need to further publicise and promote the policies and their benefits. Additionally, further research on physician knowledge of and uptake of research made freely available via public access policies is warranted. Finally, funding agencies may also want to consider the extent to which physicians find value in having access to categories of research that are not as well funded as biomedical research and thus not as likely to be made open access, such as review articles, which physicians found useful for training students or policy papers to use in advocating for patients. Therefore, we encourage policymakers to more broadly consider the types of information that should be funded and covered under public access policies.
It needs to be recognised that the physicians who consulted the research literature testify to the perceived value and application of the research to which they had access. There is no evidence of effect on patient care, in manner of evidence-based medicine. To establish the extent to which this access contributed to measurable improvements in the physicians' various roles will take further research, whether involving natural experiments or randomised control trials that compare results over an extended period. However, the results of this study do speak to changes in education, policy and outreach that can be expected to increase the usage and value of this research across the full spectrum of a physician's work, in light of the increased access to biomedical research that is taking place through various strategies. Additionally, as we interviewed only a sample of physicians, all practising in the USA, we cannot generalise our findings to the general physician population or to those practising outside the USA. Finally, this study does not exclude participants who viewed research outside the study's portal; however, participants' commentary on their experience of access to the literature external to this study is valuable and allowed us to better understand their overall information use.
Among a sample of physicians provided with relatively complete access to the biomedical research for 11 months as part of an experiment in open access, roughly a third consulted this literature in their roles as clinicians, educators, researchers, learners, administrators and advocates. These results draw attention to the need for educators and policymakers to take into consideration how these roles can be better served by growing levels of public access to research when designing training and future policy. They suggest addressing physician assumptions about the extent to which prohibitive paywall barriers apply to this research literature. These findings speak to a need, given current, future and perceived state of access, for greater research savvy among physicians, involving the informed use of and access rights to this literature, through both educational and policy initiatives. Although a minority of physicians accessed the research made available to them in the course of the larger study, this proportion can be expected to increase with the growth of awareness and training associated with continuing increases in public access to the biomedical research, with results that will, in general, contribute to the quality of healthcare.
The authors would like to express our gratitude to our participants, who shared their time and experiences and allowed us access to their web log data. They also thank Ryan M Steinberg and Alain Boussard for creating and maintaining the online portal and Cheryl A Holzmeyer for her contributions to the larger study.
Twitter Follow Lauren Maggio at @laurenmaggio
Contributors LAM, LLM and JMW conceived and designed the experiments, analysed the data, contributed reagents/materials/analysis tools, and wrote the paper. LLM collected the data.
Funding This work was supported by the National Science Foundation, grant number 1158673
Disclaimer The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Uniformed Services University of the Health Sciences, the Department of Defense or the US Government.
Competing interests None declared.
Ethics approval Stanford University Institutional Review Board.
Data sharing statement Additional data can be accessed via the Dryad data repository at http://datadryad.org/ with the doi:10.5061/dryad.0q2mq.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.