International Journal of Medical Informatics
Critical incidents and journey mapping as techniques to evaluate the impact of online evidence retrieval systems on health care delivery and patient outcomes
Introduction
Online evidence retrieval systems provide health professionals with fast and easy access to a wide range of information resources to inform their decision-making processes. Provision of online evidence systems to support clinical work at the point-of-care has been adopted as a strategy to support evidence-based practice in the UK, USA and Australia [1], [2], [3]. Few evaluations of their effectiveness have been reported [4]. Most studies have focused on assessing frequency of use rather than effectiveness and greater attention has been placed upon system use in family medicine rather than in hospitals [2], [4]. Measuring the impact of an information system on clinical practice is problematic. There are many potentially confounding variables for which it is often not possible to control. Thus establishing the relationship between online evidence system use and changes in clinical practice and patient outcomes is difficult. One of the largest studies to attempt this task was conducted by Lindberg et al. [5], who applied an adapted critical incident technique to gather reports from 552 US physicians about when they had found MEDLINE searches to be especially helpful (or not) in carrying out their professional activities. Physicians reported using MEDLINE for a range of clinical tasks, from choosing diagnostic tests to developing treatment plans. In 2% of instances of MEDLINE use physicians reported finding essential information that led to saving patients’ lives or sparing limbs and organs.
The introduction of MEDLINE into clinical settings in a US hospital was evaluated through interviews with a random sample of physicians who participated in MEDLINE training. Physicians reported that 47% of searches influenced their clinical decisions, most frequently through confirmation of a decision [6]. Another study found that length of stay was shorter for patients for whom a MEDLINE search had been conducted earlier in their hospitalisation. A causal relationship could not be established and a number of confounding factors such as the severity of patients’ conditions limited interpretation of the findings [7]. Laboratory studies have shown that answers to clinical questions are improved when participants are given access to online evidence retrieval systems [8], [9], [10].
In 1997 the state of New South Wales, Australia implemented the Clinical Information Access Program (CIAP) (http://www.ciap.health.nsw.gov.au) a website providing around 55,000 clinicians (doctors, nurses and allied health professionals) with 24 h access to evidence at the point-of-care in public health care facilities. A 2-year evaluation was undertaken 5 years after system implementation to assess the extent to which the system was used, supported clinical decision-making and resulted in improvements to patient care. A triangulated research program consisted of four stages of data collection. Results from the first three stages which investigated how frequently and why clinicians used the system have been reported elsewhere and are summarised below [11], [12], [13], [14], [15], [16], [17].
An analysis of 7 months of CIAP web-logs of the 55,000 clinicians who have access allowed a quantification of the rates of use by geographical Area Health Services, hospitals and professional groups. Considerable variation in the rates of use for individual hospitals and professional groups was found [15]. Medical staff used CIAP at double the rate of nursing staff. Results showed that use of CIAP was related to patient care decisions. For example, searching activity was highly positively correlated with patient admissions at individual hospitals. The wide variation in use required further investigation to understand factors related to high and low use organisations. Stage 2 sought to address these issues.
In-depth case studies investigated the influence of professional and organisational factors on health professionals’ use of CIAP. Results from focus groups, interviews and surveys from clinical groups within three case study sites identified a range of cultural, organisational and team factors which were important in explaining variations in CIAP use within and between professional groups and hospitals [11], [12]. Positive promotion of CIAP, and support and encouragement to use it were major factors influencing use. CIAP was used by all three professional groups. Nurses and allied health staff reported needing most support in the effective retrieval of information. Doctors were the highest users and most likely to use CIAP for patient care. Nurses’ awareness of CIAP was the lowest of the three groups.
Technical issues were also highlighted but were found not to be the central factor influencing CIAP use. Investigation of the association between clinical team functioning and use of CIAP to improve patient care produced interesting results, demonstrating that members of well-functioning clinical teams were more likely to report effective use of CIAP. However clinical team functioning was not associated with greater awareness of CIAP [11].
A statewide survey of 5511 clinicians (doctors, nurses and allied health professionals) from a random selection of 65 hospitals was undertaken to determine awareness and use of CIAP, how and why clinicians use, or do not use, online evidence, technical issues related to use of CIAP, and clinicians’ perceptions of its impact on clinical care. The survey showed that 63% of clinicians had heard of CIAP and of those who had, 75% had used it [13]. Colleagues were the primary channel via which clinicians heard about CIAP. The survey results reinforced the key findings of the log analysis and in-depth cases study findings. For example, clinicians’ actual and reported use of CIAP was consistent in terms of types of resources used. Clinicians’ views about the importance of organisational factors in supporting an evidence-based approach to care and the integrated use of CIAP, reported during the focus groups in stage 2, were reflected in the survey responses from the respondents. The survey identified high levels of satisfaction with the technical features of CIAP, such as ease of use and search speed. Analysis of the log data indicated that use was related to direct patient care decisions and the survey results reinforced this with, for example, 55% of doctors reporting that they had had direct experience of CIAP use resulting in improved patient care. There were significant differences between professional groups [14], [17] and also between senior and junior clinicians [16], [17] within the same profession. For example, senior and specialist nurses had more positive attitudes to use, perceived more support and legitimation for its use, and reported more frequent use than did junior nurses [17].
The results from stages 1–3 provided the context and direction for the last stage of the evaluation reported here which sought to investigate two techniques, the critical incident (CI) [18] and journey mapping (JM) [19] techniques, to assess clinicians’ integration of the use of CIAP and impact on clinical decision-making and patient care. These methods create a narrative of change from the perspective of system users allowing an examination of professional and organisational cultural changes and how users experience these changes.
The CI and JM techniques complement traditional approaches of questionnaires and interviews frequently used to assess the impact of clinical information systems on care delivery and outcomes. Both techniques incorporate the collection of rich narrative accounts along with some quantitative measure of the results obtained. The CI technique involves obtaining direct reports from system users about the ways in which use of the online evidence retrieval system had been useful (or not) to their clinical work. JM uses narrative accounts of clinicians’ introduction to, use of, and experiences with, the online evidence retrieval system. A structured numerical coding system, which reflects the level to which individuals have integrated the system and achieved improvements in care delivery, is then applied to these accounts. Our aim was to assess how useful these techniques were in measuring and understanding the impact of online evidence system use on care delivery and patient outcomes.
Section snippets
Methods
Sixteen clinical nurse specialists (CNS) and 13 hospital-based specialist physicians were selected who were experienced and regular CIAP users. We used a purposive sample as our aim was to examine what could be achieved when the online evidence system was used frequently and by experienced users in order to provide an indication of what outcomes were possible. We hypothesized that if we were unable to measure a positive impact of system use among this population it would be unlikely that the
Results
In total, 85 critical incidents were described by 29 clinicians interviewed. The average number was 2.9 per clinician (range 1–5). Three main themes emerged from the critical incident data, namely impact of CIAP use on: (1) clinical practice, (2) individual clinicians and (3) colleagues via the dissemination of information gained from CIAP.
Discussion
The data from the critical incidents provided an enhanced understanding of how the online evidence system was being used in clinical practice and reinforced and elaborated upon the information gained from other stages of the evaluation [11], [12], [14], [15], [17]. Clinicians were able to relate clear stories of how CIAP use had impacted on their own clinical practice and patient care. There were some differences between the types of clinical impacts described by doctors and CNCs, in line with
Conclusions
Our study has shown that an online evidence system assists clinicians in their work across a range of clinical activities. Interviewing clinicians about their experiences using the CI technique elicited detailed accounts of the positive impact of system use on care provision, and patient outcomes, as well as on clinicians’ confidence in decision making. The JM technique provides a useful method for quantifying the impact of system use on clinical practice and potentially for monitoring changes
Acknowledgement
This research was funded by a grant from the NSW Health Department.
References (24)
- et al.
Impact of clinical information-retrieval technology on physicians: a literature review of quantitative, qualitative and mixed methods studies
Int. J. Med. Informat.
(2005) - et al.
Do online information retrieval systems help experienced clinicians answer clinical questions?
J. Am. Med. Informat. Assoc.
(2005) - et al.
Variation in the use of online clinical evidence: a qualitative analysis
Int. J. Med. Informat.
(2003) - et al.
Allied health professionals’ use of online evidence: a survey of 790 staff working in the Australian public hospital system
Int. J. Med. Informat.
(2004) - et al.
Do clinicians use online evidence to support patient care? A study of 55,000 clinicians
J. Am. Med. Assoc.
(2004) National electronic Library for Health (NeLH)
Br. Med. J.
(1999)Information retrieval
A Health and Biomedical Perspective
(2003)- et al.
The clinical information access project
Med. J. Aust.
(1999) - et al.
Use of MEDLINE by physicians for clinical problem solving
J. Am. Med. Assoc.
(1993) - et al.
Online access to MEDLINE in clinical settings. A study of use and usefulness
Ann. Intern. Med.
(1990)
Effect of online literature searching on length of stay and patient care costs
Acad. Med.
The impact of an online evidence system on confidence in decision making in a controlled setting
Med. Dec. Mak.
Cited by (0)
- 1
Dr. Gosling undertook this research while she was a Research Fellow at the Centre for Health Informatics, University of NSW.