Critical incidents and journey mapping as techniques to evaluate the impact of online evidence retrieval systems on health care delivery and patient outcomes

https://doi.org/10.1016/j.ijmedinf.2006.03.006Get rights and content

Abstract

Background

Online evidence retrieval systems are a potential tool in supporting evidence-based practice. Effective and tested techniques for assessing the impact of these systems on care delivery and patient outcomes are limited.

Purpose and methods

In this study we applied the critical incident (CI) and journey mapping (JM) techniques to assess the integration of an online evidence system into everyday clinical practice and its impact on decision making and patient care. To elicit incidents semi-structured interviews were conducted with 29 clinicians (13 hospital physician specialists, 16 clinical nurse consultants (CNCs)) who were experienced users of the online evidence system. Clinicians were also asked questions about how they had first used the system and how their use and experiences had changed over time. These narrative accounts were then mapped and scored using the journey mapping technique.

Results

Clinicians generated 85 critical incidents. Three categories of impact were identified: impact on clinical practice, impact on individual clinicians and impact on colleagues through the dissemination of information gained from the online evidence system. One quarter of these included specific examples of system use leading to improvements in patient care. Clinicians obtained an average journey mapping score of 22 out of a possible score of 36, demonstrating a good level of system integration. Average scores of doctors and CNCs were similar. However individuals with the same scores often had very different journeys in system integration.

Conclusions

The CI technique provided clear examples of the way in which system use had influenced practice and care delivery. The JM technique was found to be a useful method for providing a quantification of the different ways and extent to which, clinicians had integrated system use into practice, and insights into how system use can influence organisational culture. The development of the journey mapping stages provides a structure by which the program logic of a clinical information system and its desired outcomes can be made explicit and be based upon users’ experiences in everyday practice. Further work is required using this technique to assess its value as an evaluation method.

Introduction

Online evidence retrieval systems provide health professionals with fast and easy access to a wide range of information resources to inform their decision-making processes. Provision of online evidence systems to support clinical work at the point-of-care has been adopted as a strategy to support evidence-based practice in the UK, USA and Australia [1], [2], [3]. Few evaluations of their effectiveness have been reported [4]. Most studies have focused on assessing frequency of use rather than effectiveness and greater attention has been placed upon system use in family medicine rather than in hospitals [2], [4]. Measuring the impact of an information system on clinical practice is problematic. There are many potentially confounding variables for which it is often not possible to control. Thus establishing the relationship between online evidence system use and changes in clinical practice and patient outcomes is difficult. One of the largest studies to attempt this task was conducted by Lindberg et al. [5], who applied an adapted critical incident technique to gather reports from 552 US physicians about when they had found MEDLINE searches to be especially helpful (or not) in carrying out their professional activities. Physicians reported using MEDLINE for a range of clinical tasks, from choosing diagnostic tests to developing treatment plans. In 2% of instances of MEDLINE use physicians reported finding essential information that led to saving patients’ lives or sparing limbs and organs.

The introduction of MEDLINE into clinical settings in a US hospital was evaluated through interviews with a random sample of physicians who participated in MEDLINE training. Physicians reported that 47% of searches influenced their clinical decisions, most frequently through confirmation of a decision [6]. Another study found that length of stay was shorter for patients for whom a MEDLINE search had been conducted earlier in their hospitalisation. A causal relationship could not be established and a number of confounding factors such as the severity of patients’ conditions limited interpretation of the findings [7]. Laboratory studies have shown that answers to clinical questions are improved when participants are given access to online evidence retrieval systems [8], [9], [10].

In 1997 the state of New South Wales, Australia implemented the Clinical Information Access Program (CIAP) (http://www.ciap.health.nsw.gov.au) a website providing around 55,000 clinicians (doctors, nurses and allied health professionals) with 24 h access to evidence at the point-of-care in public health care facilities. A 2-year evaluation was undertaken 5 years after system implementation to assess the extent to which the system was used, supported clinical decision-making and resulted in improvements to patient care. A triangulated research program consisted of four stages of data collection. Results from the first three stages which investigated how frequently and why clinicians used the system have been reported elsewhere and are summarised below [11], [12], [13], [14], [15], [16], [17].

An analysis of 7 months of CIAP web-logs of the 55,000 clinicians who have access allowed a quantification of the rates of use by geographical Area Health Services, hospitals and professional groups. Considerable variation in the rates of use for individual hospitals and professional groups was found [15]. Medical staff used CIAP at double the rate of nursing staff. Results showed that use of CIAP was related to patient care decisions. For example, searching activity was highly positively correlated with patient admissions at individual hospitals. The wide variation in use required further investigation to understand factors related to high and low use organisations. Stage 2 sought to address these issues.

In-depth case studies investigated the influence of professional and organisational factors on health professionals’ use of CIAP. Results from focus groups, interviews and surveys from clinical groups within three case study sites identified a range of cultural, organisational and team factors which were important in explaining variations in CIAP use within and between professional groups and hospitals [11], [12]. Positive promotion of CIAP, and support and encouragement to use it were major factors influencing use. CIAP was used by all three professional groups. Nurses and allied health staff reported needing most support in the effective retrieval of information. Doctors were the highest users and most likely to use CIAP for patient care. Nurses’ awareness of CIAP was the lowest of the three groups.

Technical issues were also highlighted but were found not to be the central factor influencing CIAP use. Investigation of the association between clinical team functioning and use of CIAP to improve patient care produced interesting results, demonstrating that members of well-functioning clinical teams were more likely to report effective use of CIAP. However clinical team functioning was not associated with greater awareness of CIAP [11].

A statewide survey of 5511 clinicians (doctors, nurses and allied health professionals) from a random selection of 65 hospitals was undertaken to determine awareness and use of CIAP, how and why clinicians use, or do not use, online evidence, technical issues related to use of CIAP, and clinicians’ perceptions of its impact on clinical care. The survey showed that 63% of clinicians had heard of CIAP and of those who had, 75% had used it [13]. Colleagues were the primary channel via which clinicians heard about CIAP. The survey results reinforced the key findings of the log analysis and in-depth cases study findings. For example, clinicians’ actual and reported use of CIAP was consistent in terms of types of resources used. Clinicians’ views about the importance of organisational factors in supporting an evidence-based approach to care and the integrated use of CIAP, reported during the focus groups in stage 2, were reflected in the survey responses from the respondents. The survey identified high levels of satisfaction with the technical features of CIAP, such as ease of use and search speed. Analysis of the log data indicated that use was related to direct patient care decisions and the survey results reinforced this with, for example, 55% of doctors reporting that they had had direct experience of CIAP use resulting in improved patient care. There were significant differences between professional groups [14], [17] and also between senior and junior clinicians [16], [17] within the same profession. For example, senior and specialist nurses had more positive attitudes to use, perceived more support and legitimation for its use, and reported more frequent use than did junior nurses [17].

The results from stages 1–3 provided the context and direction for the last stage of the evaluation reported here which sought to investigate two techniques, the critical incident (CI) [18] and journey mapping (JM) [19] techniques, to assess clinicians’ integration of the use of CIAP and impact on clinical decision-making and patient care. These methods create a narrative of change from the perspective of system users allowing an examination of professional and organisational cultural changes and how users experience these changes.

The CI and JM techniques complement traditional approaches of questionnaires and interviews frequently used to assess the impact of clinical information systems on care delivery and outcomes. Both techniques incorporate the collection of rich narrative accounts along with some quantitative measure of the results obtained. The CI technique involves obtaining direct reports from system users about the ways in which use of the online evidence retrieval system had been useful (or not) to their clinical work. JM uses narrative accounts of clinicians’ introduction to, use of, and experiences with, the online evidence retrieval system. A structured numerical coding system, which reflects the level to which individuals have integrated the system and achieved improvements in care delivery, is then applied to these accounts. Our aim was to assess how useful these techniques were in measuring and understanding the impact of online evidence system use on care delivery and patient outcomes.

Section snippets

Methods

Sixteen clinical nurse specialists (CNS) and 13 hospital-based specialist physicians were selected who were experienced and regular CIAP users. We used a purposive sample as our aim was to examine what could be achieved when the online evidence system was used frequently and by experienced users in order to provide an indication of what outcomes were possible. We hypothesized that if we were unable to measure a positive impact of system use among this population it would be unlikely that the

Results

In total, 85 critical incidents were described by 29 clinicians interviewed. The average number was 2.9 per clinician (range 1–5). Three main themes emerged from the critical incident data, namely impact of CIAP use on: (1) clinical practice, (2) individual clinicians and (3) colleagues via the dissemination of information gained from CIAP.

Discussion

The data from the critical incidents provided an enhanced understanding of how the online evidence system was being used in clinical practice and reinforced and elaborated upon the information gained from other stages of the evaluation [11], [12], [14], [15], [17]. Clinicians were able to relate clear stories of how CIAP use had impacted on their own clinical practice and patient care. There were some differences between the types of clinical impacts described by doctors and CNCs, in line with

Conclusions

Our study has shown that an online evidence system assists clinicians in their work across a range of clinical activities. Interviewing clinicians about their experiences using the CI technique elicited detailed accounts of the positive impact of system use on care provision, and patient outcomes, as well as on clinicians’ confidence in decision making. The JM technique provides a useful method for quantifying the impact of system use on clinical practice and potentially for monitoring changes

Acknowledgement

This research was funded by a grant from the NSW Health Department.

References (24)

  • M. Klein et al.

    Effect of online literature searching on length of stay and patient care costs

    Acad. Med.

    (1994)
  • J. Westbrook et al.

    The impact of an online evidence system on confidence in decision making in a controlled setting

    Med. Dec. Mak.

    (2005)
  • Cited by (0)

    1

    Dr. Gosling undertook this research while she was a Research Fellow at the Centre for Health Informatics, University of NSW.

    View full text