Evaluation in the design of health information systems: application of approaches emerging from usability engineering

Abstract

This paper examines the role of evaluation in the design of health care information systems. A framework is presented for considering evaluation in the context of software development processes, in particular, the systems development life cycle (SDLC). Variations on standard design methodologies are then discussed, including methods based on rapid development and continual evaluation of prototype systems. Usability testing is presented as a key method for conducting evaluations during iterative system development. The emergence of design methodologies, where evaluation is viewed as a central part of the development cycle is also discussed. Evaluation methodologies are then considered along a continuum, ranging from studies involving a high degree of experimental control to observational approaches. A full cycle approach to evaluation of health care systems is argued for, involving deployment of new methods across the SDLC. Implications for future work exploring the integration of design and evaluation processes in health informatics are discussed.

Introduction

In the field of health informatics, summative evaluations have been conducted to evaluate the impact of a wide variety of systems in clinics and other health care environments. Numerous evaluations have assessed the effects of health information systems on dependent measures such as health outcomes, length of stay, cost-effectiveness, quality of care and other measures [1]. These studies typically apply randomized clinical control trials methodologies. Such evaluations are analogous to the testing of drugs and other discrete health care interventions where there is a clearly defined independent variable (e.g., presence or absence of a drug, or in the case of health informatics studies, presence or absence of an information system). Although these types of evaluations are necessary in order to ensure that systems that are developed are both safe and effective, greater emphasis is needed to ensure that the process of system design is effective. In addition, it can be argued that we will not understand the full meaning of outcomes we obtain from the use of information technology, unless we can understand outcomes in terms of the design decisions that lead to the outcomes.

A wide range of methodologies have been developed in the general software industry for providing frameworks for designing information systems. Life cycle models provide a foundation for guiding the development and evaluation of complex systems [2], [3]. Traditional system development life cycles (SDLC) presuppose a set of fixed stages for system development, with evaluation of the system predominantly occurring in the final stages. Such approaches have proven difficult to successfully apply in health care where information needs may be hard to precisely determine. The possible reasons for this are that the health care environment is often complex and characterized by missing information, shifting goals and a great deal of uncertainty. Health care decision making processes are complex, poorly understood and consequently difficult to model effectively in the analysis of the SDLC. Health care decisions are subject to a level of uncertainty not found in traditional business environments and consequently health care technology and the knowledge on which it is based is often very volatile. In fact, even before decision making processes are understood, they may change within the time span of the traditional SDLC.

In recent years, a number of software engineering methodologies have been developed that focus upon deploying evaluation methods throughout the software life cycle—from an initial needs analysis through to design and implementation, in addition to summative evaluation conducted upon completion of the system. For purposes of developing an evaluation framework this paper will initially consider methodologies in terms of the traditional SDLC, followed by methods involving rapid iterative development, based on a usability engineering approach. The traditional SDLC is characterized by the following phases: (1) planning, (2) analysis, (3) design, (4) implementation, and (5) maintenance/support [2], [3]. In this model each phase consists of similar activities that are somewhat sequential, with one phase typically being completed prior to beginning the next phase.

A wide range of iterative design methods have appeared, where the distinction between the phases has become blurred and the role of continual evaluation is emphasized. For example, methodologies such as rapid application development (RAD) and various approaches to prototyping have become increasingly popular as the basis for system development [3]. Using such approaches, system requirements are quickly converted to a prototype system that is iteratively revised, based on evaluations, until an acceptable product is developed and the system is completed [3]. Continual evaluation is a central component of these design methodologies. Along these lines, a greater emphasis on iterative formative evaluation, closely integrated with the design and development process, has been called for in the health care information technology industry [4]. In conjunction with this, deployment of a wider range of methods for acquiring information relevant to design in health informatics evaluations is also argued for.