Medical errors are the eighth leading cause of death in the USA. They cause substantial morbidity and add up to $14.5 billion annually in direct healthcare costs in the USA.^1–4 Studies in other countries have yielded similarly concerning results.^5–7

Patient safety—defined as freedom from accidental injury⁴—has therefore become a major concern of healthcare providers, the general public, and policy makers. In the USA this heightened awareness has been driven in large part by a series of reports from the Institute of Medicine dealing with quality and safety.^4,8,9 A simple definition of error is the “failure of a planned action to be completed as intended or the use of a wrong plan to achieve an aim”.⁴

There is an emerging literature which shows that medical errors are a significant problem in neonatal intensive care.^10,11 Initial studies focused on medication errors, documenting potential adverse drug events occurring at a rate of eight times that of adult hospitalized patients.¹⁰ A recent study summarizing anonymous voluntary reports from 54 neonatal intensive care units (NICUs) participating in a Vermont Oxford Network quality improvement collaborative revealed large numbers of errors in virtually all domains of neonatal intensive care.¹²

Timely error detection including feedback to front line clinical staff is a prerequisite for focused improvement in patient safety.^13,14 Detection methods should not only capture preventable adverse events—defined as injury resulting from medical intervention that could have been prevented—but also failures or defects in the reliability of the system of care that place patients at risk for harm in the future.¹⁵

A number of error detection methods are currently in use by healthcare institutions including voluntary incident reporting, chart auditing (including “trigger tools” designed to detect specific types of events), and automated data mining of laboratory, pharmacy, and case mix data. While these methods have improved error detection, they tend to focus on detection of adverse events, are relatively insensitive, and are not well suited for routine monitoring of error prone points in diverse systems of care.^4,8,16–21

To facilitate compliance with safe practices—for example, patient identification, alarm settings, hand hygiene—most institutions also establish quality assurance monitoring procedures. This approach is becoming increasingly time consuming and burdensome as healthcare providers recognize the numerous safety hazards imbedded within patient care systems. Moreover, few quality assurance programs are designed to provide real time feed back to caregivers, a key to behavior change which allows personnel to become fully engaged in real time patient safety improvement.

To address these concerns, some quality and patient safety programs are beginning to experiment with safety methods borrowed from industry that may monitor error prone points in the system of care more efficiently and effectively.¹⁴ These methods include the use of checklists, continuous quality improvement, statistical process control, lean production, blame free reporting of near misses, root cause analysis of errors, and failure mode and effects analysis.^14,22–28

Random process audits

We were particularly interested in industrial methods that could potentially be applied directly by front line clinical staff in real time; methods that would permit monitoring of a broad range of errors without draining time and energy from the busy staff. Random process auditing, a remarkably intuitive and simple method used routinely in banking, the pharmaceutical industry, and high risk industries such as steel manufacturing, has many of these characteristics.^29,30

In contrast to system and product quality audits which are typically done for purposes of formal evaluation, process audits are mainly used to engage employees directly in continuous improvement efforts.^31,32 Rather than attempting to monitor all potential errors all the time, random process auditing systematically chooses a subset of error prone points to monitor at any given moment, thereby permitting meaningful coverage of complex systems over time.

Checklists of questions or review topics are compiled for each monitoring point to assure a systematic approach that is focused on important items. The process audit team randomly selects a checklist and then goes out to that point in the process to engage staff in an immediate review of the work in progress relative to the checkpoints. In this sense, audits are pre-planned and can be distinguished from the typical “management walk around” in which findings occur more serendipitously.³²

A further distinction is the constructive tone of the discussion. Tunner³³ describes the typical ground rules for process audits:

• results are not to be used to compare one area with another;

• audits should be part of the routine of work;

• they should be constructive, not destructive;

• they should use findings to drive improvement;

• they should never use findings in punitive ways; and

• findings should be openly shared and reviewed with all staff and management.

Because the discussion is occurring among front line staff in the work area and about work in progress, data are immediately available to the production (or healthcare) team, permitting prompt identification of the systems problem. Dominguez and Galarza³⁴ describe a typical application of process audits on the shop floor of Arrow Electronics, a manufacturer of cable assemblies. They note that these audits have resulted in immediate improvements such as updated standards, revised job descriptions, better training, processes changes, and new tooling and fixtures. In essence, through the use of random process audits, the front line team and management are continually engaged in the error proofing and improvement process.

Properly designed and implemented, a random safety audit can address many key elements of behavior change theory including audit and feedback, self-efficacy, social norms, and reinforcement.³⁵ It permits focused “just in time” education and reminders and provides an opportunity for opinion leaders and role models to motivate staff.

In this study we pilot tested a broad range of patient safety checks during routine multidisciplinary patient care activities as a first step in developing a robust real time random patient safety audit for use by clinicians in busy high risk healthcare settings. The objective of this pilot study was to determine the feasibility (whether audits were completed each day they were attempted and whether staff disclosed errors during routine daily work) and utility (whether the safety questions audited detected important errors) of real time safety auditing during routine clinical work in an ICU.

METHODS

Development of the patient safety audit

The Center for Patient Safety in Neonatal Intensive Care developed a 36 item patient safety audit using a modified Delphi technique.³⁶ Members of the Delphi group included experts in clinical neonatology, pediatrics, health services research, systems engineering, infection control, and advance practice nursing. Questions were formatted in a checklist and were refined iteratively by consensus based on the perceived potential clinical impact of mistakes or systems failures, or their perceived frequency. The checklist was then reviewed and refined with nursing leadership and physicians from the study NICU to ensure safety questions were relevant to this NICU. The checklist was not intended to be comprehensive for all safety or quality issues relevant to neonatal intensive care.

The audit questions were designed to detect a broad range of errors associated with care of patients in the NICU in real time, largely during routine patient care activities. By coupling error detection with daily patient care, NICU personnel were provided with concurrent reminders of critical patient safety practices.

The questions were divided into two categories. Category I (containing 22 of the 36 safety questions) generally evaluated for errors associated with: (1) delays in care, (2) equipment failure, (3) communication, and (4) laboratory/radiological studies. Category II (containing the remaining 14 safety questions) focused on evaluating compliance with hospital policy or guidelines.

The utility of real time safety auditing during routine clinical work was determined by counting the number of errors detected as well as any unit policy or guideline changes prompted by information gained from the audits. The feasibility of auditing was determined by the completion of auditing and staff disclosure of errors each day audits were attempted. In addition, the study team solicited feedback from NICU leadership (nursing and physician) regarding any concerns reported by clinical staff concerning safety auditing. Furthermore, NICU staff occasionally provided unsolicited subjective feedback to the research nurse concerning safety auditing.

Implementation of the safety audit

Safety audits were conducted for a total of 13 days during a 36 day period from January 28 through 4 March 2003 in a 20 bed tertiary care medical-surgical NICU with an average daily census of 19.5 patients. All data were recorded on standardized forms by the research nurse, an infection control professional. Each day the research nurse selected 5–7 items from category I for assessment, and all patients rounded on were evaluated for those items. Items were selected by the research nurse to allow each item to be evaluated on 4–10 different days during the study period. The clinical team did not know in advance which items were to be audited on a given day. The research nurse, who was not previously a part of the multidisciplinary team conducting morning work rounds, attended rounds with the team on days auditing occurred.

Morning work rounds usually began at 08.30 hours and lasted for approximately 2 hours. The following clinical staff attended morning work rounds: an attending neonatologist, neonatology fellow, neonatal nurse practitioner, a supervising “charge” nurse, the patient’s bedside nurse (who typically cares for 1–3 other patients depending on patient acuity), and a respiratory therapist. Rounds occurred at the patient’s bedside; the patient’s clinical course was reviewed, a plan of care was formulated or modified, and orders were written. The patient typically had been examined before rounds commenced. Family members of patients were occasionally present during rounds but were not directly queried concerning errors. The research nurse queried the clinical team regarding errors associated with any of the 5–7 questions being audited as they rounded on the patients. Errors were disclosed by members of the care team on a voluntary and non-punitive basis. They were documented on a standardized form by the research nurse.

After work rounds the research nurse spent approximately 2 hours directly evaluating patients and their medical record for errors associated with the 14 questions from category II. Two examples of these evaluations included determining if the patient’s identification band was located on the patient in accordance with hospital policy, and whether unlabeled syringes or medication bags were at the bedside. On average, a convenience sample of seven patients could be evaluated in this time. Evaluation of some patients was delayed or omitted because of clinical activity at the patient’s bedside. A convenience sample was used in evaluating these questions as it was less intrusive to clinical care, while allowing for rapid assessment of the utility of these questions to detect errors during the brief study period. Errors were documented in a study notebook by the research nurse. If NICU staff disclosed errors not related to the items being audited, these errors were also recorded in the study notebook.

This project was implemented by the research team with full support from NICU physician and nursing leadership in collaboration with multidisciplinary NICU bedside care teams. Following institutional policy and in collaboration with the institution’s quality improvement program, neither Institutional Review Board approval nor informed consent was required.

The research nurse frequently reassured clinicians that the goal of the project was to detect systems problems that contribute to errors in patient care, rather than to assign blame to individual caregivers. Further, clinicians were reminded of physician and nursing leadership’s support for a culture of “blame free” error reporting. All provider and patient identifiers obtained in the process of data collection were deleted before verbal presentations or preparation of summary reports.

Clinical staff commonly gave unsolicited feedback to the research nurse during or after work rounds concerning their impression of safety auditing during work rounds. The research nurse recorded these comments in the study notebook. The study coordinators solicited similar feedback from the physician and nursing leadership of the unit.

Data were entered in a Microsoft Excel database for descriptive analysis. Errors were tabulated and standardized to errors detected per 100 patient days.

RESULTS

Utility

The safety audits detected a total of 338 errors. These errors represented a broad spectrum of systems problems. Twenty seven of the 36 safety questions detected at least one error. The question concerning patient identification bands detected 70 errors, including use of a band from another hospital (4%), no band present (12%), and band not attached to the infant (75%).

For each question from category I (audited during work rounds), an average of 138 (range 78–195) patient evaluations occurred during the 13 days of auditing. Category I questions detected 35 errors including 17 associated with laboratory or radiology studies, nine associated with ineffective communication or delays in patient care, eight associated with medical devices, and one error associated with pain management (table 1).

For each question from category II (audited after work rounds by observation at the patient’s bedside supplemented by review of the medical record), an average of 63 patient evaluations occurred during the 13 days of auditing. Category II questions detected 260 errors associated with deviation from unit or hospital policy, and 43 errors associated with deviations from known safe practices (table 2). There was not a single day in which no errors were detected after work rounds.

Error detection most commonly occurred at the patient’s bedside, allowing immediate notification of clinical staff. In instances where this was not possible, appropriate NICU staff were made aware of the errors by the research nurse in a timely manner.

Apart from the immediate clinical interventions resulting from detection of an error (for example, ordering an x ray to confirm the location of a central venous catheter when its position had not been previously verified), several lasting interventions resulted from the use of the safety audits including a change in the patient identification system used in the study NICU and development of unit guidelines for pulse oximeter alarm settings (box 1).

Feasibility

Auditing was completed on all 13 days on which it was attempted. Clinical staff disclosed that errors occurred on all 13 days of auditing during work rounds. In addition to the 35 errors detected by the audit questions during rounds, on more than 17 occasions clinical staff approached the research nurse to report additional errors not evaluated by the 36 safety questions (table 3).

In auditing the 14 category II items after rounds, the research nurse could typically evaluate seven patients in a 2 hour time frame. These audits detected 303 errors during 13 days of auditing.

Only one concern of auditing was reported to the research nurse or to NICU leadership. Several clinical staff members reported that auditing 5–7 questions per patient during the work rounds was time consuming, occasionally disrupting the flow of rounds. Many staff expressed enthusiasm for continued auditing during work rounds provided that only one or two safety questions were addressed per patient. Many clinicians (nurses, nurse practitioners, and physicians) expressed interest in being a “safety auditor”.

DISCUSSION

This study demonstrates the potential feasibility of detecting errors in the system of patient care during routine daily work. Errors that are frequently difficult to detect by traditional surveillance methods—such as delays, communication problems, equipment failures, and non-standard clinical practices—were elicited by direct observation and stimulated recall of front line staff. While not designed to replace other error detection methods, this approach is far more structured than the voluntary “incident report” system that hospitals generally rely on and may be more sensitive, timely, and allow multidisciplinary participation by front line clinical staff in patient safety efforts.

Despite the brief duration of the study, errors were detected in virtually all of the safety checklist categories selected by the multidisciplinary expert group. Some care processes were found to be especially error prone, including important patient safety areas such as alarm settings, patient identification, hand hygiene, and labeling of tubing, syringes and medications.

Although it was not our primary intention to conduct a full scale qualitative assessment of staff attitudes regarding the random audit process, NICU providers were remarkably receptive and supportive. Reducing the number of questions audited during rounds was the only modification desired by clinical staff. This pilot is the first step in developing a streamlined random safety audit tool for use by front line clinical staff without the need for additional personnel. Further studies are underway with clinical staff performing one or two safety audits daily (to minimize the burden of time) during their routine clinical work.

Additional factors may have contributed to the success—even popularity—of the audit in this single institution. The design and implementation of the study involved close collaboration between the research team and NICU personnel. Perhaps the most important factor in the acceptance of the audit process was the immediate realization by clinical staff that the audits were identifying major remediable gaps in performance. This is in marked contrast to the common healthcare practice of collecting data without feeding back the results in real time, as commonly occurs with incident reporting systems.

This study took place in the context of strong institutional and NICU efforts to instil a non-punitive “culture of safety” in which reporting of errors is encouraged, as most errors are attributed to systems problems rather than individual fault.¹⁴

Nine of the questions detected no errors, and fewer errors were detected on work rounds than by direct observation outside the rounding process. This may suggest reticence of the staff to mention errors in an open forum. It is important to note that these questions may not have detected errors because of the apparent rarity of the event (such as ventilator malfunction), because the event would have been difficult for staff to observe (for example, adverse events occurring when an infant was away from the NICU), or because aggressive measures had previously been taken to reduce/prevent mistakes (for example, engineering of enteral feeding tube connections so that they cannot be inserted into parenteral tubes). However, most of the 36 items detected important errors and could serve as a basis for routine audits in the NICU environment.

Significant problems in the patient care system were generally corrected quickly when detected by the audit process. For example, patient misidentification is a common source of error in the NICU.¹² However, appropriate identification of an individual patient in a room full of babies requires reliable availability of identifiers, preferably attached directly to the patient. NICU patients offer a special challenge because of their extremely small size (some weigh only 500 g) and skin fragility. The patient safety audit revealed that an appropriate identification band was physically attached to the patient, in compliance with the institution’s policy, in only 9% of cases. Prompt purchase of a convenient non-traumatic band specially designed for neonates resulted in immediate improvement. Audits over the subsequent 16 months revealed continued compliance above 90% (data not presented).

The audit also revealed substantial problems with pulse oximeter alarm settings. In general, it is the practice in the study NICU to avoid oxygen saturations greater than 95% in very low birth weight infants receiving supplemental oxygen to reduce the risk of retinopathy of prematurity and chronic lung disease.³⁷ The audits showed inappropriately high oximeter alarm settings in 47% of infants. Although there is strong consensus among neonatologists at this institution that such high oxygen saturations are inappropriate, this opinion had never been translated into a policy or guideline or clinical practice for oximeter settings—a deficiency that was addressed as soon as the findings of the audit were known.

This pilot and feasibility study led to the important observation that audit items should not be “fixed in stone”. The audit checklist should be a flexible living vehicle for error detection and safety improvement. When problems have been addressed and repeat audits demonstrate compliance, it may be appropriate to audit these issues less frequently or to eliminate them from the audit entirely. Conversely, as new concerns arise, new audit queries can be added. Of course, different patient care settings require different safety questions, but the audit concept may be applicable to diverse clinical settings. Indeed, a similar safety audit process is now in place in this hospital’s medical intermediate care program.

Potential drawbacks of safety auditing during routine clinical work include fear of punishment or retribution for disclosing errors as well as embarrassment. It is conceivable that auditing could be disruptive to the rounding process under certain circumstances. It is also unclear how sensitive this auditing process is at detecting errors. We are performing a follow up study to formally assess staff attitudes regarding audits conducted during routine clinical work as well as to determine the sensitivity of the audit process in error detection.

Safety audits have the potential to increase safety awareness of clinical staff while providing prompt feedback regarding team performance in critical patient safety domains. Data derived from the audits can be entered directly into a database and trends followed over time, providing evidence of improvement and compliance with guidelines.

In conclusion, we have developed and pilot tested a novel real time patient safety audit system to detect errors and safety defects during routine clinical work. Safety auditing has the potential to reduce morbidity and mortality incurred by medical errors as this tool promptly detected significant problems that had not been appreciated previously, allowing for changes in policy and practice. A blame free “culture of patient safety,” as well as the identification of major remediable gaps in performance, facilitated acceptance by clinical staff.