Original contribution
The association between operator confidence and accuracy of ultrasonography performed by novice emergency physicians

https://doi.org/10.1016/j.jemermed.2005.02.008Get rights and content

Abstract

The variable accuracy of emergency department (ED) ultrasound described in the literature has limited its utility as the sole imaging modality in critical decision making. Although ultrasound accuracy is highly dependent upon the technical abilities of the operator and conditions unique to each patient, no previous study of ED ultrasound has included estimates of operator confidence. This prospective observational study explores the association between operator confidence and the accuracy of ED ultrasound. Ultrasound was not performed in our ED until a formal training module was instituted. Patients were enrolled prospectively for the first year following the training module if they underwent one of the following ultrasound studies: abdominal examination for intraperitoneal fluid, right upper quadrant examination for gallstones, renal examination for hydronephrosis, pelvic examination for intrauterine pregnancy, abdominal examination for aorta diameter > 3 cm, or cardiac examination for pericardial fluid. In addition, formal ultrasound, computed tomography, magnetic resonance imaging, or an invasive procedure was required as a “gold standard” for each patient. Operators recorded their interpretation of the ED ultrasound and rated their confidence with the analysis before the formal imaging study or procedure. Test performance characteristics for each examination type and for all studies together were determined. The association between operator confidence and accuracy was explored using logistic regression and by determining test performance characteristics with patients stratified by confidence value. A total of 276 ED ultrasound studies were included. There were no significant differences in accuracy between ED attendings and residents. Overall accuracy, sensitivity, specificity, LR+, and LR− were 90%, 92%, 86%, 6.8, and 0.09, respectively. With confidence scores of 9 or 10 (n = 113), these values improved to 96%, 99%, 90%, 9.6, and 0.01, respectively. Logistic regression revealed an association between confidence and ED ultrasound accuracy (p < 0.001). It is concluded that a significant association exists between operator confidence and the accuracy of ED ultrasound. High confidence values are associated with clinically useful test performance characteristics.

Introduction

Ultrasonography has become standard practice in many emergency departments (EDs) throughout the country, with most residency programs integrating ultrasound training into their curriculum (1, 2, 3, 4). Advantages to ED ultrasound include the rapid diagnosis of life-threatening illnesses and decreased lengths of stay in the ED (5, 6, 7, 8, 9). In addition, ultrasound can be used in place of more invasive procedures, such as diagnostic peritoneal lavage in the initial evaluation of unstable multiple trauma victims, and can assist in the placement of central venous access catheters (10, 11, 12, 13, 14, 15). Despite the successful integration of ultrasound into the routine practice of emergency medicine, the reported test performance characteristics have generally been inadequate for making critical decisions without formal diagnostic imaging (3, 6, 7, 8, 9, 16, 17, 18). As a result, the role of ED ultrasound in clinical practice remains unclear.

Unlike formal radiologist readings for computed tomography (CT), magnetic resonance imaging (MRI), and ultrasound studies, which are generally considered “definitive” for the purpose of clinical decision-making, interpretation of ED ultrasound may be less reliable for a variety of reasons. First, an emergency physician (EP) generally has considerably less experience than an ultrasound technician in performing ultrasonography or a radiologist in interpreting these studies. In addition, ED ultrasound is highly dependent upon the technical abilities of the ultrasonographer as well as the physical characteristics of the patient and the patient’s ability to cooperate with the examination (15). All of these have led to lower overall reported values for sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) (3, 6, 7, 8, 9, 16, 17, 18). On the other hand, the ED ultrasonographer is in the unique position of being aware of his or her own experience as well as the technical limitations associated with a particular examination. Thus, operator confidence with the study interpretation should be considered when deciding whether to rely exclusively on the ED ultrasound, but has not been investigated previously. We hypothesized that an association exists between operator confidence and ED ultrasound accuracy and that use of an appropriate threshold confidence value could result in clinically useful test performance characteristics.

Section snippets

Study Design

This was a prospective, observational study. Waiver of consent was granted by our investigational review board.

Study Setting and Population

This study was conducted in an urban university hospital with approximately 40,000 annual ED visits. Ultrasound was not performed in our ED before the trial period. Thus, the participating physicians were novice sonographers.

Study Protocol

A departmental training program was instituted before the trial, with ED personnel required to be “certified” by completing this program to perform ultrasound.

Results

In the 12-month study period, a total of 276 eligible ultrasound studies were submitted by emergency physicians for inclusion in the trial. A total of 12/14 full-time attendings performed 54% of studies, and 16/18 residents performed the remaining 46%. This included residents from the second, third, and fourth postgraduate years. None of the participants had substantial prior experience with ED ultrasound. Examination number frequencies are displayed in Table 1.

The total number, percent

Discussion

We document our experience over the first 12 months after the introduction of ED ultrasound in our institution. Although our overall test performance characteristics are similar to previous reports, this analysis is unique for several reasons. We employed a “gold standard” that required every eligible patient to undergo formal ultrasound, CT, MRI, or an invasive procedure. More importantly, we explored the relationship between operator confidence and the accuracy of ED ultrasound using multiple

Conclusions

Previous studies evaluating the accuracy of ED ultrasound have failed to consider operator confidence values despite the technical challenges involved in this procedure and the unique position of the sonographer to judge the technical adequacy of an individual study. Here we document our experience with ED ultrasound over the first 12 months after institution of an ultrasound program. Although our overall test performance characteristics are similar to previous reports, the integration of

Cited by (0)

View full text