You are here

Article Text

Article menu
Original research
Assessing distractors and teamwork during surgery: developing an event-based method for direct observation
  1. Julia C Seelandt1,
  2. Franziska Tschan1,
  3. Sandra Keller1,
  4. Guido Beldi3,
  5. Nadja Jenni1,
  6. Anita Kurmann3,
  7. Daniel Candinas3,
  8. Norbert K Semmer2
  1. 1University of Neuchâtel, Institute for Work and Organizational Psychology, Neuchâtel, Switzerland
  2. 2University of Bern, Institute for Psychology, Bern, Switzerland
  3. 3Department of Visceral Surgery and Medicine, University Hospital of Bern, Bern, Switzerland
  1. Correspondence to Professor Franziska Tschan, University of Neuchâtel, Rue Emile Argand 11, Neuchâtel 2000, Switzerland; Franziska.tschan{at}unine.ch

Abstract

Objective To develop a behavioural observation method to simultaneously assess distractors and communication/teamwork during surgical procedures through direct, on-site observations; to establish the reliability of the method for long (>3 h) procedures.

Methods Observational categories for an event-based coding system were developed based on expert interviews, observations and a literature review. Using Cohen's κ and the intraclass correlation coefficient, interobserver agreement was assessed for 29 procedures. Agreement was calculated for the entire surgery, and for the 1st hour. In addition, interobserver agreement was assessed between two tired observers and between a tired and a non-tired observer after 3 h of surgery.

Results The observational system has five codes for distractors (door openings, noise distractors, technical distractors, side conversations and interruptions), eight codes for communication/teamwork (case-relevant communication, teaching, leadership, problem solving, case-irrelevant communication, laughter, tension and communication with external visitors) and five contextual codes (incision, last stitch, personnel changes in the sterile team, location changes around the table and incidents). Based on 5-min intervals, Cohen's κ was good to excellent for distractors (0.74–0.98) and for communication/teamwork (0.70–1). Based on frequency counts, intraclass correlation coefficient was excellent for distractors (0.86–0.99) and good to excellent for communication/teamwork (0.45–0.99). After 3 h of surgery, Cohen's κ was 0.78–0.93 for distractors, and 0.79–1 for communication/teamwork.

Discussion The observational method developed allows a single observer to simultaneously assess distractors and communication/teamwork. Even for long procedures, high interobserver agreement can be achieved. Data collected with this method allow for investigating separate or combined effects of distractions and communication/teamwork on surgical performance and patient outcomes.

  • Human Factors
  • Teamwork
  • Communication
  • Surgery

https://doi.org/10.1136/bmjqs-2014-002860

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    It is increasingly accepted that human factors play an important role in surgical performance, and more research is needed to assess their influence.1–4 The most often discussed human factors in surgery are distractions in the operating room and intrasurgical teamwork.5–9 Although distractions and teamwork are recognised as important influences, they have rarely been assessed simultaneously. Observational methods exist to study either teamwork or distractions, but to our knowledge, there is currently no established method that allows a single observer to simultaneously assess both aspects. Furthermore, most observational studies have assessed relatively short surgeries. Because long surgeries bear higher risks for patient complications10 ,11 it is important to include long procedures in human factor research, and this inclusion may require the development of new methods. To address these gaps, we developed Simultaneous Observation of Distractions and Communication in the Operating Room (SO-DIC-OR), an event-based behavioural observation method that can be used in the operating room (OR). This method simultaneously captures distractors and teamwork and can be used to observe short and long procedures.

    We first provide a short introduction into the characteristics of different approaches to behavioural observation in OR settings, as well as their advantages and disadvantages. We then present the development of the observational method and provide information about interobserver reliability, including reliability after 3 h of continuous observation.

    Distractors and teamwork in the OR

    There are many potential sources of distractions in the OR (eg, noise from machines and manipulations, alarms, incoming phone calls or conversations outside the sterile team). Distractions are very common: even for short procedures, a distraction occurs every 1–3 min.12–15 Distractions have been found to negatively affect surgical performance,16–18 as they threaten the concentration of the surgical team members, particularly the concentration of less experienced surgeons.18–20

    Teamwork and communication in the OR are another important influence on surgical quality.21–25 Surgeons, nurses and anaesthetists have to cooperate closely and effectively; this requires complex collaboration.26 Good teamwork and optimal communication in the OR increase the quality of surgeries, whereas poor or ineffective communication jeopardises patient outcomes.26–29

    Methods for observing behaviour in the OR

    From a research perspective, the gold standard for investigating the relationship among distractions, teamwork and surgical outcomes is behavioural observation. Behavioural observation does not rely on self-reports or on retrospective analyses. Retrospective analyses are based on memory processes, which may contain errors30 and can be biased,1 particularly if the outcome is known.31

    Behaviours during surgical procedures can be observed based on videos32–34 or by direct observation in the OR. Although videotaping has many advantages,35–37 legal and ethical issues and technological constraints often limit filming in the OR. Therefore, much research in this field still relies on observers present in the OR.

    Direct observation presents several challenges. First, observers have to record behaviour and events as they happen,35 which requires a high degree of concentration and constant attention. Attention is limited, therefore, a single observer can only assess a limited number of different behaviours. Second, observers have to make fast and immutable decisions during the ongoing process. There is a risk that observers miss or misinterpret behaviours. Third, if the observation time is long, fatigue can influence the quality of the observation. It is thus crucial to develop methods that are well suited for direct observations and to show that high interobserver agreement can be achieved.38

    Before developing a new observational system, it is useful to assess existing methods, as it is an advantage to use an established methodology. Our literature search revealed that observational methods exist to assess teamwork,6 ,39–42 communication27 ,43–47 and distractors12 ,15 ,48 in the OR (for an overview, see web appendix). We found only two papers that combine observations of teamwork and distractions. One of these studies used two very different methodological approaches,14 and the other one limited observed distractors to a few categories49; none of the studies reported interobserver agreements.

    One of the reasons that observations of teamwork and distractors have not yet been combined may lie in the different methodological approaches that have traditionally been used in this field. Research groups observing communication in the OR have most often used field notes27 ,36 ,43 ,50; research groups assessing the quality of teamwork and non-technical skills have often relied on behavioural markers7 ,9 ,51–53; and research on distractions in the OR has most often used event-coding methodology.12 ,13 The following section compares these three approaches (table 1).

    View this table:
    Table 1

    Comparison of field notes, behavioural markers and event-based observations for observation in the OR

    Most common in ethnographic research,54 field notes have been used in studies assessing communication27 ,43 and leadership47 in the OR. Observers take extensive notes in a free text form.55–57 In addition to a general thematic focus, observers normally have few restrictions in terms of what they observe and take notes on. The advantage of field notes is that they can be flexibly used in almost every situation. When using this methodology, researchers should well understand the situation they observe; otherwise, they may overlook or misinterpret important events. The use of field notes is most appropriate if it is difficult to define behavioural categories in advance, which generally occurs when little is known about a situation47 and when the situations observed are very diverse. Because they allow a wide angle on a situation, field notes are well adapted for observing non-routine situations and are particularly useful for explorative studies.47 Field notes are often the basis for qualitative analyses. It is, however, possible to code and categorise field note contents after the observation, which allows the derivation of quantitative data,27 although to a limited degree. The disadvantage of field notes is that they cannot easily be used for quantitative research, and it is difficult to assess interobserver reliability for the initial taking of field notes.

    Most systems that assess the quality of teamwork in the OR use behavioural marker methodology.6 ,39 ,40 When using behavioural marker methodology, the observers are instructed to assess ‘behaviour classes’. These classes are defined in advance, based on a thorough analysis of the non-technical skills required for a specific situation or professional role.58 Within behavioural classes, exemplar behaviours that represent good or poor behaviours are defined.7 ,59 An example is the ‘exchanging information’ behavioural class within the Non-Technical Skills for Surgeons observational system.60 Optimal information exchange is described as ‘talk about the progress of the operation’, whereas poor information exchange is described as ‘fail to communicate concerns with others’. (ref. 61, p.17) In behavioural marker-based observations, observers do not report or note single behaviours; they instead provide an integrative quality score for each behaviour class for the whole procedure62 or for a predefined observational period.

    Behavioural marker-based systems have to be specific to the role or the situation. Methods have been developed for non-technical skills of surgeons,60 anaesthetists,63 scrub nurses52 and the entire surgical team.64

    The advantage of behavioural marker systems is that they focus on desired and undesired behaviour in a specific situation, and that observers provide a summary score. It is thus possible to assess the quality of teamwork and to provide immediate feedback after the observation. The disadvantage is that such integrative judgements are vulnerable to hindsight effects and observer biases.62 Observers need to (A) continuously assess the quality of behaviours, (B) relate these behaviours to the predefined classes and (C) mentally integrate their observations into an overall qualitative judgement for each behaviour class. This complex and cognitively demanding process requires extensive training and domain-specific knowledge.65 It is thus rather challenging to achieve high interobserver agreement.62 ,64 ,66 ,67 If teamwork quality is only assessed once using behavioural markers during the entire intraoperative phase, this approach may have limited usefulness in long surgeries, as different phases of the surgery have different coordination requirements, and as the quality of teamwork may not be consistent for the whole procedure.68 ,69

    Event coding is the continuous real-time observation and registration of specific, predefined events or behaviours. This methodology has been used to observe communication in the OR,39 ,46 but it is most common for assessing distractors in the OR,12 ,15 ,49 Some examples of observed events are ‘door to the OR opens’ or ‘an alarm sounds’. Observers note events as they happen. Event coding can be as simple as keeping a tally; more complex methods use time codes (ie, noting the event as well as the time it occurs). To develop an event-coding system, researchers define specific behaviours or events to be observed based on conceptual considerations. Each event is defined and described in a coding manual.70 For example, Healey and colleagues12 coded ‘case irrelevant communication’ as a distractor each time the team engaged in communication that was not related to the patient or the procedure; they coded ‘Monitor-F’ each time someone moved in front of the video display monitor in the OR (ref. 12, p.596). Event coding requires extensive observer training.70 ,71

    The advantage of event coding is that observed events and behaviours are specific. The clear definition of events requires little integrative judgement from observers; thus, observers can simultaneously chart several categories.35 If events are time-stamped, event coding allows to assess frequency, timing and sequences of events; it is thus well suited for detecting behavioural patterns.70 In addition, event coding allows for analyses and comparisons of different phases of a surgery.72 The disadvantage of event coding is that only predefined behaviours are captured; thus, some methods combine event coding with the possibility of providing open comments.49 Event coding is of limited use for immediate feedback, as it does not provide an integrative quality score.

    There is no a priori advantage for one particular observational method; method choice depends on the specific research goal. Nevertheless, in the OR setting, researchers have traditionally chosen different methods to observe communication, teamwork or distraction. Each method requires the observer to focus his or her attention on different aspects: Field notes require attention to the whole process and to its narrative structure; behavioural markers require the observer to make quality assessments by mentally integrating specific observations into overall judgements; and event coding requires attention to the occurrence of single events. Because different observation methods require different attentional foci, it is difficult to combine two existing methodological approaches.

    The web appendix provides an overview of the observational methods used for direct observation in the OR. We included papers focusing on the presentation of an observational method and papers focusing on specific content that also provide information about the observational system in the methods section. We excluded methods that focus solely on adverse events73 and papers based on subsamples from earlier published research. We also excluded systems focusing on anaesthesia (ie, Behavioural Marker System for Anaesthetists’ Non-Technical Skills63) or on the preoperative or postoperative phases.74 The overview contains information on method type, observed behaviour or events, observers, observation targets, and procedure type and duration. In addition, if provided, information on interobserver reliability is presented.

    The review of existing systems revealed two gaps that we aim to address with this paper. First, there is no observational method that combines the assessment of communication/teamwork and distractors as potential influences on the surgical process and has been shown to be reliable. Existing behavioural marker systems and event coding systems each require a different attentional focus from observers, thus, combining two existing methods would overburden observers and most likely result in low interobserver agreement. Second, most current knowledge with regards to communication/teamwork and distractors during surgeries is based on short procedures. An observational system that is suitable for observing procedures that last several hours allows extending research to procedures that bear a particularly high risk of patient complications. We thus developed an event-based coding system that allows to simultaneously assess communication and distractors in the OR and can be used for short and long procedures.

    Research goals

    We address the following research questions:

    • Q1: Is it possible to reliably assess distractions and aspects of teamwork simultaneously during surgery using an event-sampling methodology?

    • Q2: Is the observational method suitable for the observation of long procedures (3 h or longer) by maintaining acceptable interobserver reliability over time?

    Methods

    Sample

    The sample consisted of 29 elective open abdominal procedures that were entirely or partially observed by two observers. These surgeries were a subsample of 103 procedures observed over a period of 12 months at an university hospital in a western European country. General inclusion criteria for observed surgeries were elective open abdominal surgery and the observers’ availability. Throughout the study period, about every fourth procedure was observed by two researchers to assess interobserver reliability; these 29 observations are included in this study. The 29 procedures related to the digestive tract, intestines, rectum, liver, pancreas and oesophagus. There were major liver resections and minor liver resections (ie, resections of less than three liver segments); surgeries of the duodenum/pancreas, including duodenopancreatectomies and segmental duodenectomies; procedures related to the upper gastrointestinal tract, including gastrectomies (total or partial), oesophagectomies (including transhiatal) and hiatoplasties; endocrine procedures, including adrenalectomies; procedures related to the lower gastrointestinal tract, including hemicolectomies (right or left) and resections of enterocutaeneous fistula; and spleen surgeries, including splenectomies. These surgeries are representative of the surgeries performed in the department where the study took place; they were carried out in two identically designed and equipped ORs.

    Ten trained observers with at least a bachelor's degree in industrial psychology participated in the study. The local institutional review board approved the study.

    Procedure

    Development of the observational system

    Our main goal was to develop and test an observational system to assess distractors and aspects of teamwork during surgery (SO-DIC-OR). Each observational method has to satisfy the validity criteria (ie, the method measures what it is supposed to measure; thus, the observational categories have to be meaningful and adapted to the situation) and reliability criteria (ie, the observations must be consistent across observers and over time; thus, interobserver agreement has to be established).

    To satisfy the criterion of construct validity, we developed a list of events to be observed based on expert interviews, observations of five surgical procedures, and a literature review (figure 1). We performed seven in-depth expert interviews with senior and junior surgeons, anaesthetists, scrub nurses and circulating nurses about their perceptions of potential sources of distractions during the intraoperative phase and their assessment of helpful and problematic communication and teamwork in the OR. Using a guided field-note method (ie, instructing observers to concentrate on teamwork, communication and distractors), we observed five open abdominal procedures. The field notes were reviewed to extract observational categories. We also conducted an extensive literature search on observational systems already in use in the OR (see web appendix). Unsurprisingly, the behaviours that were mentioned in expert interviews, extracted from field notes and described in the literature, largely overlapped. Two observers tested a first version of the observational system during eight surgical procedures; they were advised to write comments on the coded events. After each surgery, the observers compared their observations event by event, and differences were discussed. Code definitions and descriptions were revised, and the final system was developed (table 2).

    View this table:
    Table 2

    SO-DIC-OR event codes, short descriptions and interobserver reliabilities

    Figure 1
    Figure 1

    Development process of the observational system.

    We chose a timed-event sampling methodology (ie, recording the event and the specific time at which the event occurs) for several reasons. First, clearly defining events and behaviours to observe does not require observers to make integrative judgements over time. Therefore, the system is cognitively less demanding than behavioural marker methodology, allowing the inclusion of more observational categories without overburdening observers. Second, for long procedures, an overall integrative assessment, as is customary in behavioural marker-based observations, is very difficult to make. Furthermore, and event-based system allows to assess teamwork quality separately for the different phases of a procedure, and allows for analysing sequential patterns; therefore, it is particularly suitable for the observation of long procedures.

    The observational system contains five distraction-related codes: door openings, noise distractors, technical distractors, side conversations and interruptions; these are largely based on the system developed by Healey and colleagues12 and were adapted for open procedures. The system contains eight teamwork-related codes that focus on communication within the sterile team and between sterile team members and anaesthetists. The observational codes include case-relevant communication (ie, short-term planning), teaching, leadership and problem solving. These codes are related to the patient and procedure (ie, task-related communication). We included task-related communication because it helps a team build and maintain a shared understanding of the task and may thus facilitate coordination.3 ,75 ,76 We also included case-irrelevant communication (eg, laughter and tension) because they represent social aspects of teamwork and may influence team building and team climate in the OR.41 For this reason, case-irrelevant communication within the sterile team is considered a teamwork code, not a distractor as in other systems.12 In addition, note that talking among anaesthetists or among circulating nurses/visitors is coded as side conversation and categorised as a potential distractor for the sterile team, despite the fact that these conversations could be case-relevant. The observational system also contains several contextual codes (eg, time of incision, time of the last stitch, personnel changes within the sterile team and personnel location changes around the operating table). Unusual incidents (eg, X-ray after an inconclusive sponge count) are described using an open-text option and ‘other’ code. The open-text option allows observers to describe any observation that is not covered by the predefined event codes but that they regard as important or interesting. Table 2 presents the codes and a short description of each code (a full codebook is available on request). Codes are entered into a laptop using a spreadsheet (Microsoft Excel); a macro is used to automatically time-stamp each event the moment it is entered. Observations started at incision and ended with the last stitch.

    Observers were seated behind a small movable tray close to the wall. They were about 2 m away from the sterile field at the left side of the patient, thus facing the primary surgeon for most procedures. This position allowed a good view of the room, the sterile team and the anaesthetic team, including the patient monitor; all doors were in sight of the observers. The observers were sufficiently close to the sterile team to overhear communication; however, they were sufficiently far away to not to be an obstacle for the OR personnel.

    Observer training

    Observers underwent a four-step training procedure that lasted between 25 h and 35 h. The training started with an informal visit to the OR that included instructions about dress codes, hygiene procedures and behavioural guidelines in the OR, as well as an unstructured observation of one procedure. The second step was a 4-h off-site training session during which trainees received general information about the setting (eg, roles and functions of OR team members, formal working procedures and spatial arrangements in the OR), followed by a structured introduction into the observational system (eg, explanations for each code and short video clips as behavioural examples). Trainees were then handed an information packet and asked to familiarise themselves with the coding system. The third training step consisted of observing two procedures under the direct guidance of an expert observer. In the fourth step, trainees observed two to four surgical procedures independently, but at the same time as an expert coder. After each of the surgeries, disagreements between expert and trainee were discussed. Training was considered complete if agreement between trainees and expert coders (Cohen's κ) was ≥0.75 for all codes, which was typically the case after three or four independent observations.

    Interobserver reliability

    Many studies based on observational data refer to relatively short procedures (cf web appendix). SO-DIC-OR was developed to observe long procedures with a scheduled duration of 3–7 h. Long continued observation bears a high risk of potential quality loss due to observer fatigue. We therefore tested interobserver reliability for different time periods, and we assessed fatigue effects. Reliabilities were calculated (A) for the whole procedure, (B) for the early (ie, the 1st hour) and late phases (ie, 3 h after the incision until the end of the procedure). To test for fatigue effects, we assessed interobserver reliability for the late phase using an observer present from the beginning of the procedure (‘tired’) or an observer who joined 3 h into the surgery (‘non-tired’).

    Statistical analyses

    Cohen's κ and intraclass correlation coefficients (ICCs) were estimated to assess interobserver reliability. Cohen's κ is well suited for nominal scales and expresses the proportion of agreement in terms of a given category being coded or not, controlling for chance agreement. It ranges from 0 to +1.00, with 0 indicating chance agreement.77 Values between 0.41 and 0.59 are defined as fair, values between 0.60 and 0.80 are defined as substantial, and values above 0.81 indicate very good agreement.78 We calculated Cohen's κ for the occurrence versus non-occurrence of each observational code for every 5-min segment of the observational period. To assess interobserver reliability for frequency counts, we calculated one-way random ICCs for each code between two observers for the different observational periods.79 ,80 ICC normally ranges from 0 to 1 but can also be negative. Values higher than 0.75 indicate very good interobserver reliability.81 ,82

    Results

    The mean duration of the 29 surgeries was 302 min (median: 290, SD: 121, range: 119–643 min). All but five surgeries lasted more than 3 h. Table 2 displays the results for interobserver agreement for different time periods. Cohen's κ values indicate good to excellent interobserver agreement for the whole procedure, for the 1st hour, and after 3 h of coding (for surgeries lasting 4 h or more), as well as for a tired and a non-tired observer who joined 3 h into the procedure (all κs>0.74). Similarly, most ICCs are above 0.75. Exceptions are frequency ratings of tension in the 1st hour of coding (ICC=0.703) and after 3 h of coding between two ‘tired’ observers (ICC=0.667). The frequency agreement of two tired observers was also below 0.75 for teaching activities (ICC=0.555) and for communication with external visitors (ICC=0 .446).

    Discussion

    We developed and tested SO-DIC-OR, an observational system that allows observers to simultaneously observe distractors and aspects of teamwork and communication in the OR. The importance of human factors in surgeries is uncontested, and research on communication, teamwork and distractors in the OR has become increasingly important. Thus far, their co-occurrence and potential mutual influence have not yet been evaluated. For example, it could well be that noise distractors influence task-relevant communication in the sense that distractors may lower the rate of task-relevant communication, which, in turn, may influence performance. Such research questions can now be addressed, because SO-DIC-OR provides timed observational data for distractors as well as for communication throughout the whole process.

    Observers using SO-DIC-OR achieved high interobserver agreement, a crucial indicator of the quality of the system. Of the 22 studies summarised in the web appendix, only 12 reported results about observer agreement. Compared with the values reported there, interobserver agreement of SO-DIC-OR is similar or higher. This is a good result, given that 17 different event types had to be observed and given that behavioural observation is a difficult task, requiring constant attention and often quick decision-making.

    We developed SO-DIC-OR to be suitable for the observation of long surgical procedures. Interobserver agreement was acceptable to excellent for all time phases tested, with the exception of two ICC values (teaching and communication with visitors) between two ‘tired’ observers (ie, after 3 h of observation). Note that both events occurred with low frequency, implying that any discrepancy had a rather strong influence on ICC. Apart from these two codes, there were no substantial signs of fatigue effects after 3 h of continuous observation, making the system well suited for direct observation of short as well as long procedures.

    The high interobserver agreement of SO-DIC-OR may be due to several reasons. First, we chose well-defined categories and described them as unambiguously as possible. We defined specific, rather than combined, categories because they are easier to code. For example, we distinguished between teaching and case-relevant communication, although both are examples of a broader ‘task-relevant communication’ category. More specific categories require less cognitive effort from observers because they do not have to relate different behaviours to the same category. For later analyses, categories can be used separately but can also be combined into larger categories. Second, we chose event coding, which does not require observers to judge the quality of the behaviour observed or to integrate behaviours over time. This choice reduces cognitive load and interpretational biases; we can therefore expect higher interobserver agreement and fewer differences between novices and experts than in behavioural marker-based systems.46 ,64 ,66 Third, observers underwent intense training which included theoretical aspects, coding at least five procedures with an expert present, and postobservation discussions. This training is a considerable investment, but it is not unusual for observing group interactions.83 ,84

    This study has limitations. First, SO-DIC-OR has only been tested in elective surgeries; emergency procedures have not yet been included, nor have laparoscopic procedures. Second, our data do not allow us to assess observer-specific biases. To assess such biases, multiple observers would have to observe the same procedure. Due to space limitations, it was not possible to install more than two observational stations in the OR. Third, aiming at demonstrating the reliability of our system, the current study does not allow us to establish predictive validity; doing so would have required us to compare the observations with external performance standards. Fourth, to limit the number of different categories to observe, the level of code differentiation is limited. For example, the communication categories are relatively general—future research will have to show whether these categories are sensitive enough to detect meaningful differences between high and low performing teams. In addition, some categories may not be unambiguous with regards to their categorisation. For example, side conversations (eg, among anaesthetists) may not always have a distracting effect. They could contain important information that—when overheard by the sterile team—could have positive effects on coordination and the procedure. Unlike observational systems based on behavioural markers, SO-DIC-OR does not allow an immediate assessment of teamwork quality. To be used for training purposes, it would need to be adapted. However, it is easy to produce frequency counts for the whole procedures or for specific time periods. These can serve as bases for training-related discussions within surgical teams.

    Conclusions and outlook

    Our study showed that it is possible to reliably observe teamwork and distractors simultaneously in the OR, even for long procedures. Data collection is relatively straightforward and based on an easily adaptable spreadsheet; no specialised observational software is needed. SO-DIC-OR is conceived primarily for research purposes. Data collected with SO-DIC-OR allows assessing combined influences of distractors and communication on surgical performance and outcomes.

    Acknowledgments

    The authors thank Brigitte Dubach (head nurse), Uwe Klopsch and Melanie Bolinger for their support and Guillaume Crot, Christa Gfeller, Simon Huber, Moana Monnier, Irene Mühlemann and Anna Püschel for help in data collection.

    References

    1. ↵
      1. Gawande AA,
      2. Zinner MJ,
      3. Studdert DM,
      4. et al
      . Analysis of errors reported by surgeons at three teaching hospitals. Surgery 2003;133:614–21.
      OpenUrlCrossRefPubMedWeb of Science
    2. ↵
      1. de Leval MR,
      2. Carthey J,
      3. Wright DJ,
      4. et al
      . Human factors and cardiac surgery: a multicenter study. J Thorac Cardiovasc Surg 2000;119:661–72.
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      1. Youngson GG,
      2. Flin R
      . Patient safety in surgery: non-technical aspects of safe surgical performance. Patient Saf Surg 2010;4:4.
      OpenUrlCrossRefPubMed
    4. ↵
      1. Kurmann A,
      2. Tschan F,
      3. Semmer NK,
      4. et al
      . Human factors in the operating room–the surgeons view. Trends Anaesth Crit Care 2012;2:224–7.
      OpenUrlCrossRef
    5. ↵
      1. Undre S,
      2. Sevdalis N,
      3. Healey AN,
      4. et al
      . Teamwork in the operating theatre: cohesion or confusion? J Eval Clin Pract 2006;12:182–9.
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      1. Undre S,
      2. Healey AN,
      3. Darzi A,
      4. et al
      . Observational assessment of surgical teamwork: a feasibility study. World J Surg 2006;30:1774–83.
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. Yule S,
      2. Flin R,
      3. Paterson-Brown S,
      4. et al
      . Development of a rating system for surgeons’ non-technical skills. Med Educ 2006;40:1098–104.
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Undre S,
      2. Sevdalis N,
      3. Vincent C
      . Observing and assessing surgical teams: The Observational Teamwork Assessment for Surgery (OTASS). In: Flin R, Mitchell L, eds. Safer Surgery Analysing behaviour in the operating theatre. London: Ashgate, 2009:83–102.
    9. ↵
      1. Mitchell L,
      2. Flin R,
      3. Yule S,
      4. et al
      . Evaluation of the Scrub Practitioners’ List of Intraoperative Non-Technical Skills system. Int J Nurs Stud 2012;49:201–11.
      OpenUrlCrossRefPubMed
    10. ↵
      1. Leong G,
      2. Wilson J,
      3. Charlett A
      . Duration of operation as a risk factor for surgical site infection: comparison of English and US data. J Hosp Infect 2006;63:255–62.
      OpenUrlCrossRefPubMed
    11. ↵
      1. Campbell DA Jr.,
      2. Henderson WG,
      3. Englesbe MJ,
      4. et al
      . Surgical site infection prevention: The importance of operative duration and blood transfusion—results of the first American College of Surgeons–National Surgical Quality Improvement Program Best Practices Initiative. J Am Coll Surgeons 2008;207:810–20.
      OpenUrlCrossRefPubMedWeb of Science
    12. ↵
      1. Healey AN,
      2. Sevdalis N,
      3. Vincent C
      . Measuring intra-operative interference from distraction and interruption observed in the operating theatre. Ergonomics 2006;49:589–604.
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      1. Healey AN,
      2. Primus CP,
      3. Koutantji M
      . Quantifying distraction and interruption in urological surgery. Qual Saf Health Care 2007;16:135–9.
      OpenUrlAbstract/FREE Full Text
    14. ↵
      1. Healey AN,
      2. Olsen SE,
      3. Davis R,
      4. et al
      . A method for measuring work interference in surgical teams. Cognit TechnolWork 2008;10:305–12.
      OpenUrl
    15. ↵
      1. Persoon MC,
      2. Broos HJ,
      3. Witjes JA,
      4. et al
      . The effect of distractions in the operating room during endourological procedures. Surg Endosc 2011;25:437–43.
      OpenUrlCrossRefPubMed
    16. ↵
      1. Goodell KH,
      2. Cao CG,
      3. Schwaitzberg SD
      . Effects of cognitive distraction on performance of laparoscopic surgical tasks. J Laparoendosc Adv Surg Tech A 2006;16:94–8.
      OpenUrlCrossRefPubMed
    17. ↵
      1. Pluyter JR,
      2. Buzink SN,
      3. Rutkowski AF,
      4. et al
      . Do absorption and realistic distraction influence performance of component task surgical procedure? Surg Endosc 2010;24:902–7.
      OpenUrlCrossRefPubMed
    18. ↵
      1. Feuerbacher RL,
      2. Funk K,
      3. Spight DH,
      4. et al
      . Realistic distractions and interruptions that impair simulated surgical performance by Novice Surgeons. Arch Surg-Chicago 2012;147:1026–30.
      OpenUrlCrossRef
    19. ↵
      1. Hodge B,
      2. Thompson JF
      . Noise pollution in the operating theatre. Lancet 1990;335:891–4.
      OpenUrlCrossRefPubMedWeb of Science
    20. ↵
      1. Park J,
      2. Waqar S,
      3. Kersey T,
      4. et al
      . Effect of distraction on simulated anterior segment surgical performance. J Cataract Refract Surg 2011;37:1517–22.
      OpenUrlCrossRefPubMed
    21. ↵
      1. Catchpole K,
      2. Mishra A,
      3. Handa A,
      4. et al
      . Teamwork and error in the operating room: analysis of skills and roles. Ann Surg 2008;247:699–706.
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. Christian CK,
      2. Gustafson ML,
      3. Roth E,
      4. et al
      . A prospective study of patient safety in the operating room. Surgery 2006;139:159–73.
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      1. Healey AN,
      2. Undre S,
      3. Sevdalis N,
      4. et al
      . The complexity of measuring interprofessional teamwork in the operating theatre. J Interprof Care 2006;20:485–95.
      OpenUrlCrossRefPubMed
    24. ↵
      1. Makary MA,
      2. Holzmueller CG,
      3. Thompson D,
      4. et al
      . Operating room briefings: working on the same page. Jt Comm J Qual Patient Saf 2006;32:351–5.
      OpenUrlPubMed
    25. ↵
      1. Yule S,
      2. Flin R,
      3. Paterson-Brown S,
      4. et al
      . Non-technical skills for surgeons in the operating room: a review of the literature. Surgery 2006;139:140–9.
      OpenUrlCrossRefPubMedWeb of Science
    26. ↵
      1. Leonard M,
      2. Graham S,
      3. Bonacum D
      . The human factor: the critical importance of effective teamwork and communication in providing safe care. Qual Saf Health Care 2004;13:I85–90.
      OpenUrlCrossRefWeb of Science
    27. ↵
      1. Lingard L,
      2. Espin S,
      3. Whyte S,
      4. et al
      . Communication failures in the operating room: an observational classification of recurrent types and effects. Qual Saf Health Care 2004;13:330–4.
      OpenUrlAbstract/FREE Full Text
    28. ↵
      1. Pratt SD,
      2. Mann S,
      3. Salisbury M,
      4. et al
      . John M. Eisenberg Patient Safety and Quality Awards. Impact of CRM-based training on obstetric outcomes and clinicians’ patient safety attitudes. Jt Comm J Qual Patient Saf 2007;33:720–5.
      OpenUrlPubMed
    29. ↵
      1. Lingard L,
      2. Regehr G,
      3. Orser B,
      4. et al
      . Evaluation of a preoperative checklist and team briefing among surgeons, nurses, and anesthesiologists to reduce failures in communication. Arch Surg 2008;143:12–7; discussion 18.
      OpenUrlCrossRefPubMedWeb of Science
    30. ↵
      1. Bogenstatter Y,
      2. Tschan F,
      3. Semmer NK,
      4. et al
      . How Accurate Is Information Transmitted to Medical Professionals Joining a Medical Emergency? A Simulator Study. Hum Factors 2009;51:115–25.
      OpenUrlAbstract/FREE Full Text
    31. ↵
      1. Guzzo RA,
      2. Wagner DB,
      3. Maguire E,
      4. et al
      . Implicit theories and the evaluation of group process and performance. Organ Behav Hum Decis Processes 1986;37:279–95.
      OpenUrlCrossRef
    32. ↵
      1. Guerlain S,
      2. Calland JF
      . RATE: A Customizable, Portable Hardware/Software System for Analysing and Teaching Human Performance in the Operating Room. In: Flin R, Mitchell L, eds. Safer Surgery: Analysing Behaviour in the Operating Theatre. 2009:117–28.
    33. ↵
      1. Guerlain S,
      2. Adams RB,
      3. Turrentine FB,
      4. et al
      . Assessing team performance in the operating room: development and use of a "black-box" recorder and other tools for the intraoperative environment. J Am Coll Surg 2005;200:29–37.
      OpenUrlCrossRefPubMedWeb of Science
    34. ↵
      1. Santos R,
      2. Bakero L,
      3. Franco P,
      4. et al
      . Characterization of non-technical skills in paediatric cardiac surgery: communication patterns. Eur J Cardiothorac Surg 2012;41:1005–12.
      OpenUrlAbstract/FREE Full Text
    35. ↵
      1. Yoder PJ,
      2. Symons FJ
      . Observational measurement of behavior. New York: Springer Publishing Company, 2010.
    36. ↵
      1. Lingard L,
      2. Regehr G,
      3. Espin S,
      4. et al
      . A theory-based instrument to evaluate team communication in the operating room: balancing measurement authenticity and reliability. Qual Saf Health Care 2006;15:422–6.
      OpenUrlAbstract/FREE Full Text
    37. ↵
      1. Guerlain S,
      2. Calland JF
      . RATE. A customizable, portable hardware/softowar system for analysing and teaching human perfomrnac ein the operating room. In: Flin R, Mitchell L, eds. Safer surgery Analysing Beahvior in the operating theatre. London: Ashgate, 2009:117–28.
    38. ↵
      1. Sharma B,
      2. Mishra A,
      3. Aggarwal R,
      4. et al
      . Non-technical skills assessment in surgery. Surg Oncol 2011;20:169–77.
      OpenUrlCrossRefPubMed
    39. ↵
      1. Frankel A,
      2. Gardner R,
      3. Maynard L,
      4. et al
      . Using the Communication and Teamwork Skills (CATS) Assessment to Measure Health Care Team Performance. Jt Comm J Qual Patient Saf 2007;33:549–58.
      OpenUrlPubMed
    40. ↵
      1. Yule S,
      2. Flin R,
      3. Maran N,
      4. et al
      . Surgeons’ non-technical skills in the operating room: reliability testing of the NOTSS behavior rating system. World J Surg 2008;32:548–56.
      OpenUrlCrossRefPubMedWeb of Science
    41. ↵
      1. Mishra A,
      2. Catchpole K,
      3. McCulloch P
      . The Oxford NOTECHS System: reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. Qual Saf Health Care 2009;18:104–8.
      OpenUrlAbstract/FREE Full Text
    42. ↵
      1. Robertson ER,
      2. Hadi M,
      3. Morgan LJ,
      4. et al
      . Oxford NOTECHS II: a modified theatre team non-technical skills scoring system. PLoS ONE 2014;9:e90320.
      OpenUrlCrossRefPubMed
    43. ↵
      1. Lingard L,
      2. Reznick R,
      3. Espin S,
      4. et al
      . Team communications in the operating room: talk patterns, sites of tension, and implications for novices. Acad Med 2002;77:232–7.
      OpenUrlCrossRefPubMedWeb of Science
    44. ↵
      1. Mazzocco K,
      2. Petitti DB,
      3. Fong KT,
      4. et al
      . Surgical team behaviors and patient outcomes. Am J Surg 2009;197:678–85.
      OpenUrlCrossRefPubMedWeb of Science
    45. ↵
      1. Nurok M,
      2. Lipsitz S,
      3. Satwicz P,
      4. et al
      . A novel method for reproducibly measuring the effects of interventions to improve emotional climate, indices of team skills and communication, and threat to patient outcome in a high-volume thoracic surgery center. Arch Surg 2010;145:489–95.
      OpenUrlCrossRefPubMedWeb of Science
    46. ↵
      1. Schraagen JM,
      2. Schouten T,
      3. Smit M,
      4. et al
      . Assessing and improving teamwork in cardiac surgery. Qual Saf Health Care 2010;19:e29.
      OpenUrlAbstract/FREE Full Text
    47. ↵
      1. Parker SH,
      2. Yule S,
      3. Flin R,
      4. et al
      . Surgeons’ leadership in the operating room: an observational study. Am J Surg 2011;204:347–54.
      OpenUrlPubMed
    48. ↵
      1. Parikh SN,
      2. Grice SS,
      3. Schnell BM,
      4. et al
      . Operating Room Traffic: Is There Any Role of Monitoring It? J Pediatr Orthop 2010;30:617.
      OpenUrlCrossRefPubMed
    49. ↵
      1. Gillespie BM,
      2. Chaboyer W,
      3. Fairweather N
      . Interruptions and miscommunications in surgery: an observational study. AORN J 2012;95:576–90.
      OpenUrlCrossRefPubMed
    50. ↵
      1. Gardezi F,
      2. Lingard L,
      3. Espin S,
      4. et al
      . Silence, power and communication in the operating room. J Adv Nurs 2009;65:1390–9.
      OpenUrlCrossRefWeb of Science
    51. ↵
      1. Flin R,
      2. Patey R
      . Non-technical skills for anaesthetists: developing and applying ANTS. Best Pract Res Clin Anaesthesiol 2011;25:215–27.
      OpenUrlCrossRefPubMed
    52. ↵
      1. Mitchell L,
      2. Flin R,
      3. Yule S,
      4. et al
      . Thinking ahead of the surgeon. An interview study to identify scrub nurses’ non-technical skills. Int J Nurs Stud 2010;48:818–28.
      OpenUrlPubMed
    53. ↵
      1. Sevdalis N,
      2. Davis R,
      3. Koutantji M,
      4. et al
      . Reliability of a revised NOTECHS scale for use in surgical teams. Am J Surg 2008;196:184–90.
    54. ↵
      1. Berg BL,
      2. Lune H
      . Qualitative research methods for the social sciences. 2004.
    55. ↵
      1. Hammersley M,
      2. Atkinson P
      . Ethnography: principles in practice. Taylor & Francis, 2007.
    56. ↵
      1. Emerson RM,
      2. Fretz RI,
      3. Shaw LL
      . Writing ethnographic fieldnotes. University of Chicago Press, 2011.
    57. ↵
      1. Atkinson P
      . Handbook of ethnography. Sage Publications Ltd, 2007.
    58. ↵
      1. Hull L,
      2. Arora S,
      3. Kassab E,
      4. et al
      . Observational teamwork assessment for surgery: content validation and tool refinement. J Am Coll Surg 2011;212:234–43. e231–5.
      OpenUrlCrossRefPubMedWeb of Science
    59. ↵
      1. Carthey J,
      2. de Leval MR,
      3. Wright DJ,
      4. et al
      . Behavioural markers of surgical excellence. Saf Sci 2003;41:409–25.
      OpenUrlCrossRefWeb of Science
    60. ↵
      1. Yule S,
      2. Flin R,
      3. Maran N,
      4. et al
      . Development and evaluation of the NOTTS behavior rating system for intraoperative surgery. In: Flin R, Mitchell L, eds. Safer Surgery Analysing Behaviour in the Operating Theatre. London: Ashgate, 2009:7–26.
      1. Flin R,
      2. Yule S,
      3. McKenzie L,
      4. et al
      . Attitudes to teamwork and safety in the operating theatre. Surgeon 2006;4:145–51.
      OpenUrlCrossRefPubMedWeb of Science
    62. ↵
      1. Crossley J,
      2. Marriott J,
      3. Purdie H,
      4. et al
      . Prospective observational study to evaluate NOTSS (Non-Technical Skills for Surgeons) for assessing trainees’ non-technical performance in the operating theatre. Br J Surg 2011;98:1010–20.
      OpenUrlCrossRefPubMed
    63. ↵
      1. Fletcher G,
      2. Flin R,
      3. McGeorge P,
      4. et al
      . Anaesthetists’ Non-Technical Skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth 2003;90:580–8.
      OpenUrlAbstract/FREE Full Text
    64. ↵
      1. Sevdalis N,
      2. Lyons M,
      3. Healey AN,
      4. et al
      . Observational teamwork assessment for surgery: construct validation with expert versus novice raters. Ann Surg 2009;249:1047–51.
      OpenUrlCrossRefPubMedWeb of Science
    65. ↵
      1. Hull L,
      2. Arora S,
      3. Symons NR,
      4. et al
      . Training faculty in nontechnical skill assessment: national guidelines on program requirements. Ann Surg 2013;258:370–5.
      OpenUrlCrossRefPubMed
    66. ↵
      1. Yule S,
      2. Rowley D,
      3. Flin R,
      4. et al
      . Experience matters: comparing novice and expert ratings of non-technical skills using the NOTSS system. ANZ J Surg 2009;79:154–60.
      OpenUrlCrossRefPubMedWeb of Science
    67. ↵
      1. Russ S,
      2. Hull L,
      3. Rout S,
      4. et al
      . Observational teamwork assessment for surgery: feasibility of clinical and nonclinical assessor calibration with short-term training. Ann Surg 2012;255:804.
      OpenUrlCrossRefPubMed
    68. ↵
      1. Schraagen JM,
      2. Schouten T,
      3. Smit M,
      4. et al
      . A prospective study of paediatric cardiac surgical microsystems: assessing the relationships between non-routine events, teamwork and patient outcomes. BMJ Qual Saf 2011;20:599–603.
      OpenUrlAbstract/FREE Full Text
    69. ↵
      1. Wiegmann DA,
      2. Eggman AA,
      3. Elbardissi AW,
      4. et al
      . Improving cardiac surgical care: a work systems approach. Appl Ergon 2010;41:701–12.
      OpenUrlCrossRefPubMedWeb of Science
    70. ↵
      1. McGrath JE,
      2. Altermatt WT
      . Observation and analysis of group interaction over time: Some methodological and strategic consequences. In: Hogg MA, Tindale RS, eds. Blackwell Handbook of Social Pschology: Group Processes. Oxford: Blackwell Publishers, 2001:525–56.
    71. ↵
      1. Weingart L,
      2. Olekalns M,
      3. Smith PL
      . Quantitative coding of negotiation behavior. International Negotiation 2004;9:441–55.
      OpenUrlCrossRef
    72. ↵
      1. Weingart LR
      . How did they do that? The ways and means of studying group processes. Res Organ Behav 1997;19:189–239.
      OpenUrl
    73. ↵
      1. Barach P,
      2. Johnson JK,
      3. Ahmad A,
      4. et al
      . A prospective observational study of human factors, adverse events, and patient outcomes in surgery for pediatric cardiac disease. J Thorac Cardiovasc Surg 2008;136:1422–8.
      OpenUrlCrossRefPubMedWeb of Science
    74. ↵
      1. Catchpole KR,
      2. de Leval MR,
      3. McEwan A,
      4. et al
      . Patient handover from surgery to intensive care: using Formula 1 pit-stop and aviation models to improve safety and quality. Paediatr Anaesth 2007;17:470–8.
      OpenUrlCrossRefPubMed
    75. ↵
      1. Waller MJ,
      2. Gupta N,
      3. Giambatista RC
      . Effects of Adaptive Behaviors and Shared Mental Models on Control Crew Performance. Manag Sci 2004;50:1534–44.
      OpenUrlCrossRef
    76. ↵
      1. Weaver SJ,
      2. Rosen MA,
      3. DiazGranados D,
      4. et al
      . Does teamwork improve performance in the operating room? A multilevel evaluation. Jt Comm J Qual Patient Saf 2010;36:133–42.
      OpenUrlPubMed
    77. ↵
      1. Cohen J
      . A coefficient of agreement for nominal scales. Educ Psychol Meas 1960;20:37–46.
      OpenUrlCrossRefWeb of Science
    78. ↵
      1. Landis JR,
      2. Koch GG
      . The measurement of observer agreement for categorical data. Biometrics 1977;33:159–74.
      OpenUrlCrossRefPubMedWeb of Science
    79. ↵
      1. Shrout PE,
      2. Fleiss JL
      . Intraclass correlations: uses in assessing rater reliability. Psychol Bull 1979;86:420–0.
      OpenUrlCrossRefPubMedWeb of Science
    80. ↵
      1. Von Eye A,
      2. Mun EY
      . Analyzing rater agreement: Manifest variable methods. Mahwah, NJ: Lawrence Erlbaum Associates, Publishers, 2005.
    81. ↵
      1. Weir JP
      . Quantifying test-retest reliability using the intraclass correlation coefficient. J Strength Cond Res 2005;19:231–40.
      OpenUrlCrossRefPubMedWeb of Science
    82. ↵
      1. Fleiss JL,
      2. Slakter MJ,
      3. Fischman SL,
      4. et al
      . Inter-examiner reliability in caries trials. J Dent Res 1979;58:604–9.
      OpenUrlAbstract/FREE Full Text
    83. ↵
      1. Meyers RA,
      2. Seibold DR
      . Coding group interaction. In: Hollingshead AB, Pool MS, eds. Research methods for studying groups and teams. New York: Routledge 2012:329– 57.
    84. ↵
      1. Russ S,
      2. Hull L,
      3. Rout S,
      4. et al
      . Observational Teamwork Assessment for Surgery. Feasibility of Clinical and NonClinical Assessor Calibration With Short-Term Training. Ann Surg 2012;255:804–9.
      OpenUrlCrossRefPubMed
    85. Yule S, Flin R, Maran N, et al. Development and evaluation of the NOTTS behavior rating system for intraoperative surgery. In: Flin R, Mitchell L, eds. Safer Surgery. Analysing Behaviour in the Operating Theatre. London: Ashgate, 2009:7–26.
      1. Lingard L,
      2. Garwood S,
      3. Poenaru D
      . Tensions influencing operating room team function: does institutional context make a difference?. Med Educ 2004;38:691–9.
      OpenUrlCrossRefPubMedWeb of Science
      1. Nurok M,
      2. Lipsitz S,
      3. Satwicz P,
      4. et al
      . A novel method for reproducibly measuring the effects of interventions to improve emotional climate, indices of team skills and communication, and threat to patient outcome in a high-volume thoracic surgery center. Arch Surg 2010;145:489.
      OpenUrlCrossRefPubMedWeb of Science

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • JCS and FT contributed equally.

    • Contributors Study concept and design: FT, GB, NKS, DC. Method development: FT, JCS, NKS, SK. Data acquisition: JCS, SK, NJ, FT. Drafting of the manuscript: FT, JCS, NKS. Critical revision of the manuscript: GB, NJ, SK, AK, DC. Statistical analyses: JCS, FT, NJ. Administrative, technical and material support: GB. Study supervision: FT, GB, NKS.

    • Funding This study was supported by a grant from the Swiss National Science Foundation (grant 138273).

    • Competing interests None.

    • Ethics approval Institutional review board University Hospital Bern.

    • Provenance and peer review Not commissioned; externally peer reviewed.