Article Text

PDF

The impact of feedback of intraoperative technical performance in surgery: a systematic review
  1. Abhishek Trehan1,
  2. Ashton Barnett-Vanes2,
  3. Matthew J Carty3,
  4. Peter McCulloch4,
  5. Mahiben Maruthappu2
  1. 1Medical Sciences Division, University of Oxford, Oxford, UK
  2. 2Imperial College London, London, UK
  3. 3Harvard Medical School, Boston, USA
  4. 4Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK
  1. Correspondence to Abhishek Trehan; abhishek.trehan{at}lincoln.ox.ac.uk

Abstract

Objectives Increasing patient demands, costs and emphasis on safety, coupled with reductions in the length of time surgical trainees spend in the operating theatre, necessitate means to improve the efficiency of surgical training. In this respect, feedback based on intraoperative surgical performance may be beneficial. Our aim was to systematically review the impact of intraoperative feedback based on surgical performance.

Setting MEDLINE, Embase, PsycINFO, AMED and the Cochrane Database of Systematic Reviews were searched. Two reviewers independently reviewed citations using predetermined inclusion and exclusion criteria. 32 data-points per study were extracted.

Participants The search strategy yielded 1531 citations. Three studies were eligible, which comprised a total of 280 procedures by 62 surgeons.

Results Overall, feedback based on intraoperative surgical performance was found to be a powerful method for improving performance. In cholecystectomy, feedback led to a reduction in procedure time (p=0.022) and an improvement in economy of movement (p<0.001). In simulated laparoscopic colectomy, feedback led to improvements in instrument path length (p=0.001) and instrument smoothness (p=0.045). Feedback also reduced error scores in cholecystectomy (p=0.003), simulated laparoscopic colectomy (p<0.001) and simulated renal artery angioplasty (p=0.004). In addition, feedback improved balloon placement accuracy (p=0.041), and resulted in a smoother learning curve and earlier plateau in performance in simulated renal artery angioplasty.

Conclusions Intraoperative feedback appears to be associated with an improvement in performance, however, there is a paucity of research in this area. Further work is needed in order to establish the long-term benefits of feedback and the optimum means and circumstances of feedback delivery.

  • Feedback
  • Quality improvement
  • performance

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

Statistics from Altmetric.com

Strengths and limitations of this study

  • Systematic review, minimising likelihood of relevant papers being missed.

  • Detailed extraction of data from studies.

  • Very few relevant studies in the literature despite the importance of the topic.

Introduction

In light of increasing patient demands, costs and emphasis on safety, surgeons and their outcomes have become the subject of increased expectations and scrutiny.1 Coupled with this, time spent in the operating theatre by surgical trainees is declining worldwide due to regulations that have reduced the legal number of working hours;2 ,3 this is particularly alarming in light of the now well-established relationship between surgical volume and surgical outcomes,4–6 and recent work has also directly linked intraoperative technical skill to complication and mortality rates.7 Such challenges necessitate increased efficiency of surgical training programmes such that an equivalent or superior level of surgical proficiency can be achieved in spite of the shorter length of time spent in the operating theatre.

One means by which surgical skills acquisition could be enhanced is via the dissemination of feedback on intraoperative performance. Studies in medical students performing basic surgical skills such as suturing, knot tying and basic laparoscopic tasks, have demonstrated that feedback can improve skill acquisition.8–10 Additionally, proficiency in simulated laparoscopic salpingectomy is accelerated in medical students when they receive instructor feedback,11 and feedback improves colonoscopy performance in gastroenterologists.12 Thus, provision of feedback on intraoperative surgical performance to surgical trainees may also be associated with improved performance and/or a more rapid acquisition of skills, and hence formalised feedback should potentially serve as a key component of future surgical training programmes. Although feedback of intraoperative skill and technique can be a common occurrence in the operating theatre, the impact of this on performance and requirements for optimal training have thus far not been reviewed. We therefore conducted a systematic review to evaluate the impact of feedback of technical skill in the operating theatre and in the context of simulation. For the purpose of this study, feedback was defined as the provision of information pertaining to the operator's surgical performance with the aim of improving subsequent performance.

Methods

Data sources and search strategy

The systematic review was conducted in accordance with PRISMA guidelines. A comprehensive search was undertaken to determine the impact of feedback on surgical performance via the Ovid SP interface. The following databases were searched from inception to February 2013: MEDLINE, Embase, PsycINFO, AMED and the Cochrane Database of Systematic Reviews.

We used two different domains of MeSH-terms and key words combined by ‘AND’, and within each domain the terms were combined by ‘OR’. The first domain contained terms related to surgical skill and performance, while the second contained terms related to the impact of feedback. A detailed search strategy can be found in online supplementary appendix 1. The search was limited to English publications with no other restrictions.

Study selection

Two reviewers independently reviewed citations and selected eligible studies based on predetermined inclusion and exclusion criteria. Publications were selected for review if they satisfied the following inclusion criteria: the article was published in a peer-reviewed journal; the article described a study involving surgical patients or simulation; the article investigated the impact of feedback of intraoperative surgical performance; the article used a statistical unit that was patient-focused or procedure-focused. The following exclusion criteria were applied to search results: the article was a conference abstract, editorial, letter, opinion, audit or review; the population studied was non-surgical (eg, pathology, medicine); the article described methods of feedback, not the impact of feedback; the article utilised a medical student population. Two authors (MM, AT) independently examined all retrieved articles for inclusion. Any disagreements over inclusion or exclusion were resolved by discussion between authors. References in relevant papers were also reviewed in order to identify any additional studies that may have been missed by the search strategy.

Data extraction

Thirty two data-points per study were extracted using a predesigned data collection form including: first author, year of publication, study aim, study type, study design (eg, prospective, retrospective, experimental, observational, cross-sectional, longitudinal), study population, population setting (eg, hospital), surgical specialty, surgical procedure analysed, number of surgeons, types of feedback dissemination, content of feedback, frequency of feedback, measured outcomes and interventions following feedback. The full data extraction from the studies can be found in online supplementary appendix 2.

Results

Study identification and selection

Our search yielded 1531 citations, of which 1185 articles were excluded. After detailed evaluation of the 346 remaining articles, three studies remained eligible, which comprised of a total of 280 procedures by 62 surgeons.13–15 A flow diagram of the search results is illustrated in figure 1.

Figure 1

Summary of search strategy for identification of relevant studies.

Study characteristics

All three studies were performed on surgical trainees, one involving live cholecystectomy cases, one involving simulated laparoscopic colectomy and one involving simulated renal artery angioplasty. Two studies were two-armed RCTs (with one arm receiving feedback and the control arm receiving no feedback),13 ,15 while one study was a three-armed RCT (with one arm receiving expert feedback, another arm receiving non-expert feedback and one arm receiving no feedback).14 The studies included in this review are shown in table 1, and their basic characteristics are summarised in table 2.

Table 1

Overview of studies included and impact of feedback on performance

Table 2

Basic characteristics of studies included

Feedback dissemination

In all studies, feedback was delivered orally after each procedure.13–15 No written feedback was provided in any of the studies and one study required participants to self-assess their performance after each case in addition to receiving oral feedback.13 One study utilised video footage in facilitating feedback.15

Feedback contents

The two examined studies involving simulation provided participants with feedback relevant to the exercise, including standard instrument metrics, procedural time and errors, accompanied with a description of correct methods where necessary.13 ,14 For the study of live surgery, feedback was facilitated by review of a videotape recording of the operation and a 60 min structured feedback session, during which technical deficiencies and possible errors were covered and instructions for improvement offered.15 One study provided benchmarking relative to peers13 and no studies provided surgeons with comparable data from the literature.

In one study, feedback was provided solely by a single surgeon with significant operative and teaching experience in laparoscopy.15 One study assessed the impact of expert (consultant surgeons) and non-expert (inexperienced surgical trainees) instructor feedback.14 There was no difference to be found between expert and non-expert feedback in all outcomes assessed other than error scores, which were lower when using the Vascular Interventional Surgical Trainer (VIST) error metrics and scoring during the operation (p=0.009), but not so when using a custom, more extensive scoring sheet completed by a single expert on reviewing video footage of the procedures.14

Impact of feedback

All three studies identified improvements in one or more of the outcomes assessed. Table 3 shows outcomes assessed across the three studies with associated p values. In addition, the study of simulated renal artery angioplasty assessed procedure-specific outcomes including contrast volume (mL), fluoroscopic time (seconds), balloon placement accuracy (mm), residual stenosis and lesion coverage (%).14 Of these, balloon placement accuracy was shown to be significantly improved in those receiving feedback (p=0.041).14 Although not reaching statistical significance, contrast volume utilised was 24.9 mL in control group, and 9.55 mL in those receiving feedback.14 While not demonstrated via statistical methods, a smoother learning curve and earlier plateau in performance was noted in the group with feedback.14

Table 3

Key outcomes in included studies

Discussion

Our review included three studies assessing the impact of feedback of intraoperative surgical performance. Feedback was consistently found to be a powerful method for improving surgical performance in terms of operative metrics such as error scores and instrument movement metrics, as well as metrics specific to the procedure being undertaken. Feedback could thus represent a simple but powerful means by which efficiency and safety could be improved, thereby allowing for the attainment of surgical skills to a greater level of proficiency and/or in a shorter length of time in the context of training. This is of particular relevance as, with the exception of video and virtual reality simulator training, training methods known to enhance performance in the operating theatre are few and far between.16 ,17

Only three studies were included in this review, reflecting the dearth of research in this area despite the significant benefits that feedback could bring; there appear to be many studies in the literature that describe how to assess or rate technical skill,18–20 but very few that actually assess how this data should be used. Limitations of this study include the fact that the search was conducted in February 2013, that conference abstracts were excluded and that study quality was not formally assessed.

All three studies were randomised controlled trials, however, two of these three involved simulated, as opposed to live, procedures.13 ,14 Future studies should look to further assess the impact of feedback related to live surgery such that the broader implications of feedback can be appreciated.

The impact of feedback on long-term skill acquisition was not studied; all studies only assessed surgical performance with between one and five procedures after the first feedback was provided. Thus, studies taking place over a longer time scale are necessary. It is also important to establish the clinical significance of feedback; none of the studies included assessed whether the improvement in technical skill was associated with an improvement in clinical outcomes, although one might suspect it would, particularly in light of recent findings that technical skills rated by experts based on video footage correlate with surgical outcomes.7

In one study, feedback included a review of a videotape recording and a 60 min structured feedback session with a senior surgeon.15 Although extensive feedback sessions have been suggested (but not shown) to be effective,21 provision of feedback in this manner may be resource intensive and hence cost- and time-effectiveness must also be considered. The finding from a study involving simulation that non-expert delivered feedback is still effective14 may broaden options for educationalists and time-pressed senior surgeons, although one must be careful not to implement counter-productive feedback initiatives.

Although there are a few studies on this subject, all studies included in our analysis were randomised controlled trials. Given the consistent benefit of feedback demonstrated, this supports further research on this topic and implementation of structured intraoperative feedback initiatives.

The small number of studies included in this review highlights the need for more substantive research in this area in order to establish the optimum means and circumstances of feedback dissemination such that standardised methods for future widespread implementation can be attained, and future studies should consider the effect of the following study variables

  1. Source (oral/written), facilitator (expert/non-expert), frequency (every procedure/once daily/weekly/monthly) and duration of feedback (months/years).

  2. Surgeon involvement in feedback (either active or passive), standardised means of assessing surgical performance (which may be both generic and procedure-specific), content of feedback, timing of feedback relative to the procedure (intraoperatively/postoperatively), and the opportunities available for discussion, correction and learning.

  3. Benchmarking (relative to peers as well as to literature data) and feedback based on intraoperative recordings reviewed at a later time point.

  4. Other interventions utilised, such as guidelines, education and review of instructional videos. The contributions of these interventions, and the additive effect they may have with feedback on performance and outcomes are poorly understood.

It should also be borne in mind that in some circumstances or when delivered inappropriately, feedback may not be effective; for instance, although a number of studies in medical students have found feedback to improve acquisition of basic surgical skills,8–10 some have failed to find this,22 and the effect of feedback may plateau.23 ,24 Frameworks have been suggested in order to ensure appropriate dissemination of feedback,25 which is particularly important given the fact that trainees often feel they are provided with inadequate feedback despite senior surgeons feeling their feedback provision is adequate.26 ,27

In an era of increasing demands and scrutiny of surgeons in which surgical trainees are simultaneously spending less time in the operating theatre, methods to improve the efficiency of surgical performance are needed. The findings from this review suggest that feedback of intraoperative performance is an effective means by which this might be achieved; however, despite the potential impact, there is a paucity of research in this area, and further work is needed in order to establish the optimum circumstances and means by which feedback can be delivered in a time-effective and cost-effective manner.

References

View Abstract
  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Contributors AT, MM, AB-V were involved in the systematic search, manuscript composition, proofing. MJC, PM were involved in the conception, design, drafting, proofing.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement The search criteria and data extracted from the studies are available in the Appendices.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.