You are here

Article Text

Article menu
XML
Surgery
Research
The impact of feedback of intraoperative technical performance in surgery: a systematic review
  1. Abhishek Trehan1,
  2. Ashton Barnett-Vanes2,
  3. Matthew J Carty3,
  4. Peter McCulloch4,
  5. Mahiben Maruthappu2
  1. 1Medical Sciences Division, University of Oxford, Oxford, UK
  2. 2Imperial College London, London, UK
  3. 3Harvard Medical School, Boston, USA
  4. 4Nuffield Department of Surgical Sciences, University of Oxford, Oxford, UK
  1. Correspondence to Abhishek Trehan; abhishek.trehan{at}lincoln.ox.ac.uk

Abstract

Objectives Increasing patient demands, costs and emphasis on safety, coupled with reductions in the length of time surgical trainees spend in the operating theatre, necessitate means to improve the efficiency of surgical training. In this respect, feedback based on intraoperative surgical performance may be beneficial. Our aim was to systematically review the impact of intraoperative feedback based on surgical performance.

Setting MEDLINE, Embase, PsycINFO, AMED and the Cochrane Database of Systematic Reviews were searched. Two reviewers independently reviewed citations using predetermined inclusion and exclusion criteria. 32 data-points per study were extracted.

Participants The search strategy yielded 1531 citations. Three studies were eligible, which comprised a total of 280 procedures by 62 surgeons.

Results Overall, feedback based on intraoperative surgical performance was found to be a powerful method for improving performance. In cholecystectomy, feedback led to a reduction in procedure time (p=0.022) and an improvement in economy of movement (p<0.001). In simulated laparoscopic colectomy, feedback led to improvements in instrument path length (p=0.001) and instrument smoothness (p=0.045). Feedback also reduced error scores in cholecystectomy (p=0.003), simulated laparoscopic colectomy (p<0.001) and simulated renal artery angioplasty (p=0.004). In addition, feedback improved balloon placement accuracy (p=0.041), and resulted in a smoother learning curve and earlier plateau in performance in simulated renal artery angioplasty.

Conclusions Intraoperative feedback appears to be associated with an improvement in performance, however, there is a paucity of research in this area. Further work is needed in order to establish the long-term benefits of feedback and the optimum means and circumstances of feedback delivery.

  • Feedback
  • Quality improvement
  • performance

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

https://doi.org/10.1136/bmjopen-2014-006759

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • Systematic review, minimising likelihood of relevant papers being missed.

    • Detailed extraction of data from studies.

    • Very few relevant studies in the literature despite the importance of the topic.

    Introduction

    In light of increasing patient demands, costs and emphasis on safety, surgeons and their outcomes have become the subject of increased expectations and scrutiny.1 Coupled with this, time spent in the operating theatre by surgical trainees is declining worldwide due to regulations that have reduced the legal number of working hours;2 ,3 this is particularly alarming in light of the now well-established relationship between surgical volume and surgical outcomes,4–6 and recent work has also directly linked intraoperative technical skill to complication and mortality rates.7 Such challenges necessitate increased efficiency of surgical training programmes such that an equivalent or superior level of surgical proficiency can be achieved in spite of the shorter length of time spent in the operating theatre.

    One means by which surgical skills acquisition could be enhanced is via the dissemination of feedback on intraoperative performance. Studies in medical students performing basic surgical skills such as suturing, knot tying and basic laparoscopic tasks, have demonstrated that feedback can improve skill acquisition.8–10 Additionally, proficiency in simulated laparoscopic salpingectomy is accelerated in medical students when they receive instructor feedback,11 and feedback improves colonoscopy performance in gastroenterologists.12 Thus, provision of feedback on intraoperative surgical performance to surgical trainees may also be associated with improved performance and/or a more rapid acquisition of skills, and hence formalised feedback should potentially serve as a key component of future surgical training programmes. Although feedback of intraoperative skill and technique can be a common occurrence in the operating theatre, the impact of this on performance and requirements for optimal training have thus far not been reviewed. We therefore conducted a systematic review to evaluate the impact of feedback of technical skill in the operating theatre and in the context of simulation. For the purpose of this study, feedback was defined as the provision of information pertaining to the operator's surgical performance with the aim of improving subsequent performance.

    Methods

    Data sources and search strategy

    The systematic review was conducted in accordance with PRISMA guidelines. A comprehensive search was undertaken to determine the impact of feedback on surgical performance via the Ovid SP interface. The following databases were searched from inception to February 2013: MEDLINE, Embase, PsycINFO, AMED and the Cochrane Database of Systematic Reviews.

    We used two different domains of MeSH-terms and key words combined by ‘AND’, and within each domain the terms were combined by ‘OR’. The first domain contained terms related to surgical skill and performance, while the second contained terms related to the impact of feedback. A detailed search strategy can be found in online supplementary appendix 1. The search was limited to English publications with no other restrictions.

    Study selection

    Two reviewers independently reviewed citations and selected eligible studies based on predetermined inclusion and exclusion criteria. Publications were selected for review if they satisfied the following inclusion criteria: the article was published in a peer-reviewed journal; the article described a study involving surgical patients or simulation; the article investigated the impact of feedback of intraoperative surgical performance; the article used a statistical unit that was patient-focused or procedure-focused. The following exclusion criteria were applied to search results: the article was a conference abstract, editorial, letter, opinion, audit or review; the population studied was non-surgical (eg, pathology, medicine); the article described methods of feedback, not the impact of feedback; the article utilised a medical student population. Two authors (MM, AT) independently examined all retrieved articles for inclusion. Any disagreements over inclusion or exclusion were resolved by discussion between authors. References in relevant papers were also reviewed in order to identify any additional studies that may have been missed by the search strategy.

    Data extraction

    Thirty two data-points per study were extracted using a predesigned data collection form including: first author, year of publication, study aim, study type, study design (eg, prospective, retrospective, experimental, observational, cross-sectional, longitudinal), study population, population setting (eg, hospital), surgical specialty, surgical procedure analysed, number of surgeons, types of feedback dissemination, content of feedback, frequency of feedback, measured outcomes and interventions following feedback. The full data extraction from the studies can be found in online supplementary appendix 2.

    Results

    Study identification and selection

    Our search yielded 1531 citations, of which 1185 articles were excluded. After detailed evaluation of the 346 remaining articles, three studies remained eligible, which comprised of a total of 280 procedures by 62 surgeons.13–15 A flow diagram of the search results is illustrated in figure 1.

    Figure 1
    Figure 1

    Summary of search strategy for identification of relevant studies.

    Study characteristics

    All three studies were performed on surgical trainees, one involving live cholecystectomy cases, one involving simulated laparoscopic colectomy and one involving simulated renal artery angioplasty. Two studies were two-armed RCTs (with one arm receiving feedback and the control arm receiving no feedback),13 ,15 while one study was a three-armed RCT (with one arm receiving expert feedback, another arm receiving non-expert feedback and one arm receiving no feedback).14 The studies included in this review are shown in table 1, and their basic characteristics are summarised in table 2.

    View this table:
    Table 1

    Overview of studies included and impact of feedback on performance

    View this table:
    Table 2

    Basic characteristics of studies included

    Feedback dissemination

    In all studies, feedback was delivered orally after each procedure.13–15 No written feedback was provided in any of the studies and one study required participants to self-assess their performance after each case in addition to receiving oral feedback.13 One study utilised video footage in facilitating feedback.15

    Feedback contents

    The two examined studies involving simulation provided participants with feedback relevant to the exercise, including standard instrument metrics, procedural time and errors, accompanied with a description of correct methods where necessary.13 ,14 For the study of live surgery, feedback was facilitated by review of a videotape recording of the operation and a 60 min structured feedback session, during which technical deficiencies and possible errors were covered and instructions for improvement offered.15 One study provided benchmarking relative to peers13 and no studies provided surgeons with comparable data from the literature.

    In one study, feedback was provided solely by a single surgeon with significant operative and teaching experience in laparoscopy.15 One study assessed the impact of expert (consultant surgeons) and non-expert (inexperienced surgical trainees) instructor feedback.14 There was no difference to be found between expert and non-expert feedback in all outcomes assessed other than error scores, which were lower when using the Vascular Interventional Surgical Trainer (VIST) error metrics and scoring during the operation (p=0.009), but not so when using a custom, more extensive scoring sheet completed by a single expert on reviewing video footage of the procedures.14

    Impact of feedback

    All three studies identified improvements in one or more of the outcomes assessed. Table 3 shows outcomes assessed across the three studies with associated p values. In addition, the study of simulated renal artery angioplasty assessed procedure-specific outcomes including contrast volume (mL), fluoroscopic time (seconds), balloon placement accuracy (mm), residual stenosis and lesion coverage (%).14 Of these, balloon placement accuracy was shown to be significantly improved in those receiving feedback (p=0.041).14 Although not reaching statistical significance, contrast volume utilised was 24.9 mL in control group, and 9.55 mL in those receiving feedback.14 While not demonstrated via statistical methods, a smoother learning curve and earlier plateau in performance was noted in the group with feedback.14

    View this table:
    Table 3

    Key outcomes in included studies

    Discussion

    Our review included three studies assessing the impact of feedback of intraoperative surgical performance. Feedback was consistently found to be a powerful method for improving surgical performance in terms of operative metrics such as error scores and instrument movement metrics, as well as metrics specific to the procedure being undertaken. Feedback could thus represent a simple but powerful means by which efficiency and safety could be improved, thereby allowing for the attainment of surgical skills to a greater level of proficiency and/or in a shorter length of time in the context of training. This is of particular relevance as, with the exception of video and virtual reality simulator training, training methods known to enhance performance in the operating theatre are few and far between.16 ,17

    Only three studies were included in this review, reflecting the dearth of research in this area despite the significant benefits that feedback could bring; there appear to be many studies in the literature that describe how to assess or rate technical skill,18–20 but very few that actually assess how this data should be used. Limitations of this study include the fact that the search was conducted in February 2013, that conference abstracts were excluded and that study quality was not formally assessed.

    All three studies were randomised controlled trials, however, two of these three involved simulated, as opposed to live, procedures.13 ,14 Future studies should look to further assess the impact of feedback related to live surgery such that the broader implications of feedback can be appreciated.

    The impact of feedback on long-term skill acquisition was not studied; all studies only assessed surgical performance with between one and five procedures after the first feedback was provided. Thus, studies taking place over a longer time scale are necessary. It is also important to establish the clinical significance of feedback; none of the studies included assessed whether the improvement in technical skill was associated with an improvement in clinical outcomes, although one might suspect it would, particularly in light of recent findings that technical skills rated by experts based on video footage correlate with surgical outcomes.7

    In one study, feedback included a review of a videotape recording and a 60 min structured feedback session with a senior surgeon.15 Although extensive feedback sessions have been suggested (but not shown) to be effective,21 provision of feedback in this manner may be resource intensive and hence cost- and time-effectiveness must also be considered. The finding from a study involving simulation that non-expert delivered feedback is still effective14 may broaden options for educationalists and time-pressed senior surgeons, although one must be careful not to implement counter-productive feedback initiatives.

    Although there are a few studies on this subject, all studies included in our analysis were randomised controlled trials. Given the consistent benefit of feedback demonstrated, this supports further research on this topic and implementation of structured intraoperative feedback initiatives.

    The small number of studies included in this review highlights the need for more substantive research in this area in order to establish the optimum means and circumstances of feedback dissemination such that standardised methods for future widespread implementation can be attained, and future studies should consider the effect of the following study variables

    1. Source (oral/written), facilitator (expert/non-expert), frequency (every procedure/once daily/weekly/monthly) and duration of feedback (months/years).

    2. Surgeon involvement in feedback (either active or passive), standardised means of assessing surgical performance (which may be both generic and procedure-specific), content of feedback, timing of feedback relative to the procedure (intraoperatively/postoperatively), and the opportunities available for discussion, correction and learning.

    3. Benchmarking (relative to peers as well as to literature data) and feedback based on intraoperative recordings reviewed at a later time point.

    4. Other interventions utilised, such as guidelines, education and review of instructional videos. The contributions of these interventions, and the additive effect they may have with feedback on performance and outcomes are poorly understood.

    It should also be borne in mind that in some circumstances or when delivered inappropriately, feedback may not be effective; for instance, although a number of studies in medical students have found feedback to improve acquisition of basic surgical skills,8–10 some have failed to find this,22 and the effect of feedback may plateau.23 ,24 Frameworks have been suggested in order to ensure appropriate dissemination of feedback,25 which is particularly important given the fact that trainees often feel they are provided with inadequate feedback despite senior surgeons feeling their feedback provision is adequate.26 ,27

    In an era of increasing demands and scrutiny of surgeons in which surgical trainees are simultaneously spending less time in the operating theatre, methods to improve the efficiency of surgical performance are needed. The findings from this review suggest that feedback of intraoperative performance is an effective means by which this might be achieved; however, despite the potential impact, there is a paucity of research in this area, and further work is needed in order to establish the optimum circumstances and means by which feedback can be delivered in a time-effective and cost-effective manner.

    References

    1. ↵
      1. Tavare A
      . Where are we with transparency over performance of doctors and institutions? BMJ 2012;345:e4464. doi:10.1136/bmj.e4464
      OpenUrlFREE Full Text
    2. ↵
      1. Breen KJ,
      2. Hogan AM,
      3. Mealy K
      . The detrimental impact of the implementation of the European working time directive (EWTD) on surgical senior house officer (SHO) operative experience. Ir J Med Sci 2013;182:383–7. doi:10.1007/s11845-012-0894-6
      OpenUrlCrossRefPubMed
    3. ↵
      1. Kairys JC,
      2. McGuire K,
      3. Crawford AG, et al
      . Cumulative operative experience is decreasing during general surgery residency: a worrisome trend for surgical trainees? J Am Coll Surg 2008;206:804–11; discussion 811–3. doi:10.1016/j.jamcollsurg.2007.12.055
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      1. Sosa JA,
      2. Bowman HM,
      3. Tielsch JM, et al
      . The importance of surgeon experience for clinical and economic outcomes from thyroidectomy. Ann Surg 1998;228:320–30. doi:10.1097/00000658-199809000-00005
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. Carty MJ,
      2. Chan R,
      3. Huckman R, et al
      . A detailed analysis of the reduction mammaplasty learning curve: a statistical process model for approaching surgical performance improvement. Plast Reconstr Surg 2009;124:706–14. doi:10.1097/PRS.0b013e3181b17a13
      OpenUrlCrossRefPubMed
    6. ↵
      1. Birkmeyer JD,
      2. Stukel TA,
      3. Siewers AE, et al
      . Surgeon volume and operative mortality in the United States. N Engl J Med 2003;349:2117–27. doi:10.1056/NEJMsa035205
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. Birkmeyer JD,
      2. Finks JF,
      3. O'Reilly A, et al
      . Surgical skill and complication rates after bariatric surgery. N Engl J Med 2013;369:1434–42. doi:10.1056/NEJMsa1300625
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Rogers DA,
      2. Regehr G,
      3. Howdieshell TR, et al
      . The impact of external feedback on computer-assisted learning for surgical technical skill training. Am J Surg 2000;179:341–3. doi:10.1016/S0002-9610(00)00341-X
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Porte MC,
      2. Xeroulis G,
      3. Reznick RK, et al
      . Verbal feedback from an expert is more effective than self-accessed feedback about motion efficiency in learning new surgical skills. Am J Surg 2007;193:105–10. doi:10.1016/j.amjsurg.2006.03.016
      OpenUrlCrossRefPubMed
    10. ↵
      1. Van Sickle KR,
      2. Gallagher AG,
      3. Smith CD
      . The effect of escalating feedback on the acquisition of psychomotor skills for laparoscopy. Surg Endosc 2007;21:220–4. doi:10.1007/s00464-005-0847-5
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. Strandbygaard J,
      2. Bjerrum F,
      3. Maagaard M, et al
      . Instructor feedback versus no instructor feedback on performance in a laparoscopic virtual reality simulator: a randomized trial. Ann Surg 2013;257:839–44. doi:10.1097/SLA.0b013e31827eee6e
      OpenUrl
    12. ↵
      1. Harewood GC,
      2. Murray F,
      3. Winder S, et al
      . Evaluation of formal feedback on endoscopic competence among trainees: the EFFECT trial. Ir J Med Sci 2008;177:253–6. doi:10.1007/s11845-008-0161-z
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      1. Boyle E,
      2. Al-Akash M,
      3. Gallagher AG, et al
      . Optimising surgical training: use of feedback to reduce errors during a simulated surgical procedure. Postgrad Med J 2011;87:524–8. doi:10.1136/pgmj.2010.109363
      OpenUrlAbstract/FREE Full Text
    14. ↵
      1. Boyle E,
      2. O'Keeffe DA,
      3. Naughton PA, et al
      . The importance of expert feedback during endovascular simulator training. J Vasc Surg 2011;54:240–248 e1. doi:10.1016/j.jvs.2011.01.058
      OpenUrlPubMed
    15. ↵
      1. Grantcharov TP,
      2. Schulze S,
      3. Kristiansen VB
      . The impact of objective assessment and constructive feedback on improvement of laparoscopic performance in the operating room. Surg Endosc 2007;21:2240–3. doi:10.1007/s00464-007-9356-z
      OpenUrlCrossRefPubMedWeb of Science
    16. ↵
      1. Gurusamy K,
      2. Aggarwal R,
      3. Palanivelu L, et al
      . Systematic review of randomized controlled trials on the effectiveness of virtual reality training for laparoscopic surgery. Br J Surg 2008;95:1088–97. doi:10.1002/bjs.6344
      OpenUrlCrossRefPubMedWeb of Science
    17. ↵
      1. Dawe SR,
      2. Windsor JA,
      3. Broeders JA, et al
      . A systematic review of surgical skills transfer after simulation-based training: laparoscopic cholecystectomy and endoscopy. Ann Surg 2014;259:236–48. doi:10.1097/SLA.0000000000000245
      OpenUrl
    18. ↵
      1. Gofton WT,
      2. Dudek NL,
      3. Wood TJ, et al
      . The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): a tool to assess surgical competence. Acad Med 2012;87:1401–7. doi:10.1097/ACM.0b013e3182677805
      OpenUrlCrossRefPubMedWeb of Science
    19. ↵
      1. van Hove PD,
      2. Tuijthof GJ,
      3. Verdaasdonk EG, et al
      . Objective assessment of technical surgical skills. Br J Surg 2010;97:972–87. doi:10.1002/bjs.7115
      OpenUrlCrossRefPubMedWeb of Science
    20. ↵
      1. Vassiliou MC,
      2. Feldman LS,
      3. Fraser SA, et al
      . Evaluating intraoperative laparoscopic skill: direct observation versus blinded videotaped performances. Surg Innov 2007;14:211–16. doi:10.1177/1553350607308466
      OpenUrlAbstract/FREE Full Text
    21. ↵
      1. Middaugh DJ
      . The 2-minute warning part v: conduct a post-game analysis and press conference. Medsurg Nurs 2012;21:177–8.
      OpenUrlPubMed
    22. ↵
      1. Snyder CW,
      2. Vandromme MJ,
      3. Tyra SL, et al
      . Effects of virtual reality simulator training method and observational learning on surgical performance. World J Surg 2011;35:245–52. doi:10.1007/s00268-010-0861-1
      OpenUrlPubMed
    23. ↵
      1. O'Connor A,
      2. Schwaitzberg SD,
      3. Cao CG
      . How much feedback is necessary for learning to suture? Surg Endosc 2008;22:1614–19. doi:10.1007/s00464-007-9645-6
      OpenUrlCrossRefPubMedWeb of Science
    24. ↵
      1. Stefanidis D,
      2. Korndorffer JR Jr.,
      3. Heniford BT, et al
      . Limited feedback and video tutorials optimize learning and resource utilization during laparoscopic simulator training. Surgery 2007;142:202–6. doi:10.1016/j.surg.2007.03.009
      OpenUrlCrossRefPubMedWeb of Science
    25. ↵
      1. Ahmed M,
      2. Arora S,
      3. Russ S, et al
      . Operation debrief: a SHARP improvement in performance feedback in the operating room. Ann Surg 2013;258:958–63. doi:10.1097/SLA.0b013e31828c88fc
      OpenUrlCrossRefPubMed
    26. ↵
      1. Sender Liberman A,
      2. Liberman M,
      3. Steinert Y, et al
      . Surgery residents and attending surgeons have different perceptions of feedback. Med Teach 2005;27:470–2. doi:10.1080/0142590500129183
      OpenUrlPubMed
    27. ↵
      1. Jensen AR,
      2. Wright AS,
      3. Kim S, et al
      . Educational feedback in the operating room: a gap between resident and faculty perceptions. Am J Surg 2012;204:248–55. doi:10.1016/j.amjsurg.2011.08.019
      OpenUrlPubMed

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • Contributors AT, MM, AB-V were involved in the systematic search, manuscript composition, proofing. MJC, PM were involved in the conception, design, drafting, proofing.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement The search criteria and data extracted from the studies are available in the Appendices.