The impact of external feedback on computer-assisted learning for surgical technical skill training

doi:10.1016/S0002-9610(00)00341-X

The American Journal of Surgery

Volume 179, Issue 4, April 2000, Pages 341-343

Presented at the 19th Annual Meeting of the Association for Surgical Education, Boston, Massachusetts, April 7–10, 1999.

https://doi.org/10.1016/S0002-9610(00)00341-X Get rights and content

Abstract

Background: Computer-assisted learning (CAL) offers a number of potential advantages for surgical technical skills teaching. The purpose of this study was to evaluate the impact of individualized external feedback on surgical skill acquisition when a CAL package is used for instruction.

Methods: Freshman and sophomore students participated in a 1-hour CAL session designed to teach them how to tie a two-handed square knot. One group received individualized external feedback during the session and the other group did not. Subjects were videotaped performing the skill before and after the session. The tapes were independently analyzed, in blinded fashion, by three surgeons. Three measures were obtained: the total time for the task, whether or not the knot was square, and the general quality of the performance using a rating scale.

Results: Data from 105 subjects were available for final analysis. For both groups there were significant increases in the proportion of knots that were square when the posttest performance was compared with the pretest performance but there was no difference between groups on this measure. Comparison of the performance scores demonstrated that both groups had a significant improvement after the session but the performance scores were significantly better in the group that had received feedback.

Conclusions: Novices in both groups using CAL showed improvement in two of the outcomes measured, suggesting that subjects in both groups attained some degree of competence with this skill. The higher posttest performance score for the group receiving feedback demonstrates that external feedback results in a higher level of mastery when CAL is used to teach surgical technical skills.

Section snippets

Methods

Approval was obtained from the Institutional Review Board at the Medical College of Georgia. Freshman and sophomore medical students were enrolled in the study, and each was randomly assigned to either the CAL alone group or the CAL and feedback group. The subjects were not paid to participate in the study but were allowed to keep the knot-tying board, manual, and rope. Both groups used the CAL program that has been described previously.1 The subjects were placed in groups of seven or eight and

Results

One hundred and eight subjects were enrolled in the study. Data from 3 of the subjects were excluded because it was not possible to evaluate one of the performances, and so data from 105 subjects were available for final analysis. Fifty-four subjects were in the CAL with feedback group and 51 subjects were in the CAL alone group. Average age, gender distribution, and handedness were similar for both groups.

A two-way analysis of variance (ANOVA) evaluating the effect of time (pretest versus

Comments

The results of this study not only provide additional evidence for the importance of external feedback from a content expert in the instruction of basic surgical skills but they also provide some evidence for the effectiveness of CAL alone for this purpose.

The acquisition of motor skills is subject to continuing research and several theories have evolved. These theories have become divergent enough to be considered a different “schools.”2, 3 One unifying characteristic of these theories is the

References (7)

D.A. Rogers et al.
Computer-assisted learning versus a lecture and feedback seminar for teaching a basic surgical technical skill
Am J Surg
(1998)
H.H. Kaufman et al.
Teaching surgeons to operateprinciples of psychomotor skills training
Acta Neurochir
(1987)
J.G. DesCoteaux et al.
Learning surgical technical skills
Can J Surg
(1995)

There are more references available in the full text version of this article.

Cited by (96)

Application of mobile-based web app to enhance simple suturing skills of nurse practitioners
2023, Nurse Education Today
Suturing is a crucial clinical skill for nurse practitioners (NPs), but the effectiveness of traditional training methods (e.g., physical suture kits combined with video content) is low.
This study compared the effectiveness and usability of a mobile-based web app (MoWa) developed for NPs to learn simple suturing skills with those of traditional instructional video–based training.
The MoWa system utilizes mobile devices to simulate hands-on suturing and provides learning guidance and feedback to support self-learning with a physical suturing kit. Fifty-four suturing novices (NPs) were recruited as participants, divided into an experimental group (EG: 28 participants) and a control group (CG: 26 participants), and instructed to self-learn for 3 weeks. Learning effectiveness and system usability were evaluated through a pretest and posttest.
The EG exhibited significant improvements in learning outcomes, self-confidence, self-efficacy, and learning anxiety and expressed satisfaction with the MoWa system. Furthermore, the EG also considerably enhanced learning outcomes, self-efficacy, and learning anxiety compared to the CG, with no significant difference in self-confidence.
The MoWa system combined with deliberate practice is an effective strategy for supporting suturing skills training.
The Effect of Anonymity on Quality of Resident Feedback
2022, Journal of Surgical Education
Quality feedback is critical to facilitate better performance and quicker learning. However, faculty may be hesitant to provide quality constructive feedback, especially in written form, due to fear of retaliation. We evaluated the impact of faculty anonymity on the quality of faculty-to-resident feedback.
A retrospective review was undertaken of faculty evaluation of resident performance from 2017 to 2018, when evaluations were identifiable, compared to 2018-2019, when evaluations were anonymous. Evaluations included 27 individual items with Likert type scoring and 2 open-ended questions. Open-ended responses and overall performance were de-identified and scored by 2 reviewers independently using the task, performance gap, action scoring model. Comparisons between groups were performed with the Wilcoxon-Mann-Whitney test.
Tertiary Care Institution, University of Michigan, Ann Arbor, MI
415 resident performance evaluations were available for analysis with 251 in the identifiable group and 164 in the anonymous group.
The average composite score for the identifiable group was 105.2 and 103.4 in the anonymous group (p = 0.22). The effect size of the impact on composite score was small (Cohen's d 0.084, 95% CI -0.11-0.28). There was excellent inter-rater reliability. There were no differences between feedback groups for any of the 3 components of task, performance gap, and action model. While average scores for all 3 components were low, action scores were lowest.
Anonymity did not significantly impact faculty evaluations of resident performance. The quality of open-ended feedback on written evaluations was generally poor, especially in identifying actions for continued performance improvement. Additional mechanisms to improve feedback quality should be sought.
Objective Evaluation of Clock Face Suture Using the Objective Structured Assessment of Technical Skill (OSATS) Checklist
2022, EJVES Vascular Forum
Assessment of the quality of the final product (QFP) is critical in simulation training, such as the clock face suture (CFS) exercise that is used to assess trainees’ needle handling and suturing accuracy. Objective Structured Assessment of Technical Skill (OSATS) scores are the gold standard for the evaluation of trainees. The aim was to investigate variability in the use of OSATS checklists and to evaluate a semi-automatic method of suture analysis vs. OSATS scores.
Details of 287 CFSs performed by trainees during Fundamentals in Vascular Surgery examinations were collected. All were rated according to a seven item OSATS checklist, including QFP score and an overall score by one or two expert surgeons immediately after completion. Interassessor variability was assessed for the CFS that were assessed by two assessors.
In order to assess intra- and interassessor variability, 50 CFS pictures were chosen randomly and submitted to three expert surgeons to rate the QFP twice and to carry out a semi-automatic image analysis of each CFS and the estimated cumulative error (CE; mm) recorded. It was hypothesised that the CE correlates to OSATS checklist items or overall score. Variables were compared for correlation with OSATS results using a linear regression. A Pearson's test was used to confirm the proposed hypothesis.
Mean ± standard deviation overall score for the OSATS checklist was 20.61 ± 6.33. Inter- and intra-assessor correlation were statistically significant regarding OSATS checklist items. Both correlations presented a low coefficient of determination, indicating variability. The mean CE was 16.07 ± 4.84 mm, and the correlation between the QFP and CE was statistically significant, proving that CE is an objective metric by which to assess the QFP.
OSATS score demonstrated intra- and interassessor variability, although there was a significant correlation between scores. CE is an objective metric that is not subject to assessor subjectivity or interassessor variability and is correlated with the gold standard of evaluation.
Maximizing Learning in the Operating Room: Residents’ Perspectives
2021, Journal of Surgical Research
Few studies examine how residents can optimize their educational experience in the OR on their terms. This study aimed to examine residents’ perceptions of how learners can maximize their education in the OR.
Using constructivist grounded theory methodology, the authors conducted focus groups with general surgery residents, PGY1-5, followed by semi-structured interviews with attending surgeons from a single, academic medical center. Constant comparison was used to identify themes and explore their relationships. Theoretical sampling was used until saturation was achieved.
Residents and attendings participated. Two phases of OR learning were identified, intra-operative and inter-operative. Characters that made optimized learning included control, struggling, and reflection. Residents who practiced self-reflection with their experiences, and were able to articulate this awareness to attendings, felt the OR was an ideal learning environment. Attendings echoed similar findings.
 Providing residents with a method of maximizing OR learning is critical to postgraduate clinical education. Currently, observation passively morphs into active learning and eventually independent operating in the OR. However, residents who practice self-regulated learning, and are able to discuss their educational goals with attendings, seem to find the OR a better learning environment and progress to independence more quickly. This was echoed by practicing attendings. Providing residents with a generalizable, self-regulated learning framework specific to operative educational experiences could maximize learning potential and expedite resident progression in the OR.
Learning in the Operating Theatre: A Thematic Analysis of Opportunities Lost and Found
2021, Journal of Surgical Education
The operating theatre (OT) is an important learning environment. Trainees face barriers to learning in the OT that may reduce meaningful educational interactions. The impact of these barriers on the intraoperative learning experience of trainees and the strategies that they employ to overcome them are not known. This qualitative study aimed to describe the intraoperative learning experiences of senior general surgery trainees in Australia and their strategies to optimize learning in the OT.
The authors developed a semi-structured interview guide based on published literature. Purposive sampling was used to identify a representative group of general surgery trainees in Australia, who were interviewed in a private setting with audio recordings deidentified for verbatim transcription and analysis. Thematic analysis was conducted using an interpretivist approach to produce a coding framework.
Ten trainees participated in the study. Themes were divided into external and internal barriers to learning, promoters of effective learning and actions to facilitate learning. External barriers included cultural neglect of an important issue, with inadequate prioritization of teaching and a lack of structure for intraoperative learning. From this, we identified the theme of missed opportunities. Internal barriers included difficulties in developing assertiveness required to address these issues and a failure to adequately plan for learning, with reliance on the mentor to initiate. Actions to facilitate learning were rarely employed by trainees, as most were unaware of strategies to maximize intraoperative learning.
Trainees find the barriers to learning in the OT difficult to address and are not well acquainted with strategies that may allow them to maximize their learning.
Computer-based video training is effective in teaching basic surgical skills to novices without faculty involvement using a self-directed, sequential and incremental program
2021, American Journal of Surgery
Computer-based video training (CBVT) of surgical skills overcomes limitations of 1:1 instruction. We hypothesized that a self-directed CBVT program could teach novices by dividing basic surgical skills into sequential, easily-mastered steps.
We developed a 12 video program teaching basic knot tying and suturing skills introduced in discrete, incremental steps. Students were evaluated pre- and post-course with a self-assessment, a written exam and a skill assessment.
Students (n = 221) who completed the course demonstrated significant improvement. Their average pre-course product quality score and assessment of technique using standard Global Rating Scale (GRS) were <0.4 for 6 measured skills (scale 0–5) and increased post-course to ≥3.25 except for the skill tying on tension whose GRS = 2.51. Average speed increased for all skills. Students’ self-ratings (scale 1–5) increased from an average of 1.4 ± 0.7 pre-elective to 3.9 ± 0.9 post-elective across all skills (P < 0.01).
Self-directed, incremental and sequential video training is effective teaching basic surgical skills and may be a model to teach other skills or to play a larger role in remote learning.

View all citing articles on Scopus

View full text

The impact of external feedback on computer-assisted learning for surgical technical skill training

Abstract

Section snippets

Methods

Results

Comments

Am J Surg

Teaching surgeons to operateprinciples of psychomotor skills training

Acta Neurochir

Learning surgical technical skills

Can J Surg