Article Text
PDF
Abstract
Introduction The high incidence of unsafe anaesthetic care leads to adverse events and increases the burden on patient safety. An important reason for unsafe anaesthesia care is the lack of non-technical skills (NTS), which are defined as personal cognitive, social or interpersonal skills, among anaesthetists. The anaesthetists’ NTS (ANTS) behavioural marker system has been widely used to evaluate and improve anaesthetists’ behavioural performance to ensure patient safety. This protocol describes a planned systematic review aiming to determine the validity and reliability of the ANTS behavioural marker system and its application as a tool for the training and assessment of ANTS and for improving patient safety.
Methods and analysis This systematic review follows the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Protocol. Studies that applied the ANTS behavioural marker system in a meaningful way, including using the ANTS behavioural marker system to guide data collection, analysis, coding, measurement, and/or reporting, which have been published in peer-reviewed journals, will be eligible. A citation search strategy will be employed. We will search Scopus and Web of Science for publications from 2002 to May 2022, which cite the three original ANTS behavioural marker system publications by Fletcher et al. We will also search the references of the relevant reviews for additional eligible studies. For each study, two authors will independently screen papers to determine eligibility and will extract the data. The quality of the included studies will be assessed using the Joanna Briggs Institute (JBI) Critical Appraisal Checklists. A framework analysis approach that consists of five steps—familiarisation, identifying a thematic data extraction framework, indexing, charting, mapping and interpretation—will be used to synthesise and report the data.
Ethics and dissemination Ethics approval is not required for this study. The findings will be disseminated primarily through peer-reviewed publications and conference presentations.
PROSPERO registration number CRD42022297773.
- adverse events
- adult anaesthesia
- health & safety
- risk management
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
STRENGTHS AND LIMITATIONS OF THIS STUDY
To the best of our knowledge, no previous systematic review has illustrated the validity and reliability of anaesthetists’ non-technical skills (ANTS) behavioural marker system and its application as a tool for training and assessment of NTS and for improving patient safety.
In this systematic review, a citation search strategy will be employed to guide a comprehensive literature search.
We will apply the Standards for Educational and Psychological Testing (the Standards) framework to various study designs to evaluate and analyse the reliability and validity of the ANTS behavioural marker system.
Our results may be limited by reporting bias, the heterogeneity of the included studies and the exclusion of grey literature.
The systematic review will consider studies published only in English, which may exclude potentially related articles published in other languages.
Introduction
The incidence of adverse medical events is high. It has been estimated that adverse events resulting from unsafe care may be among the top 10 causes of death and disability worldwide, resulting in 64 million disability-adjusted life years lost worldwide each year.1 A previous systematic review and meta-analysis found that approximately 1 in 20 patients suffered preventable harm in medical care.2 Statistics show that, in high-income countries, up to 1 in 10 patients suffer from a series of adverse events while receiving hospital care, and nearly 50% of these are considered preventable.3 About two-thirds of adverse events occur in low-income and middle-income countries (LMICs), resulting in 2.6 million deaths per year.4 Mazer assessed preventable medical errors as the third leading cause of death in the USA.5
The economic burden created by adverse events and particularly preventable adverse events is substantial.6 Recent evidence suggests that 15% of total hospital expenditures and activities in Organisation for Economic Co-operation and Development countries were a direct result of adverse events, with the total cost of injuries in these countries alone amounting to trillions of dollars per year.7 The annual cost associated with medical errors is estimated at US$42 billion worldwide.8 The WHO has indicated that unsafe surgical care procedures can lead to complications in up to 25% of patients globally, with nearly 7 million surgical patients suffering serious complications each year and 1 million dying during or immediately after surgery.9 Although the number of deaths related to surgical complications has decreased over the past 50 years, due to improvements in patient safety measures, it remains two to three times higher in LMICs than in high-income countries, indicating a need for further attention.
The core causes of surgical adverse events have been shown to be human factors; specifically, non-technical skill (NTS) failures affect surgical team performance, rather than a lack of technical expertise. Kirschbaum et al reported that more than 25% of operating room communication failures result in procedural errors.10 Engelfound that limited perioperative information sharing doubled the risk of surgical complications as compared with procedures in which team members frequently shared information.11
As a high-risk facet of the surgical medical system, anaesthetists have been increasingly required to assume greater accountability and professional responsibility for patient safety in recent years. Surveys have shown that up to 80% of anaesthetic incidents are caused by gaps in NTS, such as inadequate monitoring, inadequate communication and not checking machines or double-checking drugs.12 Studies among anaesthesia residents have indicated that 79% of professionals felt that work pressure influenced their health and that such stress contributed to lower productivity, while 28.2% of professionals reported making errors in anaesthesia due to fatigue.13 14 It is worth mentioning that medical education institutions in both the USA and Canada consider NTS, such as interpersonal skills, communication skills, leadership skills, collaboration skills and situational awareness, as core competencies for anaesthetists.15 This has led to the development of training in NTS in anaesthesiology to improve patient safety. Therefore, evaluation and training of anaesthetists are essential to promote patient safety and improve the quality of the medical environment.16
Study rationale
Drawing on experience from the aviation industry, where the concept of NTS originated, Fletcher et al at the University of Aberdeen (UK) developed and tested the anaesthetists’ NTS (ANTS) behavioural marker system, in 2002, which can be used for training and evaluating anaesthetists’ performance.17 The ANTS behavioural marker system is a framework that includes four skill categories necessary for good anaesthesia practice, including task management, team working, situation awareness and decision-making, with 15 component elements, and examples of good and bad behaviours for each element.18 19 It uses a set of 4-point rating scales for rating observed behaviours in relation to the elements and categories and also provides space to write brief free-style comments.18 19
Rigorous testing showed that the ANTS behavioural marker system has a reasonable level of accuracy and reliability.18 Researchers have investigated the effect of interventions to improve ANTS, such as use of cognitive aid, mental practice and simulation interventions.20 21 In this context, measuring non-technical performance is paramount to ensuring the provision of safe, high-quality care.
Overall, as a high-risk profession in medical units, anaesthetists are bound to improve their NTS to ensure patient safety. Although the ANTS behavioural marker system has been widely applied as an assessment tool, its use appears to have escalated since its publication, and no formal reviews have been conducted to investigate its impact on the performance of medical personnel and patient safety.
Therefore, we intend to perform a systematic review aimed at synthesising evidence on the development and applications of the ANTS behavioural marker system as a tool for training and assessment of NTS in order to improve patient safety. Such a review will strengthen the anaesthesia-related personnel’s understanding of the ANTS behavioural marker system and improve their attention to educating, training, evaluating and applying the ANTS. These findings will further help to provide ideas for future research. In addition, the review will provide a basis for the theoretical and practical development of NTS across disciplines and teams, as multidisciplinary collaboration in healthcare continues to be emphasised.16 Findings from this systematic review can also be used as a reference for other profiles or researchers in other disciplines, as they integrate the ANTS behavioural marker system into their work.
Review objectives
The specific research objectives are as follows:
Objective 1: To determine the types of studies that develop and apply the ANTS behavioural marker system.
Objective 2: To determine the reliability and validity of the ANTS behavioural marker system.
Objective 3: To determine how the ANTS behavioural marker system has been applied, including the depth of application.
Objective 4: To determine the contribution of the ANTS behavioural marker system to improving patient safety.
Methods and analysis
This study protocol was submitted to PROSPERO (CRD42022297773). This systematic review will be conducted in accordance with the Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) Protocols checklist (see online supplemental file 1). The results of the literature search and screening will be presented in a PRISMA flow diagram. The systematic review will be performed between 1 July and 31 December 2022, in seven stages, including identifying the relevant studies, developing search strategies, screening and selecting studies, assessing study quality, collecting and extracting data, analysing data, and summarising and reporting the results.
Supplemental material
Study eligibility criteria
We will include all English language studies published in peer-reviewed journals between 2002, when the ANTS was first published by Fletcher et al,17 and May 2022, which meet the following criteria.
Population
We will include all anaesthetists and other healthcare profiles or disciplines.
Intervention
The intervention will involve using the ANTS behavioural marker system in a meaningful way, including the use of the ANTS behavioural marker system to guide data collection, measurement, coding, analysis and/or reporting.
Comparator
A comparator is not applicable.
Outcomes
The main outcomes of the study and other reported ANTS-related outcomes include the following:
For studies in which the ANTS behavioural marker system was used as a tool for training and assessment of NTS, the outcomes will be the ANTS scores.
For studies in which the ANTS behavioural marker system was used as a tool for improving patient safety, the outcomes will be the completion of the checklist, frequency of adverse events and frequency of errors.
Study design
We will include empirical studies, including quantitative studies, such as randomised controlled trials, quasiexperimental studies, cohort studies, case series and qualitative studies.
We will exclude protocols, editorials and articles that cited the ANTS, but which do not report the development and application of the ANTS behavioural marker system; for example, studies in which the ANTS behavioural marker system was cited in the introduction section to acknowledge the importance of human factors.
Search strategy
We will employ a citation search strategy to identify published peer-reviewed articles that describe the use of the ANTS behavioural marker system to guide their research.22 The cited articles used for our search were the original ANTS behavioural marker system publications by Fletcher et al.17–19 Two citation index databases, Web of Science and Scopus, will be searched. These databases were chosen because they provide the most comprehensive databases of articles that can be tracked using citations and allow for reference searches.23 Although other databases, such as Google Scholar, may provide wider coverage of certain types of publications (international, non-English journals, conference proceedings), these publications will be excluded from our review.24 In addition, the literature showed that Web of Science and Scopus yielded more consistent and accurate results than other databases that may provide wider coverage (eg, Google Scholar).25 We also searched the reference lists of previously published systematic reviews for additional relevant references.
In Scopus, the search string was REF (‘The role of non-technical skills in anaesthesia: a review of current literature’ AND ‘Anaesthetists’ Non-Technical Skills (ANTS): evaluation of a behavioural marker system’ AND ‘Rating non-technical skills: developing a behavioural marker system for use in anaesthesia’). In Web of Science, the search string was TITLE: (‘The role of non-technical skills in anaesthesia: a review of current literature’ AND ‘Anaesthetists’ Non-Technical Skills (ANTS): evaluation of a behavioural marker system’ AND ‘Rating non-technical skills: developing a behavioural marker system for use in anaesthesia’). The full texts of all resulting articles will be downloaded for review. The full search strategy is described in online supplemental file 2.
Supplemental material
Study selection process
The study selection will be performed independently by two authors (JK and CY). Thereafter, the two authors will screen the methods and results sections of the papers to determine whether the full-text manuscript should be further reviewed. We will also screen the reference lists of published reviews for additional relevant studies. There will be no initial screening based on titles or abstracts because this review is based on citations of the ANTS behavioural marker system, which is not always mentioned in those sections. Studies identified for full-paper screening will be searched for evidence of meaningful use of the ANTS behavioural marker system. Any discrepancies arising between the two authors during study selection will be resolved by discussion to consensus or discussion with a third review author (JH).
Quality assessment
We will use the JBI Critical Appraisal Checklists to assess the quality of the literature. Two review authors (JK and CY) will qualitatively assesse all studies. Any differences in opinion regarding quality will be resolved by consensus or discussed by a third author (JH).
Data extraction and analysis materials/tools
Two independent authors (JK and CY) will use a data extraction form to extract information in duplicate, with disagreements resolved via discussion to consensus or by discussion with a third author (HJ), as needed. The information extracted based on our framework analysis matrix format will include the following:
Study identification: The name of the first author and the year of publication.
Study description: Geographic location (eg, name of the region/province/city), study objective, duration, setting (eg, department or scenario), study design (eg, observational, experimental, or mixed-design study), methods (eg, qualitative, quantitative, or mixed-method study), and length of retention test.
Participant description: Unit of analysis (eg, individual or organisation), participants, and sample size.
Intervention details: Intervention method, timing of intervention, and phase of intervention (eg, design, evaluation, or implementation phase).
Psychometric testing evaluation: The ANTS behavioural marker system includes reliability and validity according to the Standards for Educational and Psychological Testing (the Standards) framework.26 Reliability evaluation will include stability, internal consistency and inter-rater reliability. Validity evaluation will include content, response processes and internal structure validity.
The ANTS behavioural marker system application includes the depth of its application: Selection and use of the ANTS behavioural marker system level (eg, category level, element level or both), application of the ANTS behavioural marker system in the methods (eg, qualitative, quantitative or mixed methods study) and how it was used (eg, to guide data analysis, data collection, data measurement, data coding or data reporting), raters’ details (which are important for applying the ANTS system, including the number of raters, whether or not they were trained before assessment, their skill level and the way of rating).
Outcomes: The main outcomes of the studies, the ANTS scores and the completion of the system, frequency of adverse events and frequency of errors.
Statistical methods.
Data analysis
We will use a framework analysis approach to guide the summarisation and synthesis of the included articles.27 The study team will develop a standardised data abstraction tool in Microsoft Excel, where content is arrayed in a matrix format consisting of rows (articles), columns (codes) and cells (summarised data), involving the steps outlined below.
The first step will be the familiarisation phase. Members of the research team will review the included studies to familiarise themselves with the literature. Second, we will identify a thematic data extraction framework based on our four research objectives to identify and extract content from the included studies. The development of the ANTS behavioural marker system, including its reliability and validity, is in accordance with the Standards for Educational and Psychological Testing (the Standards) Framework for data extraction. The quality and the depth of the ANTS behavioural marker system that has been applied will be assessed by the inclusion criteria for this systematic review, that is, these studies used the ANTS behavioural marker system in a meaningful way and research objective 3, which focuses on the depth of application. Next, in the indexing and charting phase, two authors (JK and CY) will extract text selections from the included studies into a summary matrix. Two authors will independently code, index and chart the included studies, comparing results. Differences will be discussed until a consensus is reached. We will chart at least four matrices based on these four research objectives.
In the final mapping and interpretation phase, the contents from the abstraction matrix will be analysed by JK, JH and FZ to develop overarching themes for each code. Themes will be discussed among all the coauthors until a consensus is reached. Table 1 lists the final codes for the analysis included in the research objectives.
The codes for analysis included in the research objectives
Patient and the public involvement
The patient and public will not be involved in designing, conducting, reporting or disseminating this study.
Discussion
Unsafe anaesthesia care due to the lack of NTS has led to many adverse events and a significant burden on patients. As a high-risk profession in medical units, anaesthetists are bound to improve their NTS to ensure patient safety. In healthcare, the ANTS behavioural marker system was first created and developed to evaluate the NTS of anaesthetists specifically. The proposed review will employ a systematic approach to explore the current development and application status of the ANTS behavioural marker system.
This systematic review will provide anaesthesia-related personnel with a better understanding of the latest development and application status of the ANTS behavioural marker system and will improve their attention to educating and training professionals in the use of the ANTS, and in evaluating and applying the ANTS. The results obtained are expected to provide ideas for future research. First, the ANTS behavioural marker system itself does not distinguish between the types of NTS needed in different scenarios. Anaesthesia work covers the entire perioperative period; thus, in future, lessons should be drawn from the ANTS to guide utilisation skills related to preoperative evaluation, formulation of anaesthesia plans, acting as a consultant to other clinicians, performing invasive procedures, postoperative care and pain management.19 In addition, simulation teaching has proven to be useful for improving NTS.28 Improved application of the ANTS to the formative and summative evaluation of NTS in a clinical environment has continuously been considered. Finally, the findings of the proposed review could also be used as a reference for other profiles or researchers from other disciplines as they integrate the ANTS behavioural marker system into their work.
Limitations
Some unavoidable methodological limitations must be recognised and considered to make an appropriate interpretation of the study findings and finally arrive at a supported conclusion. First, in this systematic review, we will define reporting bias as the comprehensiveness of reliability and validity reported according to the Standards framework.29 Descriptive analysis will be conducted to assess the reporting bias of the included studies. Second, the studies included according to our eligibility criteria are likely to be heterogeneous in their study designs, participants, interventions and outcomes. Despite the clinical and methodological diversity this causes, we will apply the Standards framework applicable to various study designs to evaluate and analyse the reliability and validity of the ANTS behavioural marker system.26 In addition, our study will only include research articles published in English. It will not include other research outputs, such as unpublished papers, grey literature and conference abstracts. Citation search is a method of finding relevant research in a field or subject by considering what has been referenced in an article and who has since used the article in question as a reference.30 Moreover, the databases we chose, that is, Web of Science and Scopus, provide the most comprehensive databases of articles that can be tracked using citations and they allow reference searches.23 30
Ethics and dissemination
This study will not involve human participants or unpublished secondary data. Ethical approval will not be required for this study. The findings will be disseminated through peer-reviewed publications and conference presentations.
Ethics statements
Patient consent for publication
Acknowledgments
The authors acknowledge the support of the Jiangsu Province Phase VI 333 high-level talent-training project. We would like to thank Editage (www.editage.com) for English language editing.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
JK and JH are co first authors and contributed equally .
ST and FZ are co-corresponding authors and contributed equally.
Contributors All the authors designed the project. JK and JH developed the search strategy. JK and CY will be responsible for searching the literature and managing the preliminary data. JK and JH drafted this manuscript. FZ, ST, XX, JH and JK reviewed, revised and edited the manuscript. All authors provided feedback on the manuscript and approved the final manuscript.
Funding The proposed study will be supported by the Jiangsu Province Phase VI 333 high-level talent-training project. No grants were provided. The contents of this paper are solely the responsibility of the authors and do not necessarily represent the official views of Jiangsu province.
Competing interests None declared.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.