Article Text
PDF
Abstract
Objectives To investigate agreement and potential differences in the application and interpretation of the definition among surgical departments of various hospitals.
Design 24 cases were formulated including general, trauma, gastrointestinal and vascular surgery, and based on points of discussion about the definition and ambiguities regarding complication registration as encountered in daily practice. The cases were presented to the surgical staff and residents in seven Dutch hospitals, using the national registration system of complications and an electronic response system.
Results In total, 134 participants responded. Interpretation differences were particularly found regarding: (1) complications considered as logical consequences of a surgical procedure; (2) complications occurring after radiological interventions; (3) severity criteria such as when to consider a complication as a ‘(probably) permanent damage or function loss’; (4) registering a cancelled operation as a complication and (5) patients with serial complications during hospital stay.
Conclusions The definition of surgical complications as currently applied in the Netherlands does not ensure a uniform complication registration. Improvement of this registration system is mandatory before benchmarking of these findings in the public domain is appropriate. Modifications of the current definition of a surgical complication, and improved consensus about specific clinical situations and training of surgeons might improve the quality of benchmarking.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
Arbitrary choices of clinical problem situations for the cases.
Representative of all hospitals outside the Netherlands.
The participating hospitals included a mix of the different hospital types: university medical centres, tertiary and general hospitals.
Global issues of reliability of benchmarking of hospital data and the issue of differences in interpretation of definitions are addressed.
Introduction
The trend to develop national benchmarking data, including those regarding complications suffered by surgical patients during their hospital stay or shortly after discharge, is ongoing. For example, national benchmarking by the NSQIP institutions (the American College of Surgeons National Surgical Quality Improvement Program) appears to be improving morbidity and mortality over time.1 In order to obtain high-quality benchmarking data, it is necessary to correct for under-reporting of complications and for differences in case mix as well as in the level of complexity of the interventions.2 The validity of benchmarking data also depends on the quality control of these data.3 ,4 High-quality data require reliable and uniform registration by the participating surgical departments. This includes that, at least for identical situations, all hospitals should register the same complications with the same degree of severity. Santford et al already showed that variations in definition and methods of retrieval greatly influence what is rated as a complication in patients undergoing a pancreatoduodenectomy.5 This is especially true for complications of a lower severity level.
Other studies have shown that there is still variability about the definition of a complication or regarding the interpretation of this definition.6–8 Should we define a complication as an undesirable event following surgical medical care? By this definition, an operative scar would also be a complication.6 Or do we perhaps consider a complication to be an unexpected result? Is damage of an intra-abdominal organ—for example injury to the spleen during pancreatic surgery—a complication, or only if this negatively affects the patient outcome, for example, when an accidental splenectomy is performed and the patient has to follow a vaccination programme?
In the Netherlands, the currently used definition of a surgical complication consists of three essential components (specified in 1999 by the Association of Surgeons of the Netherlands (NVvH) and the Dutch Association of Medical Specialists).9 ,10
A complication is an unintended and undesirable event or state that:
Occurs during or following a medical specialist intervention that negatively affects the patient's health such that this requires their medical treatment to be adapted, or such that irreparable damage is caused;
Is established either during in-hospital treatment or during immediate follow-up, up to a period of 30 days after discharge;
Is the result of the actual medical specialist intervention. The chances of the complication occurring and the presence or absence of culpability are not relevant.
Registration of complications has been regulated by the Association of Surgeons in the Netherlands since 1999 and is accepted in all hospitals in the Netherlands. The focus of the present study was to investigate whether complication data in the Netherlands can be compared among surgical departments. Although the association of Surgeons of the Netherlands specified a national definition of a complication, it remains unclear whether this definition is interpreted and applied in the same way. Therefore, we investigated the agreement in the registration of complications within and among the surgical departments of hospitals in the Netherlands.
Methods
Example cases
An inventory study was carried out in a convenience sample of seven hospitals. Two surgeons formulated 24 cases based on critical points of discussion, definitions and ambiguities regarding the registration of complications taken from their experience during complication registration from daily practice (see online supplementary appendix).
The questions were divided into the following six main categories. The specifications of the categories enabled us to be more specific about potential points of improvement, while realising that some cases could be included in more categories.
Definition: Whether this is a complication according to the definition of a ‘complication’ as defined by the Association of Surgeons of the Netherlands.
Other specialty: Whether complications of a patient admitted at the surgical department were included in the registry if these occurred as a result of another specialty but within the well-defined postoperative period of registration. Part 2 of the definition describes the period of responsibility for the surgeon during admission and the 30-day period after discharge. For example, if the patient had to be admitted at the intensive care unit (ICU) after surgery or treated by an interventional radiologist during surgical admission, the surgeon is still responsible for registering complications.
Severity: Determining the grade of severity of the complication, categorised using a four-level grade scale based on the Clavien and Dindo grading system11; Severity (0) temporary health disadvantage without treatment; (1) recovering without (re)operation; (2) recovery after (re)operation; (3) (probably) permanent damage or function loss and (4) death.
Intraoperative damage: Whether complications that occurred intraoperatively were registered.
Cancellation of operations: Whether physicians registered cancelled operations as a complication. Independent of the reason for cancellation, such as other emergency operations. The cancellation of an operation meets the criteria of a complication if this unintended and undesirable event requires medical treatment to be adapted, or if irreparable damage is caused (part 1 of the definition).
Serial complications and transfers: The registration of complications in patients with severe and serial medical problems, and those transferred from other hospitals. The issue here is whether the surgeon is responsible for registering these medical problems and the following (serial) complications during the responsibility period (part 2 of the definition).
Each category was represented by at least three questions. Since some cases were relevant for several categories, these cases were also assigned to several categories and analysed as such.
Procedure
Medical professionals (surgeons, fellows and residents) working at the surgical departments of seven hospitals participated in the study. These hospitals included two university medical centres, four tertiary referral hospitals and one general teaching hospital. The cases were presented to the members of the surgical staff and residents at a time interval approved by the different departments (eg, at the end of a daily change of shift, or during a session on complication registration or research meeting). The 24 cases were presented in random order in the format of a multiple choice quiz. The responses were registered using electronic voting devices (Turning Technologies LLC, Youngstown, Ohio, USA).
The participants were first asked about their position (attending surgeon, fellow, resident) and specialty/subspecialty (gastrointestinal/oncology, vascular surgery, trauma surgery or not applicable). The approved definition of a complication was not shown to the participants before the session and they were not allowed to ask any questions for clarification during the presentation of the cases. The potential responses to the 24 example cases were either dichotomous or categorical. Participants were given 10 s to respond to each case and the time available was shown on a screen. The number of participants who voted for each case was recorded.
Data analysis
Data was analysed for each hospital, per case and per category. The dichotomous answers were used to calculate the proportion of participants (in per cent) who responded to the case with ‘yes, I register this as a complication’ and the total number of participants for that case. Proportions close to 100% were defined as unanimity in the interpretation of a particular case as a complication; the same applied for numbers close to 0% for cases not being considered as a complication. For each question, the average (with its range) percentage of ‘yes’ responses was calculated over all hospitals, weighted for the number of participants per hospital. The results of the example cases that had a categorical range of responses were analysed separately. Analysis of responses related to function (staff vs residents) were performed by χ2 test. A p<0.05 was considered statistically significant.
Results
General characteristics of participants and hospitals
The number of participants in the seven hospitals was 134. The response rate was nearly 100% because all staff and residents present participated. The appendix shows the number of responses per case. Cases 2 and 15 were the first and last cases presented to the departments (response rate: 81% and 69%, respectively). The reason for the lower response rate in these cases can be explained by the fact that some surgeons arrived later at the meeting or had to leave earlier due to other scheduled activities.
The distribution over different functions and subspecialties is shown in table 1. More than 50% of participants practiced at a university medical centre, almost 40% in a tertiary referral hospital and around 10% in a general training hospital. About 40% of the participants were attending surgeons. The largest subspecialty was gastrointestinal oncology, represented by almost 35%, while 25% of participants indicated not having any specific subspecialty.
Participants per hospital
Results per category
Category 1: Definition
Figure 1 shows the percentage of ‘yes’ responses per hospital in the category ‘Definition’. For 6 of 16 example cases (figure 1; cases 1 through 6), the agreement between hospitals was more than 80% on average, whereas agreement in case 16 was below 20%. For some of the other cases, either the variation among hospitals was extremely high, ranging from 9% to 100% in case 15 (figure 1), or there was no agreement within hospitals (range 18–58%), as shown for case 13.
Agreement within and between hospitals: ‘definition’ category. The questions are ordered from the highest to the lowest average percentage of ‘yes’ responses per case per hospital.
The highest agreement was found for complications such as postoperative wound infections or anastomotic leaks. The lowest agreement was found in cases with complications that might often directly be related to the surgical procedure, such as gastroparesis after a gastrectomy or ongoing bowel paralysis following adhesiolysis.
Category 2: Complication related to other specialty
For 2 of the 3 cases in this category, cases 17 and 18 (figure 2), a ‘complication in the ICU’ and ‘complication on a non-surgical nursing ward’, 98% of participants agreed that both cases should be registered as a complication (ranges 70–100% and 83–100%, respectively). On the contrary, a groin haemorrhage following percutaneous intervention by a radiologist (figure 2: case 19) was reported as a complication with a limited variation ranging from 50% to 82% of the participants (figure 2).
Agreement within and between hospitals: other categories. The questions are ordered from the highest to the lowest average percentage of ‘yes’ responses per category per case per hospital. The cases can appear in more than one category.
Category 3: Severity
We also found differences in responses with regard to the severity assigned to a complication (categorical variables not shown in figure 2, table 1: cases 3, 8, 10, 14, 23, 24). A complication that occurs during surgery but that is repaired during that same operation would generally not be registered as a complication with severity grade: “recovery after (re)operation” (case 14; average 0%, range 0–14.3%). In two cases (3 and 14) participants were asked whether a complication would be registered with a severity grade “(probably) permanent damage or function loss.” The percentage of participants who judged this as correct varied per hospital (17–62% for case 14 and 67–100% for case 3). The construction of an intentionally temporary ileostomy performed during a surgical intervention after a complication (case 24) was considered to be registered with severity “(probably) permanent damage or function loss” by only 0–41% of the participants.
Category 4: Intraoperative damage
Damage to the spleen (requiring splenectomy; figure 2, case 3), followed by a vaccination programme for the patient, was considered by an average of 95% of participants (range 86–100%) as a complication. However, only 32% of surgeons would register damage during a surgical procedure, such as an accidental intestinal perforation (figure 2, cases 14) with subsequent closure of the defect, as a complication (range 0–50%).
Category 5: Cancelled operations
Whether or not a cancellation of an operation is registered as a complication varied between participants and hospitals. Cancellation for medical reasons (case 22) would be registered as a complication by 0% to 40% of the participants. If the reason for cancellation was identified during the ‘time out’ procedure, this percentage was higher; 25–93% (case 20). Operations cancelled due to logistic reasons, for example, due to the urgency of other emergency surgery patients (figure 2, case 21), showed a variation among hospitals (range 8–80%).
Category 6: Serial complications and transfers
On average, more than 70% of the participants would register one or more complications (including during the further clinical course) if a patient with complications had been transferred from another hospital (figure 2, case 9; range 55–86%). Of all participants, an average of 55% would not register existing complications on admission, but would register any subsequent complications that occurred during hospitalisation in the receiving hospital (not shown). In the cases with serial complications, about half of the participants (range 25–73%) would register all complications during hospitalisation, while the other half (range 27–67%) would register only some of them (not shown).
Staff versus residents
Responses to 19 cases showed no significant differences between staff and residents, whereas three cases (1, 8 and 10) did show significant differences in responses. Staff would register a case of hypocalcaemia after thyroidectomy significantly more often as a complication than would residents (case 8; p=0.002), as well as postdischarge abdominal pain after a laparoscopic colectomy (case 10; p=0.015). Finally, residents would register more complications after hemicolectomy (case 1; p<0.001).
Discussion
Despite a uniform definition for surgical complications, the present study showed there is limited consensus both among and within hospitals as to which event should be considered as a complication and should therefore be registered, which is a pre-requisite for adequate hospital benchmarking. This is particularly important in the current era of reporting and comparing the quality of healthcare, for example, using Hospital Mortality Ratios such as the HSMR (hospital standardized mortality ratio),12 ,13 or the national and international complication registrations for heart surgery in adults (LCRHV; http://www.nvtnet.nl), or the NSQIP.1 ,5
The present study showed enormous differences in the use of the current definition of a complication. In order to improve uniform interpretation, three different aspects of the definition might require revision.
First, surgeons could consider some results of care to be ‘calculated risks’.14 Based on the findings in this study, a result should be registered as a complication only if this result is undesirable for the patient and negatively affects the patient (eg, vaccination following accidental splenectomy).6
Second, this study found limited consensus as to registering complications related to other specialties. Despite this divergence, working in multidisciplinary teams has become increasingly more important in healthcare.15 Some years ago, the report entitled “To err is human” also argued in favour of teamwork, a concept that might be able to prevent a large number of avoidable complications.16 For example, gastrointestinal (GI) surgeons form a multidisciplinary GI oncology team with gastroenterologist, medical oncologists and radiologists, or, vascular surgeons with interventional radiologists, trauma surgeons with orthopaedic surgeons and neurosurgeons in a trauma unit. A more consistent registration of all complications is advocated, meaning that all complications developed under the responsibility of the surgical department should be registered, regardless of which specialty is responsible.
Third, although complications might indicate something about the results of care, they do not inform about the process or any underlying, unintended incidents. Complication registration provides better awareness of the actions of individuals or departments and of trends in complications.17 The definition should therefore be applied as literally as possible, without interpretation or desire for self-protection. These three aspects should be added to the three parts of the definition; part 1: undesirable result for the patient; part 2: all complications under responsibility of the Surgical Department; and part 3: without interpretation or self-protection.
We should only consider whether or not the results were avoidable in retrospect. For such complications we can refer back to the processes.18 Results of a previous study suggest that differences in interpretation of definitions might be more important than the differences in the definition itself.19 Even if the same way of reviewing medical records and definition of complication is used, important differences in complication rates may occur.20 This study describes several cases that call for agreement among surgeons. For example the impact of serial complications should be addressed.21 ,22 Several studies describe extensive training in the use of the complication registration, resulting in better patient outcomes over time.23–25 Educating and training surgeons to familiarise themselves with the definition, and encouraging them to acquire knowledge about national agreements with regard to specific situations, may help achieve a more uniform registration.
Strengths and limitations
This study used real-life situations from daily clinical practice in one country to show that there are clear judgement differences between surgeons, which demonstrates that there is room for improvement in complication registration. Because the choice of clinical problem situations for the cases was arbitrary, some problem situations may well have been left out. However, this would not have changed the main conclusion of the study. For some complications, the discussion remains regarding whether or not they should be considered as permanent (eg, in the case of vocal cord paralysis or an ileostomy intended to be temporary), because it is not known beforehand. For intraoperative complications it is unclear whether these should be considered as a re-operation.26 Furthermore, one could argue whether the seven participating hospitals were representative of all hospitals in the Netherlands. Nevertheless, the participating hospitals did include a mix of the different hospital types: university medical centres, tertiary and general hospitals.
The national surgical complication registration used in the Netherlands since 1999 to register surgical complications contains all complications of the department of surgery (general, vascular, trauma and gastrointestinal). The residents of the department register the complications during hospitalisation and after discharge. These patients and their complications are plenary discussed during hand-offs by senior surgeons and residents. Therefore we chose the mixed sample of specialties, as these were all involved in the registration process. The different grades were also important, to assess possible differences in grading interpretations among the surgeons. Given the fact that the whole team of surgeons and residents attending the hand-offs was involved in the registration, they were all included, which added up to the numbers described in this study.
Nowadays, academic hospitals seem more subspecialty driven, but this represents only a small part of all surgical care. The smaller hospitals do not have specialised units as most university hospitals. Most subspecialties indeed have their specific registration system, such as the national audits for colorectal, pancreatic and oesophageal surgery. A generic registration still is of importance for general departments of surgery in these smaller hospitals. We also do realise this might change in the next decades. Moreover, complication registration is an outcome-driven registration. It enables us to review trends in complications, such as an increasing postoperative infection rate. These trends should be reviewed and analysed on the higher level of general surgery because the processes or actions for improvement may transcend the subspecialty.
Finally, nowadays, benchmarking criteria may not be based of self-reported outcomes but on outcome data gathered by ‘coders’. Coders may take over the surgeons’ task of recording complications, for example, using trigger tools. This will be even more important in the multidisciplinary units in the future. The agreement between surgeons and coders is the first step towards benchmarking. To this end, interpretation differences regarding specific clinical situations should be reconciled and regulated by the professional society first.
Conclusion
Given the considerable differences in interpretation of the current definition of a complication, it is unlikely that uniform registration of complications is actually possible. This uniformity may be increased by additions to the current definition, by more agreement about specific clinical situations, and by training of surgeons, thereby improving comparisons at local as well as national levels. This seems a pre-requisite before such data can be used at the public domain and function as one of the parameters for the quality of healthcare.
Acknowledgments
The authors would like to thank all surgeons, fellows and residents of the surgical departments of the seven hospitals that participated in the study. They also thank Anne van Wijngaarden for assisting in the data collection.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
- Data supplement 1 - Online supplement
Footnotes
Contributors AV contributed to design, acquisition, analyses and interpretation of the data. DTU contributed to analyses and interpretation of the data. DJG and JCG contributed to design and interpretation of the data.
Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement No additional data are available.