Football leagues have been around for more than 100 years and more recently comparative performance is being analysed for everything from restaurants to schools. Last year the Fat Duck in Bray was named as the best restaurant in the UK, and Colchester Royal Grammar School was top of the league tables in England—pupils there did better in their A/AS levels than at any other school in the country.1 This is obviously a significant achievement, but does it mean that it is a “better” school than others? Data comparing schools only present one part of the picture; no weighting is given to sporting or social development, and while attempts have been made to adjust for cultural and socioeconomic influences by looking for “value added”—comparing the difference in results from primary schools with GCSE performance—these again only interpret academic achievement. The data from schools that do well in league tables demonstrate to prospective parents that the academic performance of the school is good, and aim to reassure pupils, parents, teachers and school governors that things are going well. The position in a league table may help parents and children choose a school, but will usually only be one factor among many.

There are obvious parallels between producing league tables in education and health care, with both agendas being driven by politicians and the media, while finding resistance from teachers and doctors. Over the last few years there has been particular scrutiny on mortality results in cardiac surgery, leading to publication of comparative mortality outcomes following coronary artery surgery, aortic valve surgery and all cardiac procedures for individual hospitals and surgeons.2^–7 The issues around “substrate”, type of outcome, analysis and interpretation regarding schools league tables also apply to cardiac surgery mortality outcomes.

WHY MEASURE SURGICAL OUTCOMES?

There are a number of reasons to collect and report hospital and surgeon results:

• Satisfactory results will reassure patients, their carers, hospital managers, commissioners, healthcare regulators and politicians about the quality of surgical care. This needs to be seen against the backdrop of various high profile failures in hospital clinical governance in the UK over recent years.

• Good quality data about patient characteristics and results can be used in a structured way to improve health outcomes. This concept includes not just mortality data, but also morbidity outcomes such as wound infection, re-exploration for bleeding, stroke rates and postoperative length of stay.

• Collection of large scale systematic patient data can facilitate resource planning and allocation, and underpin research.

To date the most significant emphasis of the cardiac surgical data collection project in the UK has been to produce comparative individual unit and surgeon mortality data for isolated coronary artery bypass surgery and all cardiac surgery.2^–7 This will almost certainly be just the initial part of the bigger picture of routine use of structured health care information to assure and improve standards.

HISTORY OF CARDIAC SURGERY DATA COLLECTION

There is a long history of data collection in cardiac surgery.4 The Society of Cardiothoracic Surgeons of Great Britain and Ireland (SCTS) has administered a cardiac surgery register since 1977. All units performing cardiac surgery were encouraged to return activity and mortality data to a central source where it was collated, analysed and disseminated. Contribution to the register was voluntary and individual units were not identified. The initiative gave useful data about activity and trends, and individual units were able to benchmark themselves against pooled national data for crude mortality. Where the register failed was that it did not permit individual units to compare themselves against other units in a risk adjusted manner. This meant that it was theoretically possible for a unit with an elevated mortality that was not significantly higher than the national mean crude mortality rate to assume that this was because they were doing high risk cases. However, the lack of risk adjustment could hide the fact that they were in reality the “worst” unit in the country and possibly that they were “significantly” worse than average. Since 1994 the SCTS has run a database project which allows hospitals to submit more extensive data, including preoperative patient characteristics, operative data and postoperative outcome, which gives the potential to adjust outcomes for case mix. Hospitals were initially asked to submit data on a voluntary basis and now all units in the UK are involved. The database contains information on over a quarter of a million patients and has led to reports on UK cardiac surgical activity, trends, methodology of analysis, outcomes and benchmarking.3

All of British medicine continues to be heavily influenced by the events that happened in cardiac surgery in Bristol and the results of the subsequent public inquiry.8 9 One of the recommendations of the Bristol Inquiry was that patients should have access to “the relative performance of the trust … and the consultant units within the Trust” (recommendations 27 and 155). The SCTS has been working towards this over recent years, and these endeavours have been accelerated by various other initiatives. In 2001 Dr Foster, a media organisation, published comparative mortality data for all hospitals performing coronary artery bypass graft (CABG) surgery in the UK.2 This analysis was based on data from hospital administration systems which are known to be inaccurate,3 and was adjusted for age, sex and urgency of surgery. Neither the data nor the technique of adjustment were independently validated and the analysis met with criticism from doctors.10 11 Dr Foster provided a further analysis on unit mortality for aortic valve replacement (AVR) in 20022 and paediatric cardiac surgery in 2004,12 again accompanied by further cautionary words from the medical profession.13 14

The obvious need for high quality audit data, set against concerns about the use of administrative data for clinical purposes and issues surrounding validation and analysis,15 led the National Health Service Information Authority, through the National Clinical Audit Support Programme, to embark on an ambitious initiative, the Central Cardiac Audit Database (CCAD).16 This project facilitates online clinical data collection, collation and analysis modules for adult and paediatric cardiac surgery along with acute myocardial infarction, interventional cardiology, and pacing/electrophysiology. In cardiac surgery in England all units are connected and are submitting data. The project has overcome concerns about confidentiality and the Data Protection Act by developing techniques of encryption for data transfer and storage. The CCAD structure also has the ability to track patients “longitudinally” through a number of cardiac events and procedures by using the NHS number. A potential stumbling block to complete rollout has been the politics of national devolution; the CCAD project is only funded to cover English units, but the professional societies have a wider geographical remit.

RISK ADJUSTMENT

It is possible to compare outcomes between units or surgeons simply by using “crude” or non-risk adjusted mortality data. This approach has simplicity and transparency on its side but is not embraced with enthusiasm by the majority of clinicians. It is clear that there are quite notable differences in patient characteristics between different units,3 and this variability is probably greater between surgeons who have different subspecialist interests. Many surgeons are concerned that any attempt to produce comparative performance using non-risk adjusted data will stimulate a culture whereby higher risk patients are denied surgery to help maintain good results—so called “risk averse behaviour”.

There have been a number of attempts to adjust for operative risk. The first generally accepted model was the Parsonnet score which was derived in the USA in the 1980s and found widespread use in the UK.17 More recently it has been shown to significantly overpredict observed risk and has been surpassed by other algorithms. The EuroSCORE is now the most common risk tool used. It was developed by a pan-European study in the late 1990s and its most commonly used variant, the additive EuroSCORE, allocates incremental risk points to 17 risk factors to give a score which is supposed to reflect operative mortality.18 It has a reasonable predictive ability for both coronary and valve surgery, but is known to overpredict risk compared to observed mortality in contemporary practice for coronary surgery, and it has poor predictive ability for higher risk cases.5 19 To overcome these shortcomings it has been suggested that the Logistic EuroSCORE is a better model.20 It requires a more complex program to derive the score from the risk factors than the simpler additive model, but is overall a better predictive model; however, it also overpredicts observed mortality for contemporary practice in the UK.21 22

When deciding which tool to use for a particular purpose it is important to understand the relative strengths and weaknesses of the different models. The predictive ability of a risk prediction tool is usually determined in two ways:

• How well it discriminates across the spectrum of risk (that is, does it accurately differentiate between lower and higher risk patients), which is best determined by using the area under a receiver operating characteristic (ROC) curve

• Is it an accurate predictor of observed risk—that is, is the tool “calibrated” correctly?

The Logistic EuroSCORE has a predictive ability for observed mortality for first time coronary surgery of about 0.76 as measured by the area under the ROC curve. An area of 0.5 indicates no better predictive ability than tossing a coin, and 1.0 represents a perfect predictor. A value of above 0.7 is thought to be useful, but it is important to note that 0.76 is not a very high value, which is not surprising given issues about biological variation of patients, and varying physiological reactions to major surgery. It is possible to get slightly higher values by using various mathematical and statistical techniques such as Bayes theorem,3 but the small increments gained in predictive ability are not generally thought to be worth the extra complexities involved.

In addition to advances in risk adjustment, developments have been made in techniques of presenting outcomes data. Various graphical techniques have been used to help analyse results and detect trends or outlying performance at an early stage. The first of these to find common use was the cumulative summation, or CUSUM, curve. This gives a plot of the cumulative number of cases along the x axis and cumulative number of adverse events (typically death) along the y axis (fig 1).23 This type of curve may be adapted to include predicted mortality to enable observed and expected mortality to be compared.24 The CUSUM curve will always display an upward trend, and to make the curves more intuitive the variable life adjusted display (VLAD plot) was developed (fig 2).25 These techniques are well described by Keogh and Kinsman.3 More recently, interest is developing for measuring outcomes using statistical process control charts. These use units of time, typically months when institutions are under scrutiny and the outcome of interest is mortality, and display the actual mortality against expected, using control limits to define acceptable and unacceptable performance.26 Some units find them useful to monitor activity. Funnel plots are becoming popular ways of displaying hospital or individual mortality (fig 3).3 6 27 They are simply a plot of event rates against volume of surgery, and include exact binomial control limits, to allow excessive mortality to be easily detected. They give a “strong visual display of divergent performance”,3 without resorting to the use of league tables.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1 Cumulative summation (CUSUM) curve of one hospital’s mortality data for 1 year. Each patient operated upon is plotted along the x axis, and each death is recorded on the axis. Also given are alert and alarm control limits.3

• Download figure
• Open in new tab
• Download powerpoint

Figure 2 Variable life adjusted (VLAD) plot, for one hospital. Each patient operated upon is plotted along the x axis and on the y axis the predicted risk for the patient is plotted, which will be given as a positive value if that patient survives. If the patient should not survive, the predicted risk is attributed, but −1 is subtracted. If the risk prediction is accurate, the resulting graph should run alongside the value “0” as shown below.3 CABG, coronary artery bypass graft surgery.

• Download figure
• Open in new tab
• Download powerpoint

Figure 3 Funnel plot of all UK cardiac surgeons observed mortality over a 3 year period. The number of cases is given on the x axis, and the percentage mortality on the y axis. Each surgeon is plotted on the graph. Also given are the 99.9% and 99.99% binomial “control limits”. It is felt that these plots give a good visual display of the scatter of outcomes in the context of surgical volume. They allow unacceptable performance to be seen, without having to resort to league tables.3 CABG, coronary artery bypass graft surgery; UKCSR, UK Cardiac Surgical Register.

SUMMARY OF SURGEON SPECIFIC MORTALITY DATA IN THE UK

The SCTS has been working towards the publication of surgeon specific data over a number of years. In 2002, largely in response to the Dr Foster initiative, the SCTS published non-risk adjusted CABG mortality outcomes for every unit in the UK.3 In response to the Bristol public inquiry,9 Alan Milburn, as health secretary, declared in 2001 that individual surgeon data should be placed in the public domain, but the data would first need to be “robust, rigorous and risk-adjusted”. Despite efforts from the SCTS, no complete validated dataset was available for analysis, but to respond to media and political agendas it was thought best to present named surgeon outcomes on crude data in 2004. The format of presentation was within a comprehensive database report.3 For each hospital the individual surgeons were listed along with the name of the Trust’s chief executive, the hospital’s star rating, the overall mortality for isolated CABG and AVR, and comments on the hospital’s data collection infrastructure. Surgeons within each Trust were listed in alphabetical order and the average number of cases per year (over the previous 3 years) were given along with a comment about whether they met SCTS standards. For surgeons who had not been in post for the full 3 years a comment was made to that effect. The standard for satisfactory outcomes was defined as a crude mortality which fell within 99.99% confidence intervals of the mean (1.9%). The justification for these wide confidence intervals was that the data were neither validated15 nor risk adjusted.5 Using these criteria, no surgeon performed worse than the standard. These methods have been subsequently criticised in the surgical literature as leaning too far towards protecting surgical reputations rather than protecting patient’s interests.28 In addition to open publication of the data, the SCTS ran an annual governance screen on all surgeon’s crude mortality rates. Any surgeon who fell outside 95% confidence intervals was notified, along with the clinical director and medical director of the Trust, to allow appropriate local mechanisms of clinical governance to be triggered.

In addition to the SCTS initiatives, several Trusts responded to the changing demands of society by placing their own unit results into the public domain on websites in a variety of different formats. Some went one step further by publicising individual mortality in the form of risk adjusted graphical presentations, rather than specific data.

The move towards publication of individual surgeon mortality outcomes in the UK accelerated following introduction of the Freedom of Information Act in England and Wales in 2005.29 This gives people a right of access to information held by or on behalf of public bodies in England, including NHS Trusts, primary care trusts and health authorities. The information needs to be made available within 20 working days of a request. There are a number of exemptions within the Act which include excessive cost, breach of confidence, revealing trade secrets, prejudicing commercial interests and issues of national security, but it is not yet clear exactly how these exclusions will come to be interpreted. It does, however, seem that requests for individual clinician outcome data, such as named surgeon mortality statistics, are discloseable. Early in 2005 the Guardian newspaper made a request to all Trusts in the UK under the Act. They asked for named surgeon “crude” mortality data for isolated CABG and AVR over the previous 3 years. Because of lack of validation and the inability to perform complete risk adjustment, the SCTS felt it could not endorse this project, but given the inevitability of publication of individuals’ outcomes, members of the SCTS worked with the Guardian on issues related to analysis and interpretation of the data.

In direct response to the Guardian request, a number of units published named surgeon data on websites and through peer review publications.6 Some chose to publish crude mortality, others to use some form of risk adjustment or stratification. A group of hospitals in northwest England felt strongly that publication of non-adjusted data would set a precedent for this type of analysis, both for cardiac surgery and other branches of medicine.5 6 They also believed that available risk adjustment techniques were limited and it was therefore appropriate to present data on all cases, and then to further subdivide outcomes into low and high risk groups, with comments about the major potential benefits from surgery for patients in the high risk group, along with concerns about making direct comparisons between surgeons due to the heterogeneous nature of high risk patients.

The Guardian published their analysis in March 2005,7 and as well as listing all surgeons in the UK with their mortality outcomes, they were critical of some Trusts which did not seem to be able to collect or present outcomes data of satisfactory quality. Subsequent reports in local media and anecdotal reports from patient groups suggested that the analysis was reassuring to the public.

After these developments it became apparent that you could not “put the genie back in the bottle” and the SCTS embarked upon a project with the Healthcare Commission to publish surgical results through a public portal on their website. There was widespread public consultation on this development and the survival statistics of all units in England and a majority of surgeons are now available to the public.30

UK INITIATIVE IN AN INTERNATIONAL CONTEXT

There have been no similar national initiatives to collect and publish individual surgeon outcomes elsewhere in Europe, but the public disclosure agenda is ahead of the UK in a number of states in the USA, particularly in New York,31 where named surgeon data were released in 1991, again following a media request under a freedom of information Act. Some argue that this has led to an overall improvement in quality as indicated by a reduction in mortality; indeed a 40% reduction in risk adjusted mortality has been reported in both the USA and the UK following the introduction of public reporting of results.31 32 Others claim that the initiative led to surgeons turning away high risk cases, and that the improvements seen were no greater than those seen elsewhere where there was no policy of public disclosure,33^–37 but this has been disputed.32 38 Data from a survey of UK cardiac surgeons conducted independently by the BBC programme Newsnight in 2000 suggested that 80% of surgeons were in favour of public disclosure of data, but 90% thought that high risk cases were already being avoided in anticipation of the publication of results.3 When asked how well current algorithms adjusted for surgical risk, only 6% of surgeons felt that the available algorithms adjusted accurately for operative risk and 37% felt that the methodologies were woefully inadequate. It would prove interesting to repeat that analysis following recent events. Anecdotal reports in the UK from cardiologists again suggest that it has been more difficult to get high risk cases accepted for surgery in recent years, but it has not been possible to show this effect on examination of a large regional database.32

An authoritative report by the Nuffield Trust suggests three differing models for release of outcome data39:

• A professionally orientated model that assumes healthcare workers and professional organisations have a desire to monitor and improve standards—this model applies to the work of the SCTS and other clinicians listed above

• A public accountability model whereby openness of data with associated education and subsequent debate will help to improve quality—this applies to the Guardian and Dr Foster initiatives

• A market orientated model which assumes that informing and educating patients and their carers will allow consumers to exercise choice which will itself drive improvement—this is the principle of the NHS choice agenda, and the relentless political agenda of “choice” will inevitably drive the dissemination of institutional and individual clinician outcome data.

There is still much debate within the profession about public disclosure of outcomes. In schools it is thought that a good head teacher can lead to significant improvements in the processes within the school, which can improve results. It is the head teacher whose name is used in discussions, rather than listing a detailed analysis of outcomes under individual teachers. After cardiac surgery the outcome depends on many factors including: “the socioeconomic status of the local population; severity of cardiac illness; prevalence of co-morbidities; threshold of referral from both the general practitioner and the cardiologist; threshold of acceptance by the surgeon; standards of anaesthesia, surgery and intensive care; adequacy of the facilities and staffing levels; attitude to training; interpersonal relationships between staff; and the geographical layout of the unit”.4 However, it is also accepted that individual surgeons are important drivers to institutional change and that publishing named surgeon outcomes within units, with a knowledge that multidisciplinary clinical and managerial structures will influence outcome, may contribute to improving standards.37 Most UK cardiac surgeons have now accepted that outcomes of patients undergoing surgery under their care will continue to be published against their name.

WHAT NEXT FOR CARDIAC SURGERY?

The publication of named surgeon mortality data in the UK has come as a relief to many. The work associated with developing data collection infrastructure and techniques of analysis has been significant and many individuals have had concerns over how they would fare in any “league table”, yet no surgeon has been individually named in recent analyses as having excessive mortality. There are some questions about the quality of current surgical data3 15 and a programme of data validation visits has been commenced.3 It is anticipated that there will now be ongoing annual presentations of individual surgeon’s mortality data, probably for the previous 3 years’ activity.

Over recent years the roles of differing organisations in clinical governance is becoming clearer. Most Trusts now have increasingly effective management structures for promoting quality improvement and detecting suboptimal performance. The SCTS is looking to support these processes by providing online contemporary benchmarking tools through CCAD and undertaking annual analysis of its members’ mortality data. The overall direction of national cardiac surgery audit is overseen in England by a committee made up of membership from the SCTS, the Department of Health and the Healthcare Commission. Mortality within 30 days of a bypass operation has been one of the NHS “high level performance indicators” used for several years and the Healthcare Commission has included data on mortality following isolated CABG in its Trust star rating system.40 It is accepted that mortality outcomes are only one aspect of clinical quality and the CCAD software has been developed to benchmark other outcomes including re-exploration for bleeding, stroke rates and other complications. This is in line with quality developments on outcomes and process measures from the Society of Thoracic Surgeons in the USA.41 There is also a pilot project underway in northwest England looking at the concept of “payment for quality” whereby units will be rewarded for achieving satisfactory results, and providing care according to the various processes which are thought to be associated with the best quality outcomes.42 The National Confidential Enquiry into Patient Outcome and Death (NCEPOD) has also embarked upon a project to look in detail at the causes of death after CABG, including both procedural and institutional issues, which will be published in mid 2008.43

PROFESSIONAL RECERTIFICATION

The Chief Medical Officer’s publication Good doctors, safer patients, and the subsequent white paper “Trust, assurance and safety”, have changed the way the medical profession will be regulated.44 45 It is no longer accepted that specialists have a right to remain on specialist registers, and a necessary part of the recertification process will be to demonstrate that outcomes following procedures performed by any particular doctor are satisfactory. It is planned that the General Medical Council will be the regulator, the Royal Colleges will administer the recertification process, and the specialist societies will be responsible for setting the standards which specialists will have to achieve. Data about cardiac surgical mortality will obviously play into this process, but there is further work which needs to be done on the systems and methods.

WHERE DOES THE REST OF CARDIAC CARE FIT IN?

Cardiac surgery is only one small part of cardiac care. Because of the fairly homogeneous nature of cardiac surgery, the relatively small number of cases performed each year, the sophistication of cardiac surgery data collection, and the political and media emphasis on surgical outcomes, both unit and surgeon specific mortality data are now in the public domain. This focus on collecting surgical data means that extensive and often high quality information is being given to patients, particularly through hospital websites, and this seems to be a useful byproduct of an agenda which has been heavily driven by politicians and the media.46

Data are not yet widely available for institution mortality after percutaneous coronary intervention (PCI). There are issues related to case mix which are important when comparing angioplasty results, particularly with the advent of primary angioplasty. The outcomes following elective PCI are now so low that mortality is probably not a useful indicator. Progress is, however, being made with risk adjustment techniques using combined end points47; it is likely that named practitioner angioplasty results will be published in the UK at some stage, and it will probably be necessary to generate individual outcomes results of some description to feed the recertification process. The MINAP project is progressing well and has led to data on various process measures, such as door to needle time for treating myocardial infarction48; like deaths after CABG, these have been used in the Healthcare Commission’s star rating.40 It is planned that the CCAD project will give useful information about overall disease management, by allowing linkage between the different modules of cardiac care. Putting together these various facets of cardiac audit and quality assurance should give exciting data over the coming years.