The Use of Scoring Systems in Selecting Patients for Lung Resection: Work-up Bias Comes Full-Circle

doi:10.1016/j.thorsurg.2007.10.005

Thoracic Surgery Clinics

Volume 18, Issue 1, February 2008, Pages 107-112

https://doi.org/10.1016/j.thorsurg.2007.10.005 Get rights and content

Statistical models of perioperative risk and long-term postoperative survival facilitate the fair assessment of surgical outcomes and provide insight into the association between certain clinical features and outcome. They provide quantitative estimates of risk or long-term survival. These models, however, have a number of limitations in informing decisions concerning the selection of patients for lung resection. This article examines those limitations.

Section snippets

The purpose of selecting patients for surgery

Surgeons judge themselves and are judged by others according to the levels of perioperative mortality and morbidity among their patients and the duration and quality of postoperative survival. During the workings of the multidisciplinary team and in discussions with patients, selection of patients appropriate for surgery, balancing the short-term risks against long-term survival, is a key contribution of the surgeon.

The original use of scoring systems in cardiothoracic surgery was to redress

Risk models: what goes in and what comes out

The first task in constructing a model of perioperative risk or postoperative survival is to identify “candidate prognostic factors” and amass data on these factors and the outcomes of interest. More often than not, it is a case of identifying candidate factors from an existing data set. Ideally, candidate factors should be objective measures that are available for all patients and that bear a putative relationship to the outcome to be predicted (for a broader discussion of these points, see [3]

Interpreting models of risk and survival

If numerous studies identify the same clinical factor as being associated with increased risk or poor survival, this agreement strengthens the notion that this identification is a true finding rather than a statistical fluke. If a factor is identified in only one study, this fact could reflect simply that the item of data was collected in only that study. For instance, when constructing a model for the risk of in-hospital death following lung resection, using data from the European Thoracic

The appropriateness of risk models over time

Models of perioperative risk become out of date. Although the language is of “intrinsic risk faced by the patient” and “patient related risk,” improvements in surgical and anesthetic techniques (including the adoption of new technology) and the standards of intensive care mean that the risks faced by different groups of patients change over time. This development is widely understood; for example, few cardiac surgeons today boast of “outperforming” the Parsonnet risk-scoring system developed in

Summary

The construction of statistical models of perioperative risk and long-term postoperative survival is a useful activity. It facilitates fair assessment of surgical outcomes and provides insight into the association between certain clinical features and outcome. It provides quantitative estimates of risk or long-term survival. There are, however, a number of limitations to the use of such models in informing decisions concerning the selection of patients for lung resection. In essence, the

References (17)

O. Birim et al.
Prognostic factors in nonsmall cell lung cancer
Eur J Surg Oncol
(2006)
E.H. Blackstone et al.
Caveat emptor: the treachery of work-up bias
J Thorac Cardiovasc Surg
(2004)
T. Iizasa et al.
Prediction of prognosis and surgical indications for pulmonary metastasectomy from colorectal cancer
Ann Thorac Surg
(2006)
V. Parsonnet et al.
A method of uniform stratification of risk for evaluating the results of surgery in acquired adult heart disease
Circulation
(1989)
T. Treasure et al.
Assessment of whether in-hospital mortality for lobectomy is a useful standard for the quality of lung cancer surgery: retrospective study
BMJ
(2003)
Treasure T, Utley M, Berrisford R. A risk model for lung resection: data from the European thoracic database project....
D.W. Hosmer et al.
Applied logistic regression
(1989)
S.G. Dolgobrodov et al.
Artificial neural network: predicted vs. observed survival in patients with colonic cancer
Dis Colon Rectum
(2007)

There are more references available in the full text version of this article.

Cited by (9)

An Alternative Method for Predicting the Risk of Postoperative Complications in Lung Resection
2014, Archivos de Bronconeumologia
Citation Excerpt :
Despite the refinement and adaptation of published algorithms and mathematical decision models,1 prediction of surgical risk is difficult due to selection bias inherent to these models.2
The aims of this study were to design a best fit linear regression model to estimate VO_2max (estimated VO₂) and to compare the ability of VO₂ values (measured and estimated) predicting cardiorespiratory complications in a series of patients undergoing lung resection for lung cancer.
This was a prospective, observational study performed in 83 patients. Variables analyzed were: demographic characteristics, comorbidity, body mass index (BMI), FEV1%, FVC%, diffusion capacity (DLCO%), mean daily distance walked in kilometers, VO_2max measured by cardio-pulmonary exercise test (CPET) and postoperative complications. Descriptive and comparative statistical analysis of the variables was performed using the Mann–Whitney test for categorical variables and the Student's t-test for continuous variables. A new linear regression model was designed, where the dependent variable (measured VO_2max) was estimated by the distance, DLCO% and age, resulting in the estimated VO₂. The predictive power of the measured and estimated consumption was analyzed using the Student's t-test, grouping by the occurrence or absence of cardiorespiratory complications.
Both groups were homogeneous for age, sex, BMI, FEV1%, DLCO%, comorbidity, type of resection performed and mean distance walked per day. Estimated VO₂ and measured VO₂ were normally distributed (K–Smirnov test, P>.32). VO₂ means estimated by the model (age, DLCO% and mean distance walked per day) were significantly different between patients with and without complications (Student's t-test, P=.037) compared with measured VO₂ values, which did not differentiate groups (Student's t-test, P=.42).
The VO_2max estimated by the model is more predictive in this case series than the VO_2max measured during a standard exercise test.
Los objetivos de este trabajo son rediseñar un modelo de regresión lineal para estimar el valor de VO_2max (VO₂ estimado) ya publicado y comparar la capacidad predictiva de los valores VO₂ medido y VO₂ estimado en la aparición de complicaciones cardiorrespiratorias en una serie de pacientes sometidos a resección pulmonar por cáncer de pulmón.
Estudio prospectivo y observacional en 83 pacientes. Variables analizadas: demográficas, comorbilidad, IMC, FEV1%, FVC%, DLCO%, distancia recorrida media diaria (km), VO_2max medido en el test de ejercicio cardiopulmonar (CPET) y complicación postoperatoria. Estadística descriptiva y comparativa de variables usando Mann-Whitney test para categóricas y t de Student para continuas normales. Se diseña un modelo de regresión lineal donde VO_2max, la variable dependiente, se estima desde la distancia recorrida, DLCO% y edad del paciente y su resultado es la VO₂ estimada. Se compara la capacidad predictiva de los VO_2max medido y estimado mediante t de Student, agrupando por ocurrencia o no de complicaciones cardiorrespiratorias.
Los 2 grupos son homogéneos en edad, distribución de sexos, IMC, FEV1%, DLCO%, comorbilidad, cirugía realizada y distancia media recorrida/día. VO₂ medida y VO₂ estimada tienen distribución normal (K-Smirnov p > 0,32). En la predicción de complicaciones, las medias del VO₂ estimado a partir del modelo son significativamente diferentes entre los pacientes con/sin complicación (t de Student p = 0,037); frente a los valores de VO₂ medido que no distinguen grupos (t de Student p = 0,42).
El VO_2max estimado por el modelo es más predictivo en esta serie de casos que el VO_2max medido en una CPET.
Carcinoma embryonic antigen: Its place in decision making for pulmonary metastasectomy in colorectal cancer
2010, Journal of Thoracic Oncology
Risk assessment for pulmonary resection
2010, Seminars in Thoracic and Cardiovascular Surgery
Citation Excerpt :
They were not designed for patient selection purposes. The findings of model-building exercises, if based on surgical databases, can only augment, and not replace, clinical judgment13 and moves to deny informed patients lung resection by estimates of risk or “poor” survival should be considered carefully.14 A risk-model may be accurate enough to predict that in a certain group of patients the mortality rate would be 2%, but it fails in identifying which of the 100 patients would actually die.15
Risk assessment for pulmonary resection must include a preliminary cardiac evaluation. Patients deemed at prohibitive cardiac risk should be evaluated and treated as per American Heart Association/American Society of Cardiology guidelines. Those with low cardiac risk or with optimized treatment can proceed with pulmonary assessment. A systematic measurement of lung carbon monoxide diffusing capacity is recommended. In addition, predicted postoperative forced expiratory volume in 1 second should not be used alone for patient selection because it is not an accurate predictor of complications, particularly in patients with chronic obstructive pulmonary disease. The use of exercise testing should be emphasized. Low-technology tests, such as stair climbing, can be used whenever a formal cardiopulmonary exercise test is not readily available. However, in case of suboptimal performance (ie, <22 m in the stair-climbing test) patients should be referred to cardiopulmonary exercise testing with measurement of Vo_2max for a better definition of their aerobic reserve. A Vo_2max less than 10 mL/kg/min (or <35% of predicted) indicates a high risk for major lung resection.
Monitoring Risk-Adjusted Outcomes in Congenital Heart Surgery: Does the Appropriateness of a Risk Model Change With Time?
2009, Annals of Thoracic Surgery
Citation Excerpt :
When predictive risk models are to be used to monitor performance, they need to be regularly reviewed and recalibrated to reflect the true evolution in performance over time. The role of risk models in predicting risk and informing decisions at the level of the individual patient is very limited [15]. Several investigators have made significant contributions to tackling the challenging issue of risk adjustment [8, 13] and the monitoring of performance [12, 14, 16] in pediatric cardiac surgery.
Risk adjustment of outcomes in pediatric congenital heart surgery is challenging due to the great diversity in diagnoses and procedures. We have previously shown that variable life-adjusted display (VLAD) charts provide an effective graphic display of risk-adjusted outcomes in this specialty. A question arises as to whether the risk model used remains appropriate over time.
We used a recently developed graphic technique to evaluate the performance of an existing risk model among those patients at a single center during 2000 to 2003 originally used in model development. We then compared the distribution of predicted risk among these patients with that among patients in 2004 to 2006. Finally, we constructed a VLAD chart of risk-adjusted outcomes for the latter period.
Among 1083 patients between April 2000 and March 2003, the risk model performed well at predicted risks above 3%, underestimated mortality at 2% to 3% predicted risk, and overestimated mortality below 2% predicted risk. There was little difference in the distribution of predicted risk among these patients and among 903 patients between June 2004 and October 2006. Outcomes for the more recent period were appreciably better than those expected according to the risk model. This finding cannot be explained by any apparent bias in the risk model combined with changes in case-mix.
Risk models can, and hopefully do, become out of date. There is scope for complacency in the risk-adjusted audit if the risk model used is not regularly recalibrated to reflect changing standards and expectations.
Value of the average basal daily walked distance measured using a pedometer to predict maximum oxygen consumption per minute in patients undergoing lung resection
2011, European Journal of Cardio-thoracic Surgery
The Pulmonary Metastasectomy in Colorectal Cancer cohort study: Analysis of case selection, risk factors and survival in a prospective observational study of 512 patients
2021, Colorectal Disease

View all citing articles on Scopus

View full text

The Use of Scoring Systems in Selecting Patients for Lung Resection: Work-up Bias Comes Full-Circle

Section snippets

The purpose of selecting patients for surgery

Risk models: what goes in and what comes out

Interpreting models of risk and survival

The appropriateness of risk models over time

Summary

Eur J Surg Oncol

J Thorac Cardiovasc Surg

Ann Thorac Surg

A method of uniform stratification of risk for evaluating the results of surgery in acquired adult heart disease

Circulation

Assessment of whether in-hospital mortality for lobectomy is a useful standard for the quality of lung cancer surgery: retrospective study

BMJ

Applied logistic regression

Artificial neural network: predicted vs. observed survival in patients with colonic cancer

Dis Colon Rectum