Prospective External Validation of a Predictive Score for Postoperative Pulmonary Complications

POSTOPERATIVE pulmonary complications (PPCs) are a major contributor to the overall risk of surgery.^1–4  They are associated with substantially longer time spent in hospital⁵  and higher in-hospital postoperative mortality.⁶  In the United States, the reported annual economic burden of PPCs is approximately $3.42 billion (USD).⁷ 

A wide range of patient, anesthetic, and surgical factors are associated with PPCs.^2,8,9  To date, only a few studies have developed predictive models for PPCs in settings that reached beyond very specific disease or surgical contexts. Two of these studies, with pneumonia¹⁰  and respiratory failure¹¹  as outcomes, were performed in a population of American veterans; as 90% of the patients were men, the generalizability of the findings may be limited. Six others used retrospective data sets to develop a score to predict single PPC outcomes: unplanned reintubation,^12–14  postoperative pulmonary failure,¹⁵  and adult respiratory distress syndrome.^16,17  Finally, two^1,3  were prospective studies in patients undergoing a wide range of surgeries and only one was internally validated.¹  To our knowledge, none of these studies have been replicated in other settings to externally validate the scores in new prospectively collected samples of patients, and for this reason, none can be confidently generalized.^18,19  The lack of validated models affects the clinician’s ability to predict and plan strategies to prevent PPCs in high-risk groups.

The recent Assess Respiratory Risk in Surgical Patients in Catalonia (ARISCAT) study¹  addressed the problem of differences across surgical contexts by using a population-based approach that was representative of a wide range of procedures and patients in a geographically defined, mixed urban-rural practice setting. A clinically practical seven-factor scoring system to assess the risk of a composite PPC—the likelihood of developing any complication in a list of well-defined events—was internally validated.

To test the hypothesis of the geographic transportability of the ARISCAT score to different but plausibly related^19,20  surgical populations, 63 European centers in 21 countries prospectively recruited a new patient cohort for the Prospective Evaluation of a RIsk Score for postoperative pulmonary COmPlications in Europe (PERISCOPE) study. The aim of this study was to measure the accuracy of ARISCAT score predictions of PPCs in the PERISCOPE cohort overall and in three subsamples of that cohort. Cohort splitting was intended to reflect possible case-mix differences that might appear with increased geographic distance from the setting where the ARISCAT model was developed.

The PERISCOPE cohort was established following a prospective, observational multicenter design in which 63 European hospitals (appendix) volunteered to recruit surgical patients during continuous 7-day periods. Recruitment within a center started on a date between May 2, 2011, and August 15, 2011, chosen on the basis of the local researchers’ convenience, and follow-up ended in November 2011. The hospitals were identified through membership in the European Society of Anaesthesiology and approached directly by national study coordinators. The study was registered at www.clinicalTrials.gov (identifier, NCT01346709).

Consecutive patients undergoing a nonobstetric in-hospital elective or emergency surgical procedure under general, neuraxial, or plexus block anesthesia were recruited. Exclusion criteria were age under 18 yr, obstetric procedures or any intervention during pregnancy, procedures in which only local or peripheral nerve anesthesia would be used, procedures outside an operating theater, procedures related to a previous postoperative complication, transplantation, patients with preoperatively intubated trachea, and outpatient procedures (hospital stay of <24 h).

Three numerically comparable subsamples were defined, based on their geographic distance from the development population as follows: Spain, Western Europe (WE), and Eastern Europe (EE).

Ethics requirements differed in the 21 countries, but formal approval from a research ethics review board was applied for and given in each. The locally responsible investigator also applied for and obtained approval from the ethics committee of each participating hospital. Each center investigator sent a scanned copy of the ethics committee approval to the European Society of Anaesthesiology secretariat, where files were centralized. Written informed consent to use the data was obtained from each enrolled patient in all centers.

The international research team consisted of a steering committee and nationally and locally responsible investigators on behalf of the European Society of Anaesthesiology.

Postoperative pulmonary complications were recorded by the investigators throughout the postoperative hospital stay up to a maximum of 5 weeks. Patients with PPCs were identified by data collectors who consulted medical records in real time, daily while they were being created, to find events that fulfilled any PPC definition; they did not modify a center’s customary management of patients. A structured paper questionnaire was filled in for each patient; later, information that could identify the patients was removed before transfer to secure online record forms (OpenClinic Optimized Cloud Hosting, Boston, MA), an electronic system with quality control algorithms to validate data entry and identify missing data. A central data manager checked entries to confirm completeness of records and asked the designated local contact person to provide additional information if necessary.

Data for the seven risk factors described in the ARISCAT model (table 1) were collected preoperatively by the anesthesiologist in charge of the patient after signed informed consent had been given, as follows: age in years; peripheral oxyhemoglobin saturation measured by pulse oximetry (Spo₂) breathing air in supine position after resting 1 min or, in patients on oxygen, Spo₂ after 10 min without oxygen; respiratory infection in the last month; hemoglobin concentration; surgical incision site; surgical duration in hours; type of surgery (scheduled or emergency).

Postoperative pulmonary complication was defined as the occurrence of at least one event on a list of in-hospital fatal or nonfatal PPCs (table 2). Thus, a patient was considered to have had a PPC when at least one of these events was recorded. This outcome was therefore considered as a binary categorical variable (yes/no) for the purposes of statistical analysis.

To compare subsamples, we also recorded administrative data (dates of surgery and discharge and vital status at discharge), general information (sex, height, and weight), preoperative variables (chronic pulmonary disease, smoking status, hypertension, cerebrovascular disease, coronary artery disease, chronic heart failure, liver disease, chronic kidney disease, and physical status using the American Society of Anesthesiologists’ classification), and intraoperative variables (anesthetic technique and surgical specialty). Postoperative hospital length of stay and in-hospital postoperative mortality were followed up to a maximum of 90 days.

The sample size was calculated considering that at least 100 PPCs were needed in each of the three external validation subsamples.²¹  The incidence of each PPC was calculated in the PERISCOPE cohort (and its three subsamples: Spain, WE, and EE), and the associations between number of PPCs per patient and both length of hospital stay and in-hospital mortality were assessed. Comparative analysis of demographic and clinical characteristics between the ARISCAT development sample versus the whole PERISCOPE cohort and each subsample was performed.

In the ARISCAT regression model (risk of PPC = 1/(1 + e^{−linear predictor}), the linear predictor (lp_ariscat) was built using β coefficients derived from the original regression model. A risk score was also calculated for each patient by assigning points derived by multiplying the regression coefficients by 10, rounding to the nearest integer and adding the integers (table 1). Three predicted risk groups were then defined according to the cutoffs identified in the ARISCAT study¹  by means of the minimum description length principle: <26 (low), ≥26 and <45 (intermediate), and ≥45 (high risk).

The model’s performance was then assessed by studying discrimination, calibration, and clinical usefulness in the PERISCOPE data set overall and the three subsamples (Spain, WE, and EE). For measures of discrimination and diagnostic accuracy, the ARISCAT clinical score was used, whereas for calibration the linear predictor equation was used because it gives a more accurate mathematical assessment of each patient’s outcome risk.

Accurate predictions discriminate between patients with the outcome and those without. The ability of the ARISCAT score to rank patients with and without at least one PPC was quantified with the c-statistic, which is the equivalent of the area under receiver operating characteristic curve for a dichotomous outcome variable.

Calibration refers to the agreement between observed outcomes and predictions. It can be broken down into two components: a constant (a) and a coefficient (b), which represent, respectively, the intercept and the calibration slope of a line plotting observed frequencies against predictions.²²  These two components can be calculated by logistic regression with the linear predictor as the only risk factor for the outcome; this regression defines a new linear predictor: lp_calibrated = a + b × lp_original.^23,24  The calibration slope (b) also reflects the average effect of predictors in the outcome; the adjusted value of the intercept (a) after setting the value of b to 1 (assuming hypothetical optimal performance of predictors) reveals a systematic deviation (bias) of predictions. Following these statistical premises, under the hypothesis that the ARISCAT linear predictor (lp_ariscat) was the only valid PPC predictor, calibration was verified by performing logistic regression in the complete PERISCOPE cohort and in each geographic subsample, with lp_ariscat as the only independent variable and the observed PPC composite outcome as the dependent variable. From each of these regressions, we obtained a new linear predictor (lp_calibrated) as lp_calibrated = a + b × lp_ariscat. The PPC risk in each PERISCOPE subsample was then expressed as follows (using a different lp_calibrated for each): Risk of PPC = 1/(1 + e^−lp_calibrated). Finally, to reflect the clinical implications of calibration, we also calculated the predicted and observed PPC frequencies in each of the PERISCOPE data sets according to the cutoffs for three levels of risk.

The utility of a predictive model can be assessed by means of measures of accuracy of outcome diagnosis (sensitivity, specificity, positive and negative likelihood ratios, and positive and negative predictive values). These measures of PPC diagnostic accuracy were analyzed for intermediate and high-risk scores compared to lower risk scores, respectively.

To adjust the ARISCAT score for the influence of European regional influence, we performed a logistic regression with PPC occurrence as the dependent variable and the ARISCAT score (three levels of risk) and geographic area (Spain, WE, and EE) as independent variables. We also carried out a supplementary exploration of the performance of the ARISCAT model’s ability to predict single components of the composite and alternative composite outcomes in the PERISCOPE sample by calculating adjusted odds ratios for each predictor and c-statistics for the each individual PPC outcome and alternative composites.

The Mann–Whitney U test was used to compare means and the chi-square test or Fisher exact test to compare percentages. The Kruskal–Wallis test was used to compare postoperative length of stay between subgroups formed according to the number of PPCs found. The Mantel–Haenszel test was used to analyze trends in mortality rates between those subgroups. Statistical analyses were performed using the SPSS Software package (IBM SPSS Statistics 19.0, Armonk, NY). Categorical variables were expressed as number of cases, and percentage and continuous variables were expressed as the median and interquartile range. All performance measures were expressed with 95% CIs.

A total of 5,859 surgical patients were recruited by the participating hospitals (fig. 1); 475 (8.1%) were lost because of recruitment or protocol violations or missing follow-up data, and 285 (4.9%) were lost due to missing data in candidate risk factors (see table 1, Supplemental Digital Content 1, https://links.lww.com/ALN/B55). The most important missing variables lost were Spo₂, preoperative anemia (<10 g/dl), and respiratory infection in the last month (see table 2, Supplemental Digital Content 1, https://links.lww.com/ALN/B55). In the PERISCOPE cohort overall, 725 PPCs were recorded in 404 patients (7.9% of the 5,099 patients studied). Respiratory failure was the most frequent complication (241 patients, 4.7%), followed by pleural effusion (159, 3.1%), atelectasis (122, 2.4%), pulmonary infection (120, 2.4%), bronchospasm (42, 0.8%), pneumothorax (29, 0.6%), and aspiration pneumonitis (12, 0.2%). Among patients with PPCs, 263 (65%) had more than one complication and 141 (35%) had three or more. The time between surgery and the first PPC recorded was 3 (2 to 6) days. In-hospital mortality in the group of patients with at least one PPC (8.3%) was significantly higher than in patients with no PPC (0.2%; P < 0.0001).

Comparison of demographic and clinical characteristics, PPC incidence, length of hospital stay, and mortality between the ARISCAT development cohort (1,627 patients) and the overall PERISCOPE cohort and subsamples are shown in table 3. The PPC incidence was higher in the overall PERISCOPE cohort and subsamples than in the ARISCAT sample, but the in-hospital PPC-associated mortality rate and postoperative length of stay were similar. Cardiac and cerebrovascular comorbidities were higher in the PERISCOPE cohort than in the ARISCAT development sample. The relationships between length of stay and in-hospital mortality and number of PPCs are shown in table 4.

The performance measures describing discrimination, calibration, and clinical usefulness in each of the PERISCOPE samples in which the ARISCAT model was tested are shown in table 5. Predicted probabilities and observed PPC frequencies in each of the PERISCOPE data sets are shown in table 6, according to the ARISCAT score cutoffs for three levels of risk.

The adjusted odds ratios for predictors in the ARISCAT score and the c-statistics of the ARISCAT model for each component of the composite outcome with over 100 events in the overall PERISCOPE cohort (respiratory failure, suspected pulmonary infection, pleural effusion, and atelectasis) and for possible combinations of two, three, or four PPC outcomes are shown in supplemental tables (see Supplemental Digital Content 2, https://links.lww.com/ALN/B56). This supplemental analysis suggested that the variables in the model with the highest odds ratios might also be good predictors in refitted predictive models for any component of the PPC composite outcome. The adjustment of the ARISCAT score for interaction with geographic area is also shown (see table 1, Supplemental Digital Content 3, https://links.lww.com/ALN/B57). This analysis confirmed first, that EE region was an independent risk factor for the outcome and second, that there was an interactive positive effect between WE region and the ARISCAT score’s prediction of risk for the outcome.

Before prognostic scores are adopted for clinical use outside the development setting, they should be studied in new populations^19,25,26 ; yet to our knowledge, this is the first study in which a PPC risk score has been validated externally.

Our study of the performance of the seven-factor ARISCAT score (table 1) in patient samples that were progressively distant from the development setting provides an intermediate level of evidence according to the definition of Justice et al.,¹⁹  supporting the use of this model for PPC risk prediction in a broad European surgical population in which the observed incidence of PPCs fell within the ranges reported for similar settings.^4,9  The results in each external sample additionally illustrate the degree to which external validation has potential clinical significance, as it suggests that even a validated score may need further adjustments for populations with characteristics that diverge from those previously studied.^23,27,28 

The ARISCAT score, which had shown very good ability to discriminate PPC risk in the development sample (c-statistic, 0.89),¹  also showed good discrimination in the overall PERISCOPE sample (c-statistic, 0.80). Discrimination was even better or equally good in the WE and Spain samples. The c-statistic in the EE subsample (0.76) was only moderately good.

While discrimination values check that the model has good ability to stratify patients according to risk, calibration provides information on the degree to which observed frequencies of outcomes deviate from predicted in a particular population. The calibration slope b was significantly lower than the ideal (b = 1) in all PERISCOPE subsamples studied (table 5). Although this finding is potentially attributable to the optimism inherent in nearly every model,²⁹  it may also be a consequence of a real difference in the effects of predictors in the new populations. In the WE subsample, where the score performed best, the calibration slope was over 0.8, while in the EE subsample, the slope was under 0.6, suggesting that the coefficients of the ARISCAT predictors might require recalibration in a population represented by this subsample. Intercept values significantly different from 0 in the EE and the WE subsamples in this analysis suggest that factors other than the ARISCAT score’s predictors probably play a role. Specifically, differences in case mix between the PERISCOPE and development samples (table 3)—such as different rates of underlying cardiovascular disease or different distributions of surgical procedures and anesthesia techniques—could explain the significant differences in the PPC incidences, as well as differences in the intercepts between PERISCOPE subsamples. This hypothesis seems strong based on our supplemental analysis (table 1, Supplemental Digital Content 3, https://links.lww.com/ALN/B57) to explore interaction between the ARISCAT score’s prediction of PPC risk and geographic area. Significant risk in the EE region is evident even after adjustment, but the odds ratio implies that there are additional unknown risk factors in this area. The analysis also revealed a positive interaction between the score’s prediction of risk and WE geographic zone; put another way, the ARISCAT score performed best in this region, even better than in the Spanish cohort. It has been noted that external validation studies should be undertaken in patient samples that are different but plausibly related to the development sample,^17,19  but the scientific community has unfortunately not come to agreement on a definition for this combination of difference and relation in case mix. In other words, the number, or proportion, of candidate predictors not finally included in the model that should be similar in an external sample for it to serve for validation purposes has not been made explicit.

Likelihood ratios (table 5) are the measures that best summarize the usefulness of a prognostic test. Thus, the risk for PPC of a patient with a score of 45 or more (high) is 5 to 11 times higher than the risk of a patient with a lower score; a patient with a score below 26 has a level of risk from two to four times lower than others with higher scores. These results allow us to define the score as a tool with moderate to good clinical utility to estimate the risk of complications. It should be noted that apparently modest predictive values that would not be acceptable in diagnostic tests, where accuracy is essential, may still be very helpful in prognostic models, which are used in preoperative visits to predict a complication risk higher than average.

This first study to externally validate a PPC risk score has several strengths. First, we followed a prospective design calling for careful data collection in an appropriately large sample of patients from a wide spectrum of European countries and surgical settings. Second, by dividing the PERISCOPE cohort into three subsamples that were progressively distant from the development setting, we were able to illustrate the degree to which a model might behave somewhat differently in each one, a reminder of the importance of external validation generally speaking and of the demanding calibration step in particular. This calibration differentiates between miscalibration attributable to potential predictors not included in the model (when the intercept a differs significantly from 0) and miscalibration attributable to different weights of the predictors (when the slope b differs significantly from 1).

Concerning limitations, we are aware of the low representativeness of the samples with respect to the geographic areas in which they were obtained. This limitation prevents us from extrapolating definite conclusions as to the extent to which recalibration might be required in each of the areas we studied. Such recalibration might be useful, with a view to optimizing the usefulness of the model in particular settings. A second possible limitation is that some PPCs might not have been detected in patients discharged early, but this hypothetical bias is inherent to every study with in-hospital follow-up. We think it would have had only a minor effect in this study, given that the candidates for early discharge would have been patients undergoing less invasive procedures and those showing a more favorable course. Finally, a composite outcome, of the type the ARISCAT score predicts, might be considered a controversial choice. However, such composites mimic clinicians’ weighting of several risk factors at once or in series; thus, we think this approach reflects the real conditions of clinical practice during the period when decisions affecting perioperative management are taken. Our confirmation that the length of hospital stay and in-hospital mortality increase as PPCs rise in number in the external samples (table 4) underlines the importance of a patient’s development of any single respiratory event included in the composite. Moreover, the results of the exploratory analysis of the ARISCAT score’s prediction of single components of the composite outcome (Supplemental Digital Content 3, https://links.lww.com/ALN/B57) suggested that the most powerful predictors in the ARISCAT model would also be good predictors in a refitted predictive model for any component of the composite.

We conclude that assessing the seven easily recordable and clinically accessible factors identified by the ARISCAT score (age, preoperative Spo₂ in air, respiratory infection in the last month, preoperative anemia, upper abdominal or intrathoracic surgical incision, duration of surgery, and emergency procedure) is useful for differentiating three levels of PPC risk in hospitals outside the development setting, although performance differs significantly between geographic areas. That ability to distinguish risk makes the score a validated starting point for controlled trials and audits of risk-reduction strategies. Even so, we advise clinicians to use this scale cautiously when predicting risk for an individual patient, given that the calibration of the model is suboptimal in some geographic contexts. Recalibration can optimize performance of the score for use in homogeneous populations with well-defined characteristics.

The authors thank all collaborators for their recruitment of patients (listed in appendix). The authors thank Brigitte Leva (European Society of Anaesthesiology, Brussels, Belgium) for her assistance as data manager, and Mary Ellen Kerans, M.A., for article editing and advice on the English usage in some versions of the article (European Society of Anaesthesiology funding).

The Prospective Evaluation of a RIsk Score for postoperative pulmonary COmPlications in Europe (PERISCOPE) study was funded and supported by the European Society of Anaesthesiology (Brussels, Belgium) through the Clinical Trials Network. This study also received support provided by Grant 041610-2003 from Fundació La Marató de Televisió de Catalunya, Barcelona, Spain.

The authors declare no competing interests.

Chief investigator

• Jaume Canet, Hospital Universitari Germans Trias i Pujol, Barcelona, Spain

Steering Committee Members

• Jaume Canet, Spain

• Sergi Sabaté, Spain

• Olivier Langeron, France

• Marcelo Gama de Abreu, Germany

• Lluís Gallart, Spain

• F. Javier Belda, Spain

• Paolo Pelosi, Italy

• Andreas Hoeft, Germany

• Valentin Mazo, Spain

Management Team

• Brigitte Leva, European Society of Anaesthesiology, Brussels, Belgium

Albania

1. University Hospital centre "Mother Theresa" (Tirana), Jonela Burimi, M.D., Toma Halefi, M.D., Aleksander Hoxha, M.D.,* Kliti Pilika, M.D., Imelda Selmani, M.D.

Belgium

1. Cliniques Universitaires Saint Luc A.S.B.L. Université Catholique de Louvain (Brussels), Véronique Daout, M.D., Caroline Gauthier, M.D., David Kahn, M.D., Mona Momeni, M.D.,* Christine Watremez, M.D.

Bosnia and Herzegovina

1. Clinical Centre University Sarajevo "Heart Center" (Sarajevo), Slavenka Straus, M.D.*

2. General Hospital "Prim.dr Abdulah Nakas"(Sarajevo), Dejana Djonovic-Manovic, M.D., Marina Juros-Zovko, M.D.*

Croatia

1. University Hospital Rijeka (Rijka), Helga Komen-Ušljebrka, M.D.,* Vlasta Orlić, M.D., Ivana Stuck, M.D.

Czech Republic

1. Faculty Hospital Brno (Brno), Lenka Baláková, M.D., Martina Kosinová, M.D., Ivo Křikava, M.D., Roman Štoudek, M.D., Petr Štourač, M.D.,* Katarina Zadražilová, M.D.

2. Masaryks Hospital Usti nad labem (Usti Nad Labem), Sanober Janvekar, M.D.*

Estonia

1. Tartu University Hospital (Tartu), Juri Karjagin, M.D., Kadri Rõivassepp, M.D., Alar Sõrmus, M.D.*

France

1. Hôpital Pitié-Salpêtrière (Paris), Philippe Cuvillon, M.D., Cristina Ibáñez-Esteve, M.D., Olivier Langeron, M.D.,* Mathieu Raux, M.D., Armelle Nicolas-Robin, M.D.

Germany

1. Klinikum Darmstadt GmbH (Darmstadt), André Winter, M.D.*

2. Medical Center of the Johannes Gutenberg University Mainz (Mainz), Malte Brunier, M.D., Kristin Engelhard, M.D., Rita Laufenberg Feldmann, M.D.,* Raphaele Lindemann, M.D., Susanne Mauff, M.D., Anne Sebastiani, M.D., Camila Zamperoni, M.D.

3. University Hospital Bonn (Bonn), Andreas Hoeft, M.D.,* Florian Kessler, M.D., Maria Wittmann, M.D.

4. University Hospital Carl Gustav Carus, Dresden University of Technology (Dresden), Thomas Bluth, M.D., Marcelo Gama de Abreu, M.D.,* Andreas Güldner, M.D., Thomas Kiss, M.D.

Hungary

1. MISEK Kft. (Miskolc), Kristina Bráz, M.D., Csilla Ruszkai, M.D.*

Italy

1. Azienda Ospedaliera (Padova), Massimo Micaglio, M.D., Carlo Ori, M.D., Matteo Parotto, M.D.,* Paolo Persona, M.D.

2. Azienda Ospedaliera S. Croce e Carle (Cuneo), Coletta Giuseppe, M.D.*

3. Azienda USL n. 5 di Pisa Ospedale F. Lotti (Pontedera), Paolo Carnesecchi, M.D., Denise Lazzeroni, M.D., Irene Lorenzi, M.D.*

4. European Institute of Oncology (Milano), Gianluca Castellani, M.D., Daniele Sances, M.D.,* Gianluca Spano, M.D., Stefano Tredici, M.D., Dario Vezzoli, M.D.

5. Ospedale San Martino (Genova), Iole Brunetti, M.D., Anna Di Noto, M.D., Angelo Gratarola, M.D., Alexandre Molin, M.D.,* Luca Montagnani, M.D., Giulia Pellerano, M.D., Paolo Pelosi, M.D.

6. Ospedale Sant'Orsola, Malpighi (Bologna), Maurizio Fusari, M.D.*

7. University of Insubria (Varese), Laura Camici, M.D., Luca Guzzetti, M.D., Fabio Marangoni, M.D., Paolo Severgnini, M.D.*

8. University of Milano, Ospedale San Paolo (Milano), Piero Di Mauro, M.D., Francesca Rapido, M.D., Concezione Tommasino, M.D.*

Latvia

1. Pauls Stradins Clinical University Hospital (Riga), Ieva Nemme, M.D., Janis Nemme, M.D.*

Lithuania

1. Kaunas Medical University Hospital (Kaunas), Justinas Blieka, M.D., Jurgita Borodičienė, M.D., Brigita Budrytė, M.D., Aurika Karbonskiene, M.D.,* Inga Kiudulaitė, M.D., Eglė Milieškaitė, M.D., Renata Rasimavičiūtė, M.D., Ugnė Sirevičienė, M.D., Ramunė Stašaitytė, M.D., Edgaras Ūsas, M.D., Giedrė Žarskienė, M.D.

2. Vilnius University Hospital Santariskiu Clinics (Vilnius), Egle Kontrimaviciute, M.D., Jurate Sipylaite, M.D.,* Gabija Tomkutė, M.D.

Luxembourg

1. ZithaKlinik (Luxembourg), Petra Bardea, M.D., Marco Klop, M.D., Marc Koch, M.D.*

Poland

1. 10 Wojskowy Szpital Kliniczny z Polikliniką w Bydgoszczy (Bydgoszcz) Dominika Bożiłow, M.D., Robert Goch, M.D.*

Portugal

1. Hospitais da Universidade de Coimbra, EPE (Coimbra), João Bonifácio, M.D., Sofia Marques, M.D., Tânia Teresa dos Santos Ralha, M.D.*

2. Centro Hospitalar de Lisboa Ocidental (Lisbon), Daniel Alves, M.D., Inês Carvalho, M.D., Josefina Suzana Da Cruz Parente, M.D.,* Sara Tomé, M.D.

3. Hospital Fernando Fonseca (Lisbon), Cristina Carmona, M.D.*

4. Instituto Português de Oncologia Do Porto (Porto), Miranda Costa, M.D.,* Maria Lina, M.D., Sofia Sierra, M.D.

Romania

1. Emergency Clinical Hospital of Constanta (Constanta), Alina Balcan, M.D., Iulia Cindea, M.D., Viorel Ionel Gherghina, M.D.,* Catalin Grasa, M.D.

2. Emergency County Hospital Clinic of Anesthesia and Intensive Care (Târgu Mures), Ruxandra Copotoiu, M.D., Sanda-Maria Copotoiu, M.D.,* Judit Kovacs, M.D.,* Janos Szederjesi, M.D., Arthur Theil, M.D.

3. Emergency Institute of Cardiovascular Diseases Prof. Dr. C. C. Iliescu (Bucharest), Daniela Filipescu, M.D.*

Russia

1. Krasnoyarsk State Medical University (Krasnoyarsk), Alexey Grytsan, M.D.,* Tatiana Kapkan, M.D., Sergey Rostovtsev, M.D., Anastasia Yushkova, M.D.

Spain

1. Clinica Universidad de Navarra (Pamplona), Ricardo Calderón, M.D., Elena Cacho, M.D., Carolina Marginet, M.D., Pablo Monedero, M.D.,* Maria José Yepes, M.D.

2. Consorcio Hospital General Universitario de Valencia (Valencia), Jose Miguel Esparza Miñana, M.D., Manuel Granell Gil, M.D.,* Gabriel Rico Portolés, M.D.

3. Corporació Sanitària Parc Taulí (Barcelona Sabadell), Alberto Lisi, M.D.,* Gisela Perez, M.D., Nuria Poch, M.D.

4. Fundacio Althaia (Manresa); Mauricio Roberto Argañaraz Quinteros, M.D., Carme Font Bosch, M.D., Jordi Torrellardona Llobera, M.D.*

5. Fundació Puigvert (Barcelona) Sergi Sabaté, M.D.,* Pilar Sierra, M.D.

6. Hospital Arnau de Vilanova (Lleida), Mercedes Matute, M.D.*

7. Hospital Clinic de Barcelona (Barcelona) Amalia Alcon Dominguez, M.D.,* María José Arguis, M.D., Isabel Belda, M.D., Enrique Carrero, M.D., Jacobo Moreno, M.D., Irene Rovira, M.D., Marta Ubre, M.D., Roberto Castillo, M.D., Sílvia Herrero, M.D.

8. Hospital Clínic Universitari de Valéncia (Valencia) Maria Teresa Ballester Luján, M.D.,* F. Javier Belda, M.D., Jose Carbonell, M.D., Geri Gencheva, M.D., Andrea Gutierrez, M.D., Julio Llorens, M.D., Sofia Machado, M.D.

9. Hospital de Denia (Denia), Francisca Llobell, M.D.,* Daniel Paz Martin, M.D.

10. Hospital del Tajo Aranjuez (Madrid), Francisco Javier García-Miguel, M.D.*

11. Hospital General de La Palma Breña Alta (La Palma, Canarias), Aníbal Pérez García, M.D.*

12. Hospital General Universitario Alicante (Alicante), Roque Company, M.D.,* Aixa Ahamdanech Idrissi, M.D., Josefina del Fresno Cañaveras, M.D., Jose Alejandro Navarro Martinez, M.D., Estefania Paya Martinez, M.D., Ester Sanchez Garcia, M.D.

13. Hospital San Jorge (Huesca), Jorge Vera Bella, M.D.*

14. Hospital Sant Pau (Barcelona), Inmaculada India Aldana, M.D., J. Manuel Campos, M.D., Xavier Pelaez Vaamonde, M.D.*

15. Hospital Santa Maria (Lleida), Montserrat Torra, M.D.*

16. Hospital Universitari del Mar 'Parc de Salut Mar (Barcelona), Raquel Arroyo, M.D., Juan Carlos Cabrera, M.D., Jesús Carazo Cordobes, M.D.,* Lluís Gallart, M.D., Amelia Rojo, M.D., Francisco Javier Santiveri, M.D.

17. Hospital Universitari Germans Trias i Pujol (Badalona), Jaume Canet, M.D.,* Miriam González, M.D., Anabel Jiménez, M.D., Yolanda Jiménez, M.D., Agnès Martí, M.D., Valentin Mazo, M.D., Enrique Moret, M.D., Monica Rodriguez Nuñez, M.D.,* Joaquin Velasco, M.D.

18. Hospital Universitario 12 de Octubre (Madrid), Adriana Calderón, M.D., Matide González, M.D., Olga González, M.D., Ana Hermira Anchuelo, M.D.,* Eloisa López, M.D., Esther Sánchez, M.D.

19. Hospital Universitario de La Princesa (Móstoles-Madrid), Blanca Aznárez Zango, M.D.,* Francisco José García Corral, M.D., Esperanza Mata Mena, M.D., Antonio Planas Roca, M.D.

20. Hospital Universitario de Móstoles (Madrid), Raquel Fernández Rocío Ayala Soto, M.D.,* Borja Quintana, M.D.

21. Hospital Universitario Marques De Valdecilla (Santander), José Manuel Rabanal Llevot, M.D.,* Mónica Mercedes, M.D., Williams Camus, M.D., Alba Palacios Blanco, M.D., Angela Largo Ruiz, M.D.

22. Hospital Universitario Rio Hortega (Valladolid), Jesus Rico Feijoo, M.D.*

23. Hospital Universitario Virgen del Rocio (Sevilla), Elvira Castellano Garijo, M.D.*

24. Hospital Son Llatzer (Palma de Mallorca), Julio Belmonte Cuenca, M.D.,* Marcos José Bonet Binimelis, M.D., Ivaylo Grigorov, M.D., Josep Lluis Aguilar, M.D.

25. Vall d’Hebron University Hospital (Barcelona), Míriam De Nadal Clanchet, M.D., Encarnación Guerrero Viñas, M.D., Susana Manrique Muñiz, M.D., Víctor Martín Mora, M.D., Francisca Munar Bauzà, M.D., Sonia Núñez Aguado, M.D., Montserrat Olivé Vidal, M.D.,* María Luisa Paños Gozalo, M.D., Marcos Sánchez Marín, M.D., María Carmen Suescun López, M.D.

Switzerland

1. Ospedale Regionale di Lugano (Lugano), Paolo Maino, M.D.*

Turkey

1. Medical Faculty of Istanbul, Istanbul University (Istanbul), Semra Kucukgoncu, M.D., Nuzhet Mert Sentürk, M.D.,* Zerrin Sungur Ulke, M.D.

Ukraine

1. St. Katherine Hospital of Cardiology (Odessa), Yevhen Eugene Yevstratov, M.D.*