Clinical ScienceDetecting adverse events in surgery: comparing events detected by the Veterans Health Administration Surgical Quality Improvement Program and the Patient Safety Indicators
Section snippets
Methods
We used a retrospective study design to compare surgical AEs detected by VASQIP with AEs detected by the PSIs from October 1, 2002, through September 30, 2007 (FY 2003 to FY 2007) in the VA. After our assessment of PSI criterion validity, we sampled cases for chart review to understand why some AEs were detected by only 1 method. Our study was approved by the VA Boston Healthcare System Institutional Review Board and the VA Surgical Quality Data Use Group.
Results
Eighty-nine percent of the 268,771 hospitalizations in our PTF-VASQIP matched sample were eligible for ≥1 of the 5 PSIs we examined, and 6,100 of these were flagged. In comparison, VASQIP found ≥1 of the AEs of interest in approximately 15,000 hospitalizations (6%); however, most of these occurred after discharge or in cases that did not meet PSI eligibility criteria. Table 2 shows the number of hospitalizations in our matched sample that were eligible for the respective PSI, as well as the
Comments
Using a large sample of VA acute hospitalizations and 5 years of data, we compared AEs identified by PSI version 4.1a and VASQIP and estimated PSI criterion validity. We found high specificity estimates for all 5 PSIs; low sensitivity for the PSIs PMD, sepsis, and WD (31%–48%), and moderate sensitivity estimates for the remaining 2 PSIs, respiratory failure and PE or DVT (65%–68%). The PSIs PE or DVT and sepsis had low PPV estimates (31% and 44%, respectively), and the PSIs PMD, respiratory
References (42)
- et al.
Assessment of the reliability of data collected for the Department of Veterans Affairs National Surgical Quality Improvement Program
J Am Coll Surg
(2007) - et al.
Design and statistical methodology of the National Surgical Quality Improvement Program: why is it what it is?
Am J Surg
(2009) - et al.
Informatics and the American College of Surgeons National Surgical Quality Improvement Program: automated processes could replace manual record review
J Am Coll Surg
(2009) - et al.
Clinical validation of the AHRQ postoperative venous thromboembolism patient safety indicator
Jt Comm. J Qual Patient Saf
(2009) - et al.
How best to measure surgical quality? Comparison of the Agency for Healthcare Research and Quality Patient Safety Indicators (AHRQ-PSI) and the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) postoperative adverse events at a single institution
Surgery
(2011) - et al.
The measurement and monitoring of surgical adverse events
Health Technol Assess
(2001) - et al.
Surgical adverse events, risk management, and malpractice outcome: morbidity and mortality review is not enough
Ann Surg
(2003) - et al.
To Err is Human: Building a Safer Health System
(1999) - et al.
Framework for patient safety research and improvement
Circulation
(2009) - et al.
Identifying adverse drug events: development of a computer-based monitor and comparison with chart review and stimulated voluntary report
J Am Med Inform Assoc
(1998)
A comparison of hospital adverse events identified by three widely used detection methods
Int J Qual Health Care
Hospital staff should use more than one method to detect adverse events and potential adverse events: incident reporting, pharmacist surveillance and local real-time record review may all have a place
Qual Saf Health Care
The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II
N Engl J Med
The Department of Veterans Affairs' NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care
Ann Surg
Fifteen years of the National Surgical Quality Improvement Program in review
Am J Surg
Does surgical quality improve in the American College of Surgeons National Surgical Quality Improvement Program: an evaluation of all participating hospitals
Ann Surg
Assessing surgical quality using administrative and clinical data sets: a direct comparison of the University HealthSystem Consortium Clinical Database and the National Surgical Quality Improvement Program data set
Am J Med Qual
Patient Safety Indicators: using administrative data to identify potential patient safety concerns
Health Serv Res
Evaluating the patient safety indicators: how well do they perform on Veterans Health Administration data?
Med Care
Quality improvement implementation and hospital performance on patient safety indicators
Med Care Res Rev
Cited by (25)
Relationship Between Perioperative Outcomes Used for Profiling Hospital Noncardiac Surgical Quality
2021, Journal of Surgical ResearchCitation Excerpt :Similarly, hospital rankings based on complications have been compared to rankings based on mortality and FTR and found to correlate poorly.6,7 However, these comparisons were from the early 1990s and based on administrative data, for which the ascertainment of complications is less accurate when compared to clinical registry data.8-10 While information about complications is collected and reported by VASQIP and ACS-NSQIP, our data suggest there is still low correlation between hospital performance in terms of mortality and morbidity.
Surgeon-Reported Complications vs AHRQ Patient Safety Indicators: A Comparison of Two Approaches to Identifying Adverse Events
2018, Journal of the American College of SurgeonsCitation Excerpt :Other evidence supports that the AHRQ PSIs are inadequate at capturing all events of interest to surgeons. Using 5 years of data from a large sample of Veterans Health Administration hospitals, Mull and colleagues19 found poor overlap in adverse events identified by the AHRQ PSIs and the Veterans Affairs Surgical Quality Improvement Program, with differences in definitions and coding of these complications. Another study at a single academic institution found that American College of Surgeons NSQIP identified 7.4% of patients with an adverse event compared with 3.5% identified by the AHRQ PSIs.20
Combined medical quality assessment using the evidential reasoning approach
2015, Expert Systems with ApplicationsCitation Excerpt :On request from the studied hospitals, we use Hospital A, Hospital B, and Hospital C to anonymously represent the three studied hospitals in the paper. Traditionally, quality indicators including overall hospital mortality rate (Kipnis et al., 2010), readmission rate (Halfon et al., 2006; Weissman et al., 1999), and adverse events rate (Mull et al., 2014) regardless of differences in diseases and patients were used as measures to assess the overall quality of medical care of hospitals. Furthermore, for more accurate hospital quality assessment, main diseases or operations for which the outcomes are most likely to be causally related to antecedent medical care were selected to calculate indicators separately or aggregately for quality comparison (Baker et al., 2002; Glance et al., 2010; Wray et al., 1997).
The authors declare no conflicts of interest.