Article Text
PDF
Abstract
Objective To evaluate whether hospital re-accreditation improves quality, patient safety and reliability over three accreditation cycles by testing the accreditation life cycle model on quality measures.
Design The validity of the life cycle model was tested by calibrating interrupted time series (ITS) regression equations for 27 quality measures. The change in the variation of quality over the three accreditation cycles was evaluated using the Levene’s test.
Setting A 650-bed tertiary academic hospital in Abu Dhabi, UAE.
Participants Each month (over 96 months), a simple random sample of 10% of patient records was selected and audited resulting in a total of 388 800 observations from 14 500 records.
Intervention(s) The impact of hospital accreditation on the 27 quality measures was observed for 96 months, 1-year preaccreditation (2007) and 3 years postaccreditation for each of the three accreditation cycles (2008, 2011 and 2014).
Main outcome measure(s) The life cycle model was evaluated by aggregating the data for 27 quality measures to produce a composite score (YC) and to fit an ITS regression equation to the unweighted monthly mean of the series.
Results The results provide some evidence for the validity of the four phases of the life cycle namely, the initiation phase, the presurvey phase, the postaccreditation slump and the stagnation phase. Furthermore, the life cycle model explains 87% of the variation in quality compliance measures (R2=0.87). The best-fit ITS model contains two significant variables (β1 and β3) (p≤0.001). The Levene’s test (p≤0.05) demonstrated a significant reduction in variation of the quality measures (YC) with subsequent accreditation cycles.
Conclusion The study demonstrates that accreditation has the capacity to sustain improvements over the accreditation cycle. The significant reduction in the variation of the quality measures (YC) with subsequent accreditation cycles indicates that accreditation supports the goal of high reliability.
- joint commission international
- healthcare quality measures
- high reliability
- life cycle model
- interrupted time series analysis
- hospital accreditation
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- joint commission international
- healthcare quality measures
- high reliability
- life cycle model
- interrupted time series analysis
- hospital accreditation
Strengths and limitations of this study
The study uses segmented regression interrupted time series analysis, an alternative to the randomised controlled trial, which is a gold standard by which effectiveness is measured in clinical disciplines.
This is the second interrupted time series analysis on hospital accreditation.
This is also the first study on hospital accreditation over three accreditation cycles and validates the life cycle model on hospital accreditation.
The study is limited to one hospital in the UAE.
The quality measures were dependent on the accuracy of documentation in the patient record.
Introduction
Both the frequency and magnitude of medical errors in hospital settings is a matter of public concern globally. Consequently, healthcare leaders are seeking rigorous methods for improving and sustaining quality of healthcare outcomes in hospitals. Hospital accreditation is the strategy most often selected to improve quality and it has become an integral part of healthcare systems in >90 countries.1
A key constraint for hospitals is the cost of accreditation, a process that consumes resources that could be used for frontline medical services.2 There are two key questions: (1) does accreditation make a difference to the quality of care and hospital performance? and (2) to what extent is any positive effect, if evident, sustainable over time? The literature, however, shows inconsistent results over the impact and effectiveness of hospital accreditation.3–8 Greenfield et al investigated the outcomes across 66 studies and inconsistent findings were reported for the relationship between quality measures and accreditation.5 Furthermore, Devkaran and O’Farrell have argued that rigorous empirical studies that evaluate whether hospitals sustain compliance with quality and patient safety standards over the accreditation cycle are lacking.7 Most previous research has used cross-sectional designs and/or comparative static analysis of data at two points in time.9 10 In order to draw causal inferences on the impact of accreditation on quality and patient safety measures, a dynamic analysis is necessary. This was accomplished by pioneering the use of an interrupted time series model to analyse the impact of accreditation on quality compliance measures in a single hospital over a 4-year period.7 11 We also outlined a new conceptual framework of hospital accreditation—the life cycle model—and presented statistical evidence to support it.7
The primary objective of this paper is to evaluate whether hospital reaccreditation results in an improvement in quality and safety standards over three accreditation cycles by testing the effect of accreditation on 27 quality measures by comparing the results of this hospital (hospital B) accreditation time series with our previous study hospital, a 150-bed, multispecialty, acute care hospital (hospital A) in Abu Dhabi, UAE. The secondary objective is to evaluate the extent to which subsequent accreditation cycles impacts on the variation in quality.
Conceptual framework: the life cycle model
Based on the Joint Commission International (JCI) accreditation strategy, most hospitals will pass through various phases during the process of accreditation.12 Devkaran and O’Farrell hypothesised four distinct phases of the accreditation cycle and derived predictions concerning the time series trend of compliance during each phase.7 The predictions constitute the building blocks of the life cycle model. The first initiation phase is characterised by a gradual improvement in compliance to standards with a positive change of slope for the quality measures. The second—presurvey phase—occurs within 3–6 months of the accreditation survey. A marked improvement (ramp up) in compliance occurs during this phase, because staff are aware of the proximity of the survey and because the organisation invests resources in preparation. The peak level of compliance performance occurs during this phase. During the third phase—the postaccreditation slump—a drop in levels of compliance occurs immediately following the accreditation survey followed by a negative change in slope over time.7 Finally, the stagnation phase follows the postaccreditation slump and there is an undulating plateau of compliance characterised by sporadic changes, but at an overall level substantially above preaccreditation values.7
Methods
Study population
The study was conducted in a publicly funded 650-bed, multispecialty, acute tertiary care hospital in Abu Dhabi, UAE. The annual inpatient census is approximately 18 000. The hospital treats approximately 220 000 ambulatory care patients per year.
Patient involvement
No patients were involved in this study.
Data collection
To test the life cycle model, a total of 27 quality measures were recorded each month at the hospital, over an 8-year period, including three JCI accreditation surveys (table 1). The quality measures were selected by an expert panel to ensure the: (1) interpretability, enabling direct correlation with a specific JCI standard; (2) consistency, with high values indicating better quality and (3) systems-based, measures designed to evaluate a system/domain of quality rather than a single process. The measures represent both important indicators of quality which are primarily reviewed during survey tracers—including patient assessment, surgical procedures, infection control and patient safety—and 9 of the 14 chapters of the JCI Hospital Standards manual.13
Quality measure descriptions for the Sheikh Khalifa Medical City (hospital B) time series analysis
The outcome measures for the time series analysis incorporated clinical quality measures and were expressed as percentages, proportions or rates, which minimises ceiling effects (table 1). These performance differences were compared across monthly intervals between four time segments, 1-year preaccreditation, 3 years postaccreditation cycle 1, 3 years postaccreditation cycle 2 and 1 year postaccreditation cycle 3 for the selected quality measures. This study had more than the minimum number of eight data points before and after the intervention and thus had sufficient power to estimate the regression coefficients.14 The larger number of data points (96) permit more stable estimates for forecasting preintervention trends had the intervention not occurred. The principal data source was the electronic medical record. Slovin’s formula was used to calculate the sample size per month based on a 95% CI from an average monthly inpatient census of 1400 patients. Each month (during the entire investigation period), a simple random sample of 10% of patient records was selected and audited from the monthly population resulting in a total of 388 800 observations from 14 500 records. The internal data validation process in place within the hospital included: recollecting the data by second person not involved in the original data collection; using a statistically valid sample of records, cases or other data; comparing the original data with the recollected data; calculating the accuracy by dividing the number of data elements found to be same by the total number of data elements and multiplying that total by 100. A 90% accuracy level was considered as an acceptable benchmark. When the data elements differed, the reasons were noted (eg, unclear data definitions) and corrective actions were taken. A new sample was collected after all corrective actions have been implemented to ensure the actions resulted in the desired accuracy level. The sources used for the data validation included, but were not limited to, the electronic medical record and data abstracts; enterprise resource planning software; electronic insurance claims and the adverse event reporting system.
Quality measures that displayed an inverse relationship to percentage measures were transformed. For example, ‘percentage of patients with myocardial infarction within 72 hours after coronary artery bypass graft surgery’ was transformed to ‘percentage of patients without myocardial infarction within 72 hours after coronary artery bypass graft surgery’, thus equating higher values to good quality.
Study design
Interrupted time series analysis is the most powerful quasi-experimental design for evaluating the longitudinal effects of an intervention (eg, accreditation) on an outcome of interest where the trend before the accreditation intervention is used as a control period. The advantage of this method over a simple before-and-after study is due to the repeated monthly measures of variables, while controlling for seasonality and secular trends. Shifts in level (intercept) or slope, with p<0.05, were defined as statistically significant. Segmented regression models fit a least squares regression line to each segment of the independent variable, time and thus assume a linear relationship between time and the outcome within each segment.14–18 The following linear regression equation is specified to estimate the levels and trends in the dependent variable before each of three accreditations, and the changes in levels and trends after each accreditation:
(1)
Where Yt is the outcome, for example, the inpatient fall rate per 1000 patient days; time t1, t2, t3 and t4 indicates time in months from the start of each observation period to the end of the period; interventions I1, I2 and I3 are dummy variables taking the value 0 before the intervention and one after the intervention. In this model β0 is the baseline level of the outcome at the beginning of the series; β1 the slope prior to accreditation, that is the baseline trend; β2, β4 and β6 are the changes in level immediately after each accreditation and β3, β5 and β7 are the changes in slopes from preaccreditation to post the three accreditations, respectively, and represents the monthly mean of the outcome variable; and et is the random error term.
Data analysis
First, a plot of observations against time was completed in order to reveal key features of the data, including trend, seasonality, outliers, turning points and any discontinuities. Second, segmented regression models were fitted using ordinary least squares regression analysis; and the results reported as level and trend changes. Third, the Durbin-Watson (DW) statistic was used to test for the presence of two types of autocorrelation: (1) the autoregressive process and (2) the moving average process. If the DW was significant, the model was adjusted by estimating the autocorrelation parameter and including it in the segmented regression model. Fourth, the Dickey-Fuller statistic was used to test for stationarity and seasonality. A series displaying seasonality or some other non-stationary pattern was controlled by taking the difference of the series from one period to the next and then analysing this differenced series. Since seasonality induces autoregressive and moving average processes, the detection and inclusion of a seasonal component was implemented in the time series models using the autoregressive moving average (ARMA), ARMA and dynamic regression. A range of model-checking techniques have been used including plotting residuals and partial autocorrelation functions as well as sensitivity analyses. Fifth, there were no significant hospital changes (ie, change in ownership, construction, capacity or scope change >25% of the patient volume, addition of services or mergers) that occurred during the study period based on the JCI accreditation participation requirements 3.13 Furthermore, the leadership and composition of the quality and safety programme remained the same throughout. Therefore, it may be assumed that the accreditation interventions were the key events to impact the time series. The analysis was conducted using EViews 7.0. In order to verify whether the accreditation process exhibits the life cycle effect, the statistical predictions specified for the 27 measures were tested.
The ultimate confirmatory test of the life cycle model and of the impact of three separate accreditations is to aggregate the data for all 27 quality compliance measures to produce a composite score (YC) and to fit an interrupted time series regression equation to an unweighted mean monthly value of the series. The composite measure assumed that all of the 27 indicators have the same weight.
Results
The descriptive statistics of the dependent variables are depicted in table 2 and demonstrate that 88% of measures had a mean and median >90%. The data were symmetrical as the means and medians were similar for all measures. In terms of dispersion, 74% of measures have a SD of 3 or less. The measure Y22 has the lowest mean and the highest SD. Table 3 outlines the interrupted time series equations for the 27 quality compliance measures. Several equations display autocorrelation, in which cases the autoregressive (AR) or moving average (MA) variable was included to correct for it. First, 78% of the β1 coefficients (the slope prior to the first accreditation) are positive, as predicted and half are statistically significant correlating with the presurvey ramp up phase in the life cycle model (table 3). Conversely, 26% of the coefficients are negative, but only three are significant. Second, the β2 coefficients—the change in level following the first accreditation—are negative and significant in five cases and positive and significant in six. Hence, in 60% of cases the first intervention effect is not significant. The β3 slope coefficient results are more mixed following the first accreditation: in five cases, coefficients are both negative and significant, and also five are positive and significant. Conversely, for 63% of cases there is no significant effect. Fourth, in the case of the second intervention, β4, seven coefficients are both negative and significant, whereas only four positive coefficients are significant. For β5, the second postaccreditation slope, 59% of the coefficients are not significant but 8 of the 11 significant slopes are negative. Similarly, some 85% of the coefficients on (β6)—the third intervention—are not statistically significant; and, finally, 86% of the postaccreditation slopes (β7) are not significant (table 3). The mixed results at the level of individual measures provide limited support for the life cycle model with the exception of the presurvey ramp up phase (β1).
Results of descriptive statistical analysis for the 27 quality measures
Time series analysis for the 27 quality measures
The results of the overall test, using a composite score (YC), are summarised in table 4. Diagnostic assumption tests show that there is autocorrelation. Hence, the model was adjusted by estimating the autocorrelation parameter AR (1) and incorporating it in the segmented regression model; inclusion of it eliminates the autocorrelation problem (table 4).
Interrupted time series composite model for the 27 quality measures
The slope prior to the first accreditation (β1) is positive and highly significant (presurvey ramp up phase), as predicted by the life cycle model of Devkaran and O’Farrell.7 11 The change in level following the first accreditation survey (β2) is unexpectedly positive, but is not significant. The postaccreditation slope (β3), however, is negative and statistically significant (postaccreditation slump), as postulated by the model. The changes in level following the second and third accreditation surveys are both negative, but are not significant (table 4). Similarly, the postaccreditation slopes following these two later surveys are both negative, as hypothesised, but are not significant. The R2 value for the composite model with the AR (1) function indicates that over 87% of the variation in quality compliance outcomes is explained (table 4). There is, however, a problem with multicollinearity. Inspection of the three postaccreditation slopes in figure 1 shows a long gently undulating plateau of compliance which is consistent with the non-significance of the second and third accreditations; and is substantiated by the evidence that the mean compliance level before the first accreditation was 89.2% and, following the three accreditations, the mean levels were 95.2%, 96.3% and 97.4%, respectively. The evidence for the life cycle model is stronger in the case of the first hospital accreditation survey than in the subsequent accreditations. Given that our model has a high R2 value of 0.87, it is a useful predictive tool, although it results in somewhat unstable parameter an estimate which makes it more difficult to assess the effect of individual independent variables.
Phases of the accreditation life cycle: empirical evidence over 8 years.
Clearly, we cannot forecast precisely what would have occurred if the one accreditation in 2008 had not been followed by subsequent survey visits in 2011 and 2014, that is, the counterfactual position. However, if compliance had been allowed to slip following the first survey, it would be expected that improvements in quality would occur both before the second and third surveys in 2011 and 2014; and that there would also be falls in levels of compliance immediately following these surveys. None of these outcomes occurred. This implies that once a high level of compliance has been achieved after the initial accreditation survey, it is highly likely to be maintained (figure 1).
Finally, we compare the results of the composite model (YC) for the 27 measures (hospital B) with that of the 23 quality measures for the 150-bed hospital A (figure 2).7 11 A number of interesting patterns are apparent. First, the slopes prior to accreditation (β2) are both positive and highly significant, as hypothesised. Second, the change in level following the first accreditation survey (β3) signals a significant decline in compliance, as predicted, in the case of hospital A; while for the current study, hospital B, the effect is not significant. Third, as postulated, the postaccreditation slope (β3) is both negative and statistically significant for each hospital. Fourth, there is a striking similarity in the shape of the two graphs with a marked improvement in compliance during the first presurvey phase; a drop in the level of compliance following the accreditation survey at hospital A, while similar falls in level were recorded after two of the three accreditations at hospital B, followed in both hospitals by undulating plateaus of compliance, at a level substantially greater than those recorded prior to the first accreditation survey. Fifth, a notable feature of the results is that, although the pattern of compliance change is very similar, the level of compliance at hospital B is slightly higher: for the presurvey phase the average level of compliance at hospital B is 87.40% compared with 79.5% at hospital A; while for the postaccreditation period, the hospital B compliance average of 96% also exceeds that of hospital A (93%). It is important to note that both hospitals adopted the same approach to accreditation and survey preparation by following the JCI roadmap to accreditation.12 13
Life cycle model comparison between hospital A (previous study) and hospital B (current study).
Finally, having demonstrated that there is a significant difference between group means of the composite measure (YC), we tested the null hypothesis that there is no significant difference between the group variances. The results of Levene’s test show that the hypothesis of homogeneity of variances is rejected at p<0.05 (table 5). Therefore, there is a significant difference between the four group variances and figure 3 shows that the variances decrease after each successive accreditation. Hence, with the exception of the means for groups C and D, successive accreditations lead to an increase in the mean and decrease in the variance of the composite compliance measure (YC). The results of the confirmatory test of the proposed life cycle model, using a composite score (YC) of the 27 quality measures, provide proof of the life cycle model.
Anova and Levene’s test for variances between accreditation cycles
Box plot comparing variation in performance between the accreditation cycles.
Discussion
Empirical evidence to support the effectiveness of accreditation is still lacking, which creates a legitimacy problem for healthcare policymakers and hospital management.4 Is achieving and, above all, maintaining accreditation worth the time and money if there is uncertainty about whether it results in measurable improvements in healthcare delivery and outcomes?2–6 19 While accreditation enhances quality performance, its major benefit lies in organisations integrating standards into their routine workflows. Integration ensures that the ramp up to surveys is avoided and that organisations reliably apply the evidence-based practices for each patient during each encounter.
Unannounced surveys
Announced triennial surveys have been criticised for permitting healthcare organisations to perform for the ‘test’; and, when the accreditation survey is completed, facilities may return to their presurvey reality. Therefore, unannounced surveys have been proposed to mitigate this effect and to encourage a continuous improvement culture. However, there are only two published (Australian and Danish) studies comparing announced and unannounced surveys. Both studies show no evidence of increased citations of non-compliance in unannounced surveys compared with announced surveys.20 21
Continual survey readiness
Rather than assign the accountability to accreditation bodies for the associated life cycle, organisations need to review their own continual survey readiness strategies. The components of an effective continual survey readiness programmes remain unexplored. Therefore, the authors propose a survey readiness cycle that is grounded on the four phases of the accreditation life cycle model, supported by the literature and influenced by the Institute for Healthcare Improvement Model for Improvement.20–23 The proposed survey readiness cycle consists for four components: (1) a gap analysis; (2) a mock survey; (3) postsurvey action plans that occur after the actual survey and (4) intracycle internal reviews and improvement (figure 4). For the cycle to be effective, a leadership oversight body needs to be created with the objective of conducting regular reviews of compliance using associated metrics to ensure that the process is sustained. If an accredited organisation has integrated the standards into routine practice with a foundation that is built on fundamental patient safety principles, they are likely to minimise errors. Furthermore, when hospitals consistently perform according to standard, they attain the status of a high reliability organisation.23
Wheel of continual survey readiness. Adapted from Devkaran and O’Farrell.7,11
Conclusion
We pioneered the first study of hospital accreditation to conduct a dynamic analysis of the impact of accreditation on quality compliance measures using interrupted time series analysis.7 11 This paper has advanced the research in several important ways: (1) by studying another hospital over an extended 8-year period; (2) by conducting an interrupted time series analysis of 27 quality compliance measures over a period incorporating three separate accreditation evaluations and (3) by demonstrating that subsequent accreditation surveys significantly reduces variation in quality performance which correlates with higher reliability.
The evidence from both hospital studies suggests that the tangible impact of accreditation has the capacity to sustain improvements over the accreditation cycle. Our results suggest that once a high level of quality compliance has been achieved—following the first accreditation visit—it is highly likely to be sustained. In addition, repeated surveys reduce variations in quality performance therefore supporting the organisation’s journey to high reliability.
The following limitations should be acknowledged. First, the accuracy of measures is dependent on the quality of documentation in the patient record. For instance, if the documentation was deficient then this was reflected in the measure. Second, the choice of quality measures is defined by the availability of evidence in patient records. Third, this study is set in the UAE and may not be generalisable to hospitals in other settings. Fourth, both study hospitals provide acute tertiary care and have limited generalisability to specialty hospitals or primary/secondary care healthcare facilities. Fifth, interrupted time series analysis is limited by time-varying confounding improvement initiatives that may have occurred at the department level however, since this methodology evaluates changes in rates of an outcome at a system-level, confounding by individual level variables will not introduce serious bias unless it occurs simultaneously with the intervention. Sixth, although ceiling effects were minimised, it is acknowledged that if a measure is close to 100% then any subsequent improvement will only be small. However, our analysis has shown conclusively that each successive accreditation lead to an increase in means and also to a decrease in the variances of the composite measure. Finally, more studies are required to evaluate methodologies for achieving continuous survey readiness.
Acknowledgments
The authors would like to thank the SKMC Quality Team and SKMC caregivers, whose commitment supported the outcomes achieved.
References
Footnotes
Contributors SD conceived and designed the experiments. SD analysed the data: SD and PNO’F interpreted the data and wrote the manuscript. SD and PNO’F jointly developed the model and arguments for the paper. SD and PNO’F revisited and revised the article for important intellectual content. ‘SE provided access to the data.’ As well as review of the article. RA provided technical assistance with the statistical analysis. All authors reviewed and approved of the final manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement Additional data is available from the author based on request.