Accreditation of acute health services in Australia

The International Society for Quality in Health Care (ISQua), the peak body for health services accreditation, defines accreditation as ‘public recognition of the achievement of standards by an organisation demonstrated through independent assessment in relation to set standards’.6 Accreditation has been widely implemented following the establishment of the Joint Commission on Accreditation of Hospitals (now, the Joint Commission) in the USA in 1951.7 ,8 In Australia, accreditation was first adopted for acute care services in 1974,9 with the Australian Council on Healthcare Standards (ACHS) as the main accrediting agency. Over subsequent years, ACHS developed the Evaluation and Quality Improvement Program (EQuIP). This accreditation programme comprises two external surveys within a 4-year accreditation cycle.10 Facilities are assessed by trained external surveyors using standards developed by ACHS in consultation with healthcare industry experts.10 As part of the wider health system reforms implemented by the Australian Commonwealth Government, recently approved legislation requires all hospitals and day procedure services in Australia to be assessed by an accreditation provider approved by the Australian Commission on Safety and Quality in Health Care (ACSQHC) using newly developed National Safety and Quality Health Service (NSQHS) standards.11 Australian acute care accreditation is the focus of this study. We include acute and high-risk inpatient activity in our definition of acute care facilities (ACFs) in this protocol.

Economic evaluation of health interventions

Health costs are a significant proportion of gross domestic product, averaging 9.6% in 2010 for Organisation for Economic and Co-operation Development countries.12 This, combined with persistent evidence of harm during health service delivery,13–16 has resulted in an increasing international focus on accountability and safety in healthcare.17 ,18 Economic evaluation addresses these issues by using a systematic framework to identify and compare the costs and benefits of a policy or intervention to determine whether implementation is effective in achieving stated aims and also to compare different policy proposals and interventions.19–21 In cost–benefit analysis (CBA), the costs and benefits are each expressed in monetary terms. This contrasts with other techniques such as cost-effectiveness analysis (CEA) or cost-utility analysis (CUA), which are used in assessing health technologies. In CEA and CUA, the common outcome denominator is not monetised, but expressed in terms of a common utility measure such as Quality Adjusted Life Years.22–25 CBA is advocated where there is a broader range of outcomes and is usually a requirement for submitting a regulatory impact statement for Australian government approval.19 CBA seems justified as the most appropriate model to use in an Australian health services context as the NSQHS standards cover both organisational and clinical outcomes, which are best measured using a common monetary denominator. Although CBA can be used as an allocative efficiency tool for comparing different projects, the requirement to make the costs and benefits explicit in the analysis framework can help clarify the goals, costs and benefits, providing input into the design of future accreditation systems in healthcare.26

Modelling the costs and benefits of a complex intervention, such as accreditation, in a complex system, such as an ACF, is a significant undertaking.27 Given the lack of precedent discussed above, we have developed a unique framework and a number of purpose-built tools specifically designed for evaluating acute care accreditation. Our SIQNS framework is synthesised from several sources,20 ,21 ,28 ,29 and comprises five discrete activities (1) scope and objectives; (2) identify costs and benefits; (3) quantify costs and benefits; (4) calculate net social benefits and (5) sensitivity analysis (figure 1).

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

SIQNS framework.

Scope and objectives

The aim of the study is to create an interactive model that can be accessed by health service providers, accreditation bodies, quality and safety agencies, governments and researchers to both test the assumptions in the model and to determine the cost–benefit calculations of acute care accreditation at both the national and local service levels.28 The lack of research in this field, as well as the complexity of both the intervention (accreditation) and system (acute healthcare), indicates that an important objective will be to make explicit the costs and benefits of accreditation.1 ,2 ,27 ,29 ,30

Establishment of the study parameters will be informed by a review of the research literature and analysis of the characteristics of accredited ACFs. These activities will also help determine the indicator selection process to identify and quantify the benefits of accreditation. A critical element is to identify the stakeholders involved in the acute care accreditation process. Although accreditation agencies and ACFs are the most obvious groups affected, a broader societal framework is required when assessing regulatory impact to ensure that equity and impact are meaningfully accounted for in the analysis.20

Our initial analysis has identified a key constraint, in that ACFs do not account for accreditation activities as a separate cost item.31 Our approach, outlined below, is to accept that large-scale data gathering on costs would not be possible within the scope of the study and will be a challenge over time. Instead, we will use a smaller study sample and have this validated by an expert panel. Accreditation benefits, in terms of both clinical and organisational outcomes, are more likely to be assessed using secondary data, but are more difficult to identify partly owing to a lack of clarity in terms of measurable endpoints. In addition, a full impact analysis of all benefits for all stakeholders will be outside the study scope. A further constraint is that an economic appraisal would ideally be conducted with a control group (either randomly assigned or with data collected for a before and after comparison) in order to compare different outcomes.32 This is not possible in Australia where acute care accreditation is widely implemented (93% of public hospital beds in 2010 and 84% of private hospital beds in 2008–2009).33 However, we can analyse data before and after introduction of new standards, review indicator activity against accreditation scores and analyse changes over time.

Identify costs and benefits

To identify the incremental costs of accreditation, we need to determine costs that are only incurred due to the accreditation process.34 We will first review the stakeholders for potential costs and exclude transfer payments to avoid double counting. We will then assess costs for a small number of ACFs (n=10) using our purpose designed assessment tool based on questions posed in similar cost surveys,35–38 and the Business Cost Calculator which has been designed to help Australian businesses compute business compliance costs.7 ,10 ,39 ,40

International accreditation agencies, for example, the Joint Commission, ACHS and Accreditation Canada, identify improvements in patient safety and quality of care as the main benefits of health services accreditation.7 ,10 ,40 Within an Australian context, we reviewed the benefits outlined by ACSQHC in a recent regulatory impact statement on the new NSQHS standards.41 In addition, the Australian National Health Performance Authority's (NHPA's) performance and accountability framework has defined the indicators used to assess effectiveness of care in ACFs. We will map our stakeholder analysis with the ACSQHC benefits and the NHPA adverse and sentinel event measures to identify quality and safety indicators.

In our choice of indicators, we need to distinguish whether a lack of change in the indicator is due to a lack of compliance with the accreditation standard, or whether compliance with the standard results in change in an indicator. We have adapted the approach used in measuring quality outcomes in US hospitals.42 This recommends that indicators must meet four key accountability criteria: research—robust evidence; accuracy—whether the process has been carried out satisfactorily; proximity—a clear and direct link between accreditation and the indicator; and, no adverse effects—no unintended or unwanted actions. As accreditation is usually just one facet of a quality and safety framework, we have added a further criterion—specificity—to determine how easy it is to isolate the effects of accreditation from other safety and quality measures. This is important when determining the effectiveness of accreditation versus other safety and quality initiatives.43 We specify the type of indicator, as process indicators are often a preferred measure of quality over outcome indicators. This is due to a more direct link between the indicator and the process being measured (the issue of proximity in Chassin's accountability criteria),42 but this can also lead to estimation problems as only a narrow range of factors is considered. Outcome indicators have an advantage in that the data are often collected routinely, but may need to be adjusted for other factors such as patient acuity and complexity.43–45 We will need to consider whether the chosen indicators need to be weighted to reflect both suitability (in terms of adherence to the accountability criteria) and applicability (quality of data collection and adjustment for patient mix). We will ask an expert panel to review and validate our indicator selections prior to quantifying the indicators for further analysis.

Quantify costs and benefits

Once the costs have been identified for each ACF in our sample, we will need to ensure that the results can be scaled up and assessed on a national basis. Although costs for initial accreditation can be higher than for ongoing accreditation,38 ,46 we assume ongoing accreditation costs for our sample, but include questions on estimating the cost of implementing the new national standards in the survey.

As the CBA model uses monetary values as the common denominator, we will need to monetise the benefits identified. The techniques used will depend on the availability of pricing and market data. For example, clinical outcome indicators (such as hospital-acquired infection rates, or complications resulting from inpatient falls) can be matched to activity-based costing codes. These data can be used to determine the potential cost savings from a reduction in infection or fall rates. Where we identify an indicator but do not see a change in measurement, we will include this in our sensitivity analysis, for example, the costs associated with reducing hospital-acquired infections by a stated amount. Where indicators can be identified and quantified but lack pricing or market data, we will monetise the effects using techniques such as revealed and stated preference methods, where possible.21 ,26 ,47

Calculate net social benefit

Given that costs and benefits are likely to occur at different times, we will need to adjust the timing differentials using an appropriate discount rate. For CBA, both the net social benefit (NSB) and the benefits–cost ratio (BCR) are calculated. The NSB uses a net present value method to derive an absolute measure of whether the discounted (net) benefits are greater than the discounted (net) costs when assessing proposals in a regulatory impact statement.19 ,22 The BCR is derived by dividing the net benefits by the net costs to determine an effective return on the costs and is used where the absolute size of the investment is a determining factor, for example, in a resource-constrained environment. However, the BCR is subject to more variation depending on how the outcomes are treated.19 ,20 ,22

With questions on the table about whether accreditation is more of an audit tool or quality improvement tool,1 ,8 CBA can address both these issues by comparing accreditation not only with other safety and quality programmes, but also with other methods of regulatory compliance. Although there is no good counterfactual to accreditation due to widespread implementation,33 we can estimate the relative effectiveness of accreditation versus alternative forms of audit. Examples include the additional requirements that a private ACF would need to meet to qualify for private health funding, or additional auditing that would be required for public ACFs in the absence of accreditation.

Sensitivity analysis

A sensitivity analysis is an essential tool to describe the impact of changes in assumptions and variables on our SIQNS framework,20 especially given the constraints discussed. We will review the assumptions in the model and run the NSB and BCR calculations over a range of values. For some costs and benefits, including qualitative outcomes, monetisation will be beyond the scope of this study and these items will not be included in the NSB or BCR calculations. Nevertheless, these costs and benefits can still be included in the final model and used to compare other programmes with similar NSB or BCR outcomes, but where the non-monetised items may be a deciding factor.

ACCREDIT project overview

The ACCREDIT CBA (Acute) study is the 6th of 12 studies under the ACCREDIT (Accreditation Collaborative for the Conduct of Research, Evaluation and Designated Investigations through Teamwork) research collaboration,28 ,48 ,49 funded by the Australian Research Council through its industry Linkage Program.50 The ACCREDIT collaboration involves researchers in the Centre for Clinical Governance Research and Centre for Health Systems and Safety Research in the Australian Institute of Health Innovation (AIHI) at the University of New South Wales (UNSW), Australia. The ACCREDIT research team benefits from a high-profile international advisory group containing leading researchers in health safety and quality from the UK, Spain and Sweden. The collaboration includes two leading health safety and quality bodies (ACSQHC and the Clinical Excellence Commission (CEC)) plus three of the major Australian health services accreditation agencies: ACHS, Australian General Practice Accreditation Limited (AGPAL) and the Aged Care and Standards Accreditation Agency (ACSAA).