Article Text

PDF

Development of evidence-based Australian medication-related indicators of potentially preventable hospitalisations: a modified RAND appropriateness method
  1. Gillian E Caughey1,
  2. Lisa M Kalisch Ellett1,
  3. Te Ying Wong2
  1. 1Quality Use of Medicines and Pharmacy Research Centre, Sansom Institute, School of Pharmacy and Medical Sciences, University of South Australia, Adelaide, Australia
  2. 2School of Pharmacy and Medical Sciences, University of South Australia, Adelaide, Australia
  1. Correspondence to Dr Lisa Kalisch Ellett; lisa.kalisch{at}unisa.edu.au

Abstract

Objective Indicators of potentially preventable hospitalisations have been adopted internationally as a measure of health system performance; however, few assess appropriate processes of care around medication use, that if followed may prevent hospitalisation. The aim of this study was to develop and validate evidence-based medication-related indicators of potentially preventable hospitalisations.

Setting Australian primary healthcare.

Participants Medical specialists, general practitioners and pharmacists. A modified RAND appropriateness method was used for the development of medication-related indicators of potentially preventable hospitalisations, which included a literature review, assessment of the strength of the supporting evidence base, an initial face and content validity by an expert panel, followed by an independent assessment of indicators by an expert clinical panel across various disciplines, using an online survey.

Primary outcome measure Analysis of ratings was performed on the four key elements of preventability; the medication-related problem must be recognisable, the adverse outcomes foreseeable and the causes and outcomes identifiable and controllable.

Results A total of 48 potential indicators across all major disease groupings were developed based on level III evidence or greater, that were independently assessed by 78 expert clinicians (22.1% response rate). The expert panel considered 29 of these (60.4%) sufficiently valid. Of these, 21 (72.4%) were based on level I evidence.

Conclusions This study provides a set of face and content validated indicators of medication-related potentially preventable hospitalisations, linking suboptimal processes of care and medication use with subsequent hospitalisation. Further analysis is required to establish operational validity in a population-based sample, using an administrative health database. Implementation of these indicators within routine monitoring of healthcare systems will highlight those conditions where hospitalisations could potentially be avoided through improved medication management.

Statistics from Altmetric.com

Strengths and limitations of this study

  • The clinical indicators developed were based on high-level evidence together with expert clinical panel assessment.

  • Since the clinical indicators were developed using Australian-specific resources they may need to be adapted for use in other settings.

  • This study provides a set of face and content validated indicators of potentially preventable hospitalisations, linking to suboptimal processes of care and medication use with subsequent hospitalisation.

Introduction

Clinical indicators of potentially preventable hospitalisations are used as a measure of health system performance and quality of healthcare provided to patients.1 ,2 Potentially preventable hospitalisations are defined as those hospitalisations that could be prevented with the provision of timely and effective primary care.3 Medication-related hospitalisations are relatively common. A literature review found that 2–3% of all hospital admissions in the Australian healthcare setting were medication related, with half considered to be potentially preventable.4 A systematic review of studies from around the world found that a median of 3.7% of all hospital admissions were preventable medication-related admissions.5 The identification and subsequent reduction of the most common medication-related potentially preventable hospitalisations will improve morbidity and quality of life for patients, safety of the healthcare system and reduce healthcare expenditure.

Clinical indicators of medication-related potentially preventable hospitalisations have been developed which link suboptimal processes of care with medicine use to assess adverse outcomes including hospitalisation.2 ,6–8 The overall incidence of preventable medication-related hospitalisations when measured using these clinical indicator sets has been reported to range between 3% and 20%, depending on the country of the study population and the clinical indicator set used.9–11 Using the previously developed clinical indicators, the prevalence of potentially preventable medication-related hospitalisations in the Australian healthcare setting between 1 January 2004 and 31 December 2008 was examined. During the 5-year study period there were 44 416 (20.5%) potentially preventable medication-related hospitalisations, equating to 9000 preventable admissions each year.11 However, in undertaking the study, it became apparent that many of the internationally developed indicators were not relevant or applicable to the Australian healthcare setting. A cross-country comparison between the USA and the UK indicators found that of the 46 indicators assessed, 58% were relevant to the USA but not the UK, and only 41% were deemed to be relevant in the healthcare setting of both countries.8 Given the significant differences between the USA and the UK healthcare systems to that of Australia, there may be additional indicators, specifically relevant to the Australian healthcare system, that should be examined. Further, the international indicators were developed over 10 years ago and there are likely to be a number of indicators based on new medicines introduced since then.

Prior studies which developed clinical indicators for potentially preventable medication-related hospitalisations used the Delphi technique,2 ,6 ,8 ,12 ,13 which measures consensus among experts using a series of structured surveys.14 ,15 Recent studies have highlighted the need for clinical indicators to be evidence based, rather than based on expert consensus only.11 ,13 ,16 Increasingly, the RAND appropriateness method is used in indicator development,17 ,18 which develops indicators by combining evidence-based recommendations from clinical guidelines with expert clinical opinion. In addition, recent studies have highlighted the need for clinical indicators to be country specific to reflect current practice within individual healthcare systems.11 ,13 ,16 To date, no evidence-based indicators of medication-related potentially preventable hospitalisations have been developed specific for the Australian setting. The aim of this study was to develop and validate Australian evidence-based medication-related indicators of potentially preventable hospitalisations.

Methods

A modified RAND appropriateness method was used for the development of medication-related indicators of potentially preventable hospitalisations, which has characteristics of both the Delphi and Nominal Group Techniques, providing a systematic method to combine evidence with expert opinion.19 It consists of a literature review, assessment of the strength of the supporting evidence base, an initial face and content validity assessment by an expert panel, followed by an independent assessment of indicators by an expert clinical panel across various disciplines, using an online survey.

Identification of existing indicators and development of new indicators

A number of methods were used to systematically identify and develop clinical indicators for medication-related potentially preventable hospitalisations, specific for the Australian healthcare setting. A literature review was conducted to identify all published studies of indicators for preventable medication-related hospitalisations that could be adapted using specific inclusion criteria. Identification and development of additional clinical indicators was based on chronic diseases included in Australia's National Health Priority Areas.20 In addition, indicators were developed for gastrointestinal disorders, which are associated with high prevalence and morbidity in Australia.21 Australian treatment and clinical guidelines for these chronic conditions were then examined to identify potential mediation-related issues relevant for the development of clinical indicators for preventable medication-related hospitalisations.

Literature review to identify existing clinical indicators of potentially preventable medication-related hospitalisations

A literature review of all published studies on clinical indicators for preventable medication-related hospitalisations was conducted from January 2001 to December 2012, inclusive. The primary search terms used were ‘indicators’, ‘prevent$ OR avoid$’, ‘medication OR drug-related’, ‘hospitalisation OR morbidity’ and ‘adverse drug event’. MEDLINE (via Ovid) and EMBASE were searched, with results limited to articles published in English and conducted in adults. Studies which developed indicators not associated with the outcome of hospitalisation were excluded. Reference lists of relevant identified studies were further searched to identify additional papers. The following information was extracted from each suitable study: the hospitalisation outcome, the process of care leading to the outcome and references (ie, studies which developed the indicator). Clinical indicators were grouped according to broader chronic disease groupings and similar clinical indicators obtained from different studies were recorded as one clinical indicator.

Predefined inclusion criteria were used to determine the applicability and relevance of previously published clinical indicators to the Australian healthcare setting. Indicators that did not meet one or more of these criteria were excluded from the study.

The inclusion criteria were the following:

  1. Strength of supporting evidence must be Grade B or level III or higher, based on the National Health and Medical Research Council (NHMRC) evidence matrix.22

  2. The medicine must be available in Australia and subsidised under the Schedule of Pharmaceutical Benefits (PBS or RPBS).23

  3. The process of care must concur with Australian treatment guidelines.

  4. The process of care can be identified in Australian electronic health records.

Concordance of the indictors with current Australian treatment and clinical guidelines were identified from the Australian Therapeutic Guidelines,24 Australian Medicines Handbook25 and clinical guidelines including cardiovascular disease,26–30 respiratory conditions,31 ,32 diabetes,33 ,34 musculoskeletal conditions35 ,36 and mental health.37 ,38 Where the international clinical indicators differed slightly from Australian guidelines, modifications were developed if appropriate. The guidelines were also searched to determine the level of supporting evidence. The strength of the supporting evidence for each indicator was assessed and categorised into five levels based on current Australian standards, used for guideline development.39 Only those indicators with level III or greater evidence were included. The WHO International Classification of Diseases (ICD) 10–AM classification was used to identify codes for hospitalisation outcomes.40

Development of new clinical indicators of potentially preventable medication-related hospitalisations

Development of new clinical indicators was largely based on those chronic diseases included in Australia's National Health Priority Areas.20 Chronic obstructive pulmonary disease was included with asthma under the broad disease category of respiratory conditions due to its large disease burden and mortality.41 Gastrointestinal disorders which are associated with high prevalence and morbidity in Australia, were also included.21 Clinical indicators for cancer were not developed in this study. Medicine use for cancer is highly specialised and varied depending on the type of cancer, and the development of new medicines for these conditions is a fast evolving area. To develop new indicators treatment and clinical guidelines for each of the conditions were reviewed, with a focus on treatment considerations, medicine class statements and monographs, contraindications, precautions, recommended testing and follow-up. All newly developed indicators were required to meet the inclusion criteria used for previously published clinical indicators, as described above.

Initial face and content validity by a convenience sample of pharmacists

An initial face and content validity of the compiled list of indicators was undertaken with a convenience sample of eight clinical pharmacists. Based on the four elements of preventability developed by Hepler and Strand,42 they were asked the following questions: would you expect most health professionals to

  1. Recognise the problem in the process of care?

  2. Foresee the potential for hospitalisation associated with the process of care?

  3. Know how to change the process of care to reduce the likelihood of hospitalisation?

  4. Be able to change the process of care to reduce the likelihood of hospitalisation?

Responses to each of the four elements of preventability were rated on a three-point Likert scale, where ‘1’ indicates disagreement, ‘2’ uncertain or equivocal and ‘3’ agreement, together with comments to allow for feedback or suggestions regarding specific elements or readability. For each indicator, a majority agreement (5/8 or 62.5%) by the convenience sample across all four elements of preventability was required for inclusion in our final list for validation by an expert panel.

Expert panel assembly, survey and analysis

The final list of indicators for validation were grouped into subject categories (cardiovascular disease, diabetes, renal, mental health, respiratory, gastrointestinal and osteoporosis/fracture indicators) and sent to clinical experts for review. Experts were identified as clinical leaders in their field, that included both medical physicians (general practitioners and specialists) and pharmacists (including certified geriatric pharmacists and clinical pharmacists), across Australia from a range of healthcare settings. A total of 352 clinical experts were identified and contacted to be part of the expert clinical panel for validation of the indicators, between December 2012 and March 2013. They were invited to score the indicators using an online survey (SurveyMonkey http://www.surveymonkey.com) on the four elements of preventability, as described above. Participants were not able to respond to the survey more than once. A brief summary of each indicator was provided; the level of evidence for each indicator together with the reference(s) supporting the level of evidence. A priori criteria of consensus for validation for each of the indicators were defined; an average score of 70% or greater agreement by the expert panellists, across all four elements of preventability for each indicator, were deemed to meet requirements for validation of an indicator. As described above, responses to each of the four elements of preventability were rated on a three-point Likert scale, where ‘1’ indicates disagreement, ‘2’ uncertain or equivocal and ‘3’ agreement, together with comments to allow for feedback on each of the individual indicators.

Results

A total of 48 potential indicators across major disease groupings based on level III evidence or greater were developed (table 1), all of which had majority agreement in the initial face and content validity by a convenience sample of eight clinical pharmacists. Of these, 13 were from previously developed medication-related indicators of potentially preventable hospitalisations, 15 were modified to be applicable to the Australian healthcare setting and 21 were newly developed. These were then sent to the expert clinical panel for full validation and were independently assessed by 78 expert clinicians (22.2% response rate). Of the respondents, 32% were medical physicians and 68% were pharmacists.

Table 1

Australian medication-related potentially preventable hospitalisation clinical indicator set

The expert panel considered 29 of these (60.4%) to be sufficiently valid based on the a priori developed criteria (table 2). The majority of these (72.4%, n=21) were based on level I evidence. A total of 11 cardiovascular indicators were developed, of which 5 (45.5%) were validated by the expert clinical panel; four of the five were based on level I evidence. Of the five mental health indicators developed, only one had level I evidence and none were validated by the expert panel. Six respiratory indicators were developed and five of these were validated, 80% of these were based on level I evidence. A total of seven gastrointestinal indicators, six of which were based on level I evidence were developed and five (71.4%) were validated. Of eight osteoporosis/fracture indicators, half of which were based on level I evidence and five (62.5%) were validated. Only two of the five developed renal indicators were validated, with the level of evidence being level II or less for these. Finally, six diabetes indicators were developed, four of which were based on level I evidence and all were validated by the clinical panel.

Table 2

Validation of Australian medication-related potentially preventable hospitalisation clinical indicator set by expert panel

Discussion

This study provides a set of face and content validated indicators of medication-related potentially preventable hospitalisations, specific for the Australian healthcare setting linking suboptimal processes of care and medication use with subsequent hospitalisation. Of a potential 48 developed indicators, 29 achieved consensus validation by the expert clinical panel and over 70% of these were based on level I evidence. An important feature of these developed indicators is that they are evidence based, systematically combining evidence-based recommendations from clinical guidelines with expert clinical opinion. In addition, these indicators focus on those chronic conditions which are included in Australia's National Health Priority Areas20 or those that are associated with high disease burden in Australia.21 ,41

For each of the six disease clusters for which indicators were developed, the proportion validated by our expert panel ranged from only 20% (1/5) for the mental health indicators to 100% for the diabetes indicators (6/6). Interestingly, the level of evidence available for the metal health indicators around medicine use and processes of care was minimal (four of the five indicators had only level III evidence), by comparison to the diabetes indicators where the majority of evidence was level I.

The health conditions for which these indicators were developed significantly contribute to the burden of illness, social and financial costs in Australia, and prevention of hospitalisations associated with these conditions will provide significant gains in the health of Australia's population.20 Furthermore, given the high prevalence of medication-related hospitalisations in Australia, identification of areas where medication management could be improved, particularly at the primary care level, may also lead to fewer hospitalisations. An estimated 90 000 hospital admissions annually are considered to be potentially preventable mediation-related admissions in those aged 65 years and older.11 Based on the average cost of hospitalisation in Australia in 2010–2011 to be $A5400,43 these unnecessary hospitalisations cost Australia's healthcare system $A480 million annually.

Analysis of the developed indicators in a population-based sample is required to establish operational validity, and this will be the focus of the next phase of this research. With the advent of computerised administrative health databases, these indicators have been developed with the potential to be analysed in such databases at the population level. Importantly, the characteristics of those patients’ most vulnerable to mediation-related hospitalisations will also facilitate the identification of risk-factors associated with suboptimal medication management. Implementation of these indicators within routine monitoring of the Australian healthcare system will serve to highlight those conditions where hospitalisations could potentially be avoided through improved medication management, identify areas of current practice that may be suboptimal or evidence-practice gaps and facilitate the development of specific interventions to improve healthcare and subsequent patient outcomes.

The standard RAND appropriateness method employs two rounds; in the first round experts rate indicators independently and in the second round, experts meet face to face to discuss the indicators and rate the indicators again, based on the face-to-face discussion.17 Our study used a modified RAND appropriateness method, with one round of independent expert panel review, and subsequent inclusion of indicators which met a priori defined criteria but no face-to-face meeting of experts. This is a potential limitation of our study, because the face-to-face meeting provides an opportunity to discuss indicators with low levels of agreement between experts, and can identify whether this is due to true clinical disagreement or simply an issue with the wording of the indicator.44 It may be that for some indicators true consensus was not achieved if those who disagreed were strongly opposed to the indicator, and this type of issue may have been identified at a face-to-face meeting. Despite these limitations, the online survey technique used in our study eliminates any potential bias from dominant individuals who may be associated with face-to-face panel settings.44 This allows for expert clinical panel members to express their opinions in an anonymous manner but also gives them time to consider each of the four elements of preventability together with the supporting evidence base of each developed indicator. In addition, our method for developing the indicators systematically combined the available evidence base with the opinion of clinical experts to develop indicators that are both face and content valid.19 The modified RAND method used in our study has been used in indicator development studies previously.18 ,45 A recent Australian study used this method to validate 657 indicators of healthcare appropriateness.16

Our study achieved a 22% response rate, which is lower than other Australian studies involving medical practitioners, which typically achieve a response rate of around 30%.46–48 While this may limit the generalisability of our findings, our results are strengthened by having medical specialists, general practitioners and pharmacists on the expert review panel.44 In addition, 78 expert clinicians reviewed the clinical indicators for our study; by comparison, previous studies which developed clinical indicators for preventable medication-related hospitalisation used fewer than 20 expert reviewers.

In conclusion, this study has developed a set of face and content validated indicators of medication-related potentially preventable hospitalisations specific for the Australian healthcare setting, linking medication use with suboptimal processes of care resulting in adverse outcomes of hospitalisations. As a measure of health system performance these indicators could identify areas of sub-optimal medication management, particularly at the primary care level, based on routinely collected health administrative health data but with the strong focus on patient outcomes and quality of care rather than processes or quantity.

Acknowledgments

The authors would like to thank the expert clinical panel and the project advisory group for their time and expertise.

References

View Abstract

Footnotes

  • Contributors GEC conceived and designed the study, and was involved in data acquisition, analysis and interpretation, drafted the article and approved the final approval version to be published. LMKE conceived and designed the study, and was involved in data acquisition, analysis and interpretation, critically revised the manuscript for important intellectual content and approved the final approval version to be published. TYW was involved in data acquisition, analysis and interpretation, critically revised the manuscript for important intellectual content and approved the final approval version to be published.

  • Funding This work was funded by The BUPA Health Foundation

  • Competing interests None.

  • Ethics approval Ethics approval for this study was obtained from the University of South Australia's Human Research Ethics Committee.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement No additional data are available.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.