Article Text

Download PDFPDF

Adaptation of potentially preventable medication-related hospitalisation indicators for Indigenous populations in Australia using a modified Delphi technique
  1. Jean Marie Spinks1,
  2. Lisa M Kalisch Ellett2,
  3. Geoffrey Spurling3,
  4. Theo Theodoros4,
  5. Daniel Williamson5,
  6. Amanda J Wheeler6,7
  1. 1Centre for Applied Health Economics, Menzies Institute for Health Queensland, Griffith University, Brisbane, Queensland, Australia
  2. 2Sansom Institute for Health Resarch, School of Pharmacy and Medical Sciences, Quality Use of Medicines and Pharmacy Research Centre, University of South Australia, Adelaide, South Australia, Australia
  3. 3Primary Care Clinical Unit, University of Queensland, Brisbane, Queensland, Australia
  4. 4Addiction and Mental Health Services, Metro South, Brisbane, Queensland, Australia
  5. 5Aboriginal and Torres Strait Islander Health Branch, Queensland Health, Brisbane, Queensland, Australia
  6. 6Menzies Institute for Health Queensland, Griffith University, Brisbane, Queensland, Australia
  7. 7Faculty of Health and Medical Sciences, University of Auckland, Auckland, New Zealand
  1. Correspondence to Dr Jean Marie Spinks; j.spinks{at}


Objectives One of the outcomes of a medication review service is to identify and manage medication-related problems (MRPs). The most serious MRPs may result in hospitalisation, which could be preventable if appropriate processes of care were adopted. The aim of this study was to update and adapt a previously published set of clinical indicators for use in assessing the effectiveness of a medication review service tailored to meet the needs of Indigenous people, who experience some of the worst health outcomes of all Australians.i

Design A modified Delphi technique was used to: (i) identify additional indicators for consideration, (ii) assess whether the original indicators were relevant in the context of Indigenous health and (iii) reach consensus on a final set of indicators. Three rounds of rating were used via an anonymous online survey, with 70% agreement required for indicator inclusion.

Setting The indicators were designed for use in Indigenous primary care in Australia.

Participants Thirteen panellists participated including medical specialists, general practice doctors, pharmacists and epidemiologists experienced in working with Indigenous patients.

Results Panellists rated 101 indicators (45 from the original set and 57 newly identified). Of these, 41 were accepted unchanged, seven were rejected and the remainder were either modified before acceptance or merged with other indicators. A final set of 81 indicators was agreed.

Conclusions This study provides a set of clinical indicators to be used as a primary outcome measure for medication review services for Indigenous people in Australia and as a prompt for pharmacists and doctors conducting medication reviews.

Trial registration number The trial registration for the Indigenous Medication Review Service feasibility study is ACTRN12618000188235.

  • Indigenous health
  • potentially preventable medication-related hospitalisations
  • medication review
  • clinical indicators

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:

Statistics from

Strengths and limitations of this study

  • This is the first set of clinical indicators developed to identify potentially preventable medication-related hospitalisations in Indigenous Australians;

  • The set of clinical indicators developed can be used to measure serious medication-related problems in Indigenous Australians and be used as a resource by health professionals conducting medication review services;

  • The set of clinical indicators forms the primary outcome measure of an Indigenous Medication Review Service feasibility study;

  • The participant sample size for this study was limited, possibly due to workload constraints of clinicians working in Indigenous health in Australia;

  • This study makes an important contribution to the literature by developing a quantitative measure that can be used to improve medication outcomes for Indigenous Australians.


Aboriginal and Torres Strait Islander people in Australia experience higher rates of disease burden compared with other Australians, particularly for chronic disease.1 As pharmacotherapy is one of the principal tools used to manage chronic conditions, this creates a challenge for health services providers to coordinate medication services within a culturally respectful and comprehensive primary healthcare system,2 and minimise medication-related harm. Medication review is a structured evaluation of an individual’s medications to optimise medication use and health outcomes.3 An important component of a medication review involves a pharmacist identifying medication-related problems (MRPs) and, in consultation with the prescriber, suggesting management options.

Medication reviews have been shown to significantly increase the identification and resolution of MRPs, although there is limited evidence to show that they reduce hospital admissions,4 possibly because there are many types of MRPs with varying degrees of severity and preventability.5–7 Although the most serious MRPs can lead to hospitalisation8 some are unpredictable and therefore not considered preventable, for example, atypical adverse drug reactions. However other MRPs are potentially preventable, for example, where clinical care preceding the hospitalisation event is not in accordance with accepted clinical guidelines.

Potentially preventable medication-related hospitalisations (PPMRHs) are the result of a proportion of serious MRPs.8 Clinical indicators have been developed and used in a number of countries to measure PPMRHs which link suboptimal care involving medication use with subsequent hospitalisation.9–11 However, differences have been found, for example, between the UK and USA in terms of the inclusion of particular indicators, presumably guided by the prevalence of different health conditions in different population groups and health system differences.12 Thus, although a set of PPMRH indicators have been developed for use in the Australian population,13 14 it cannot be assumed that this is a robust measure for specific subsets of the Australian population with distinct healthcare needs, like Indigenous people.

There are a number of advantages of using PPMRHs as the primary outcome in a medication review intervention as they: (i) are prespecified, removing potential classification bias from the primary outcome; (ii) can be costed, for easy inclusion in an economic evaluation and (iii) offer a meaningful target for pharmacists and other clinicians undertaking medication reviews in clinical settings.

The Indigenous Medication Review Service (IMeRSe) feasibility study is being undertaken across nine Australian sites including remote, regional and urban locations, with the aim of developing and testing the feasibility of a culturally appropriate, strengths-based, medication review service.15 The IMeRSe intervention is delivered by local community pharmacists (on a fee-for-service basis) integrated with local Aboriginal health services (AHSs). Previous research has shown that Indigenous people encounter barriers to accessing medication review services,16 17 thus the aim of IMeRSe is to overcome these barriers and meet the health needs of the population.15

Here we report on the modification of the existing set of 45 PPMRH indicators which were originally developed for use in the general Australian population and validated using a large veterans cohort.13 14 However, the indicators needed to be revised; to ensure: (i) utility, as an appropriate primary outcome measure in the IMeRSe feasibility study; and (ii) currency and applicability, in light of changes to clinical guidelines and best practice. Inclusion criteria for the IMeRSe study specifies participants to be over 18 years and identify as being Aboriginal or Torres Strait Islander,15 meaning participants will likely be younger and experience different health conditions than the general Australian population.1 Thus, the list of previously identified PPMRHs needs to be revised to reflect the health problems faced by this population.

The objective of this study was to develop a meaningful and clinically relevant outcome measure for use in the IMeRSe pilot study,15 which is trialling the feasibility of a culturally appropriate, strengths-based, medication review service.


In general terms, the selection of clinical indicators to measure processes and outcomes of primary care should meet the criteria of validity, reproducibility, acceptability, feasibility, reliability, sensitivity and predictive validity.18 Consensus methods are one way of developing, or refining, a set of clinical indicators to meet these criteria. The Delphi technique has been widely used in health research to achieve consensus on a particular topic where expert opinion is the main source of evidence,19 20 including the development of healthcare quality indicators.21 Other consensus methods, such as the nominal group technique,22 or the RAND appropriateness method,23–26 may also be appropriate; however, the Delphi technique has the advantage of involving a sufficiently representative group of experts while being less resource intensive than alternative methods.

​Selection of Delphi panellists

The IMeRSe feasibility study Expert Stakeholder Panel (which included Indigenous advisors) identified potential panellists for the Clinical Validation Group (CVG).15 The function of the Expert Stakeholder Panel is to ensure that all aspects of the study are culturally appropriate and respect Indigenous practices, protocols and community engagement. Potential CVG panellists were identified by the Expert Stakeholder Panel as either having current clinical experience as a doctor or a pharmacist in an Indigenous health setting, or medication safety expertise from a public health perspective. Ideally, Indigenous clinicians and researchers would constitute the whole of the CVG, however while the CVG did have Indigenous representation and attempts were made to include more, we were not able to convene an entirely Indigenous CVG. Potential panellists were approached via email, provided with participant information forms and instructions, and contact details to obtain further information, as required. Panellists were made aware that informed consent was implied by acceptance of the invitation via return email. Of the 40 eligible panellists approached to participate, 13 agreed. Panellists were offered a small honorarium to compensate them for their time.

​Rating rounds

Prior to the start of the first rating round, consented panellists were interviewed individually by a member the research team (JS) to ensure they had a chance to clarify the Delphi process. During the interview, panellists were asked to identify any additional indicators that they believed should be considered in addition to the original 45 indicators14 or email them after the interview, if preferred. Panellists were asked to only identify indicators that met the criteria of preventable drug-related morbidity, as defined by Hepler & Strand27 who specify three necessary elements:

  1. The drug-related problem must be recognisable, and the likelihood of an undesirable clinical outcome must be foreseeable;

  2. The causes of that outcome must be identifiable;

  3. The causes must be controllable.

Panellists were also asked to consider indicators that, from their own clinical experience, represented the greatest burden to population health for Indigenous Australians. Additional indicators considered to be relevant were added to the original list of 45 indicators to form a Master List. Three rounds of rating and consensus were then undertaken using this list as a starting point.

The first two rating rounds were sent to all panellists via email link in an online format hosted in LimeSurvey.28 Panellists were asked to carefully consider each indicator presented and then choose from four options: (i) accept indicator unchanged, (ii) reject indicator, (iii) specify alternative or (iv) not sure. Panellists were asked to provide comments or a rationale for rejecting an indicator or providing an alternative. An example of the online presentation of a clinical indicator to panellists is shown in figure 1.

Figure 1

An example of the online presentation of a clinical indicator to panellists. INR, international normalised ratio.

The indicator was accepted unchanged if at least 70% of panellists chose the option ‘Accept indicator unchanged’ or rejected if at least 70% of panellists chose the option ‘Reject indicator’ in accordance with previous modified Delphi methods.29 The indicators which were accepted unchanged or rejected were removed and did not appear in subsequent rating rounds. All other indicators (where an alternative was proposed) were collated alongside the panellists’ comments or rationale, by the researchers. The researchers considered the comments, consulted any relevant clinical literature and offered alternative wording for the disputed indicator. Panellists’ comments were (anonymously) reported verbatim in the subsequent rating round, alongside the researchers proposed new wording of the indicator and links to any relevant clinical literature or guidelines. Researchers set a deadline of 2 weeks for responses after the online survey was opened. Panellists could login to the survey again if they had not completed it, and previous responses could be altered at any time prior to survey submission. Reminder emails were sent 1 week before the deadline and requests for additional time was granted for participants to complete the rating round, if required. Every effort was made by the research team to enable all 13 participants to complete the first two rating rounds.

The third rating round involved a face-to-face meeting of an invited subgroup (n=3) of the larger consensus group; a representative from each main speciality area (specialist doctor, general practice doctor, clinical pharmacist) provided expert commentary regarding any remaining discrepancies. Consensus in this final round was achieved following open group discussion which was moderated by the researchers (JS/AJW).

​Patient and public involvement

Patient and public involvement has been achieved in the IMeRSe feasibility study, and will be ongoing over the study lifetime, through extensive collaboration with the relevant representatives of both Partner organisations. As described above (Selection of Delphi panellists), working with key Indigenous groups, both locally and as members of the Expert Panel, will be integral to the ongoing engagement process (eg, via the inclusion of community juries, councils and boards). This process will be informed by the local requirements at each site throughout this feasibility study. Acceptability outcomes for consumer participants will be assessed as described previously.15 Dissemination to Indigenous participants and communities will be a priority, with processes guided by the Expert Panel and informed by key stakeholders at a local site level.


​CVG panellists

A total of 13 panellists, five females and eight males, from five clinical areas participated between May 2018 and November 2018. Panellists had a mean of 17 years experience in their clinical areas and 11 years experience working with Indigenous people in their current role (table 1); . Panellists were drawn from six of the nine states and territories across Australia and from urban, rural and remote locations (detailed information is withheld to maintain the anonymity of panellists).

Table 1

Clinical validation group panel

​Clinical indicators

In addition to the original 45 indicators,11 panellists identified a further 56 new indicators. Hence, the Master List of indicators at the start of Round 1 rating comprised 101 indicators. During each of the rating rounds, panellists made suggestions to split and merge indicators, meaning the number of indicators for consideration could increase or decrease between rounds. The number of clinical indicators from the Master List accepted or rejected in each rating round, grouped by clinical presentation, are summarised in table 2.

Table 2

Number of clinical indicators, grouped by clinical presentation and round

At the end of Round 2 rating, 65 indicators (80% of the final total) were agreed on by the panellists. The three-person subgroup of the CVG invited to undertake Round 3 rating formed consensus on the remaining 23 indicators during a 2 hour face-to-face meeting (one panellist phoned-in), moderated by the research team (JS/AJW). One clinical indicator was rejected during this round, with the remaining 22 indicators either accepted or merged with other indicators.

The final list of accepted indicators is presented in table 3. Thirty-four indicators from the original list of 45 were accepted by panellists, although 21 of these were updated in some way to reflect: (i) changes in current guidelines or new medicines, (ii) having been combined with other similar indicators for simplification, (iii) having been split into additional indicators for clarity. Forty-seven new indicators were added, giving a final total of 81 indicators.

Table 3

Final list of potentially preventable medication-related hospitalisations for Indigenous Australians#


The development of this updated clinical indicator list is an important step in addressing MRPs for Indigenous Australians. This study builds on earlier work which identified a set of clinical indicators for use in a general Australian population.13 14 The new list includes 81 indicators, sourced from 34 of the original 45 indicators and 47 newly identified indicators. In comparison to the general Australian population list, the new list contains more neurological indicators (expanded from 7 to 14), vaccine preventable diseases (expanded from 0 to 12) and ‘other’ indicators (expanded from 0 to 9), which better reflects the health burden of the Indigenous population. For example, trachoma and rheumatic heart disease are health issues seen in the Indigenous population, but rarely in the general Australian population.

Panellists included specialist and general practice doctors, pharmacists, epidemiologists and researchers, the majority of whom had extensive experience in providing healthcare for Indigenous populations. The purpose of conducting this research was two-fold: first to provide a prespecified list of PPMRHs to define the primary outcome measure for the IMeRSe feasibility study15; and as a resource for pharmacists conducting medication reviews for Indigenous Australians to assist in identifying suboptimal processes of primary care related to medication use, defined for the IMeRSe feasibility study as serious MRPs.15

AHSs offer Indigenous Australians access to holistic and person-centred primary care. The inclusion of pharmacists undertaking medication review services is important as much of the health burden experienced by Indigenous Australians results from chronic conditions such as renal and/or cardiovascular disease, type-II diabetes and mental illness, which in turn increases the requirement for ongoing medication regimens.1 30 There are reports that the levels of MRPs among Indigenous populations are of concern,31 32 although there is scant evidence of the size or extent of the problem. Further, Indigenous populations access the existing government funded medication review services, The MedsCheck and Diabetes MedsCheck Programme provides for in-pharmacy reviews of consumers who are taking multiple medications and/or have newly diagnosed or poorly controlled type 2 diabetes. These reviews are aimed at enhancing the quality use of medicines and reducing the number of adverse drug events experienced by consumers. A Home Medicines Review is designed to enhance the quality use of medicines and reduce the number of adverse medicine events by assisting consumers to better manage and understand their medicines through a medication review conducted by an accredited pharmacist in the patient's home ( at a lower rate than non-Indigenous Australian for reasons including the lack of culturally responsive services, not having established and trusting relationships with pharmacists and because pharmacists are not usually integrated into AHSs.16 17

The clinical indicator list developed in this study will be tested for predictive validity in two ways through the IMeRSe feasibility study: (i) as a primary outcome measure and as such, will be used to classify a set of serious MRPs which can be analysed against a list of all MRPs (regardless of severity); and (ii) to estimate the rate of PPMRHs in Indigenous populations using a linked administrative data set comprised of 5 years of hospital admissions from the state of Queensland, Australia. This data set will be combined with pharmaceutical and medical services usage for the same cohort of hospitalised individuals (collected by the national government). Thus, the background rate of PPMRHs can be identified, for arguably the most representative state in Australia in terms of Indigenous Australians, as urban, rural and remote populations are all included. However, it is anticipated that it will not be possible to measure some of the indicators using these existing administrative databases as insufficient clinical information (such as cardiovascular disease risk) will be available. It is possible that this problem may decline over time as individual health records become fully digitalised and shared in Australia.

The processes contributing to suboptimal clinical care specified in the final indicator list (table 3) are termed serious MRPs; these may, or may not, result in a hospitalisation. Only when a hospitalisation does occur is a PPMRH realised. Thus, we are interested not only in the rate of PPMRHs in the Indigenous population, but also the rate of MRPs and the translation rate of MRPs to PPMRHs. The reduction in MRPs of all severity, including serious MRPs, is a key outcome of IMeRSe feasibility study.

A modified Delphi technique was used in this study to reach consensus between experts. The Delphi technique allows for anonymity in responses, which permits all panellists an equal chance to have their opinion considered. A majority consensus was reached for 65 (80%) of the total number of indicators at the end of Round 2 rating. Of the remaining indicators (n=23), the majority required only a short discussion and/or brief changes to wording to reach consensus Round 3 rating. The researchers considered that this meeting expediated consensus on the remaining indicators and was a strength of the study. It must be noted that use of the term ‘consensus’ here, especially in the early phases of the Delphi process, is in fact ‘convergence’ of expert opinion. However, consensus has been assumed because : (i) panellists were made aware that they were involved in a decision-making process at the start, (ii) justification for non-acceptance was fed-back to the group between rounds and (iii) face-to-face discussions were held to reach agreement in Round 3.

Unlike the RAND appropriateness method, the modified Delphi rating process did not incorporate a formal mechanism for considering the strength of evidence of the proposed indicators. This aspect could not be incorporated into the present study, due to the lack of relevant research specifically involving Indigenous Australians, and hence the lack of evidence for this specific patient population. However, the existing indicator list, which was adapted for the present study was developed using the RAND appropriateness measure,13 and considered the strength of evidence underpinning each indicator during the indicator development process. Thirty-four of the indicators accepted in the present study were based on existing indicators, so nearly half of the indicators were developed by explicitly considering the strength of evidence for the particular indicator. During the moderated online and face-to-face discussions, the researchers observed that clinicians incorporated current clinical guidelines into their decision-making processes, although this was not undertaken in a formal way. This could be viewed as a potential limitation of the study. Another possible limitation was the relatively small number of panellists who agreed to participate, which could be due to workload pressures for clinicians working in Indigenous health in Australia. Finally, the authors note that the final list of clinical indicators developed here are not necessarily independent of each other, nor are they of equal weighting of clinical seriousness. Thus, this issue will need to be accounted for in the data analysis of the PPMRHs for the IMeRSe study.

By classifying a list of serious MRPs, the importance of other MRPs may be discounted. The lack of adherence to medication regimens among Indigenous populations is of particular concern, especially given the high rates of chronic disease such as diabetes, cardiovascular disease, severe mental illness and renal disease that require regular medication. Barriers that limit adherence including poor health literacy, lack of access to medications (cost and physical access) and medication sharing with relatives and friends can all negatively impact on health through uncontrolled illness.31 In the short-term, health decrements due to low medication adherence may not result in hospitalisation, it may nonetheless contribute to life-threatening outcomes in the medium-to-longer term. It must be stressed that the final clinical indicator list developed here should only be used by pharmacists and other health professionals undertaking medication review services as a resource to optimise medication management. It does not provide a definitive list of the most serious problems, nor does it replace clinical judgement.


The final list of clinical indicators developed in this study represents an initial, but important, step in quantifying serious MRPs and PPMRHs in Indigenous Australian populations. Such a list is not static and should be regularly updated in light of changes to clinical guidelines and medicines formularies. The health of Indigenous Australians may be enhanced by using this list as a resource during the process of medication review to identify suboptimal processes of care and then institute corrective processes to prevent a potential hospitalisation.


The Indigenous Medication Review Service (IMeRSe) feasibility study has been developed in partnership with The Pharmacy Guild of Australia, the National Aboriginal Community Controlled Health Organisation (NACCHO) and Griffith University. The authors would like to acknowledge the time and experience of the panellists who participated in this study and Dr Helen Stapleton for providing feedback on the manuscript. The authors would also like to thank the Expert Panel for their ongoing contribution to the IMeRSe study. This activity received grant funding from the Australian Government. The researchers were independent from the funder. This article contains the opinions of the authors and does not in any way reflect the views of the Department of Health or the Australian Government. The financial assistance provided must not be taken as endorsement of the contents of this article. There was no direct public or patient involvement in the research reported in this manuscript, however, patient and public involvement has been achieved in the IMeRSe feasibility study.



  • Correction notice This article has been corrected since it was published. The article title and Abstract have been updated.

  • Contributors JS was responsible for study concept, design, data collection and wrote the first draft of the manuscript. AJW designed the research protocol, is the principle investigator and was involved in the design and data collection. LK, GS, TT and DW contributed to methodology and data collection. All authors were involved in the revision of this paper and made a contribution to the intellectual content and approved the final version of the paper.

  • Funding This activity received grant funding from the Australian Government.

  • Competing interests None declared.

  • Patient consent for publication Not required.

  • Ethics approval Ethics approval was granted from Griffith University Human Research Ethics Committee (GU ref No:2018/126) for this study.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data availability statement The data set generated and analysed for this study is not publicly available as consent from participants was not sought to share the data more widely than for the purpose of this study.

  • Please note that the use of the term ‘Indigenous’ in this manuscript includes all Aboriginal and Torres Strait Islander people and acknowledges their rich traditions and heterogenous cultures.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.