Article Text

Download PDFPDF

Validation of prescribing appropriateness criteria for older Australians using the RAND/UCLA appropriateness method
  1. Benjamin Joseph Basger,
  2. Timothy Frank Chen,
  3. Rebekah Jane Moles
  1. Faculty of Pharmacy, The University of Sydney, Sydney, Australia
  1. Correspondence to Benjamin Joseph Basger; ben.basger{at}


Objective To further develop and validate previously published national prescribing appropriateness criteria to assist in identifying drug-related problems (DRPs) for commonly occurring medications and medical conditions in older (≥65 years old) Australians.

Design RAND/UCLA appropriateness method.

Participants A panel of medication management experts were identified consisting of geriatricians/pharmacologists, clinical pharmacists and disease management advisors to organisations that produce Australian evidence-based therapeutic publications. This resulted in a round-one panel of 15 members, and a round-two panel of 12 members.

Main outcome measure Agreement on all criteria.

Results Forty-eight prescribing criteria were rated. In the first rating round via email, there was disagreement regarding 17 of the criteria according to median panel ratings. During a face-to-face second round meeting, discussion resulted in retention of 25 criteria after amendments, agreement for 14 criteria with no changes required and deletion of 9 criteria. Two new criteria were added, resulting in a final validated list of 41 prescribing appropriateness criteria. Agreement after round two was reached for all 41 criteria, measured by median panel ratings and the amount of dispersion of panel ratings, based on the interpercentile range.

Conclusions A set of 41 Australian prescribing appropriateness criteria were validated by an expert panel. Use of these criteria, together with clinical judgement and other medication review processes such as patient interview, is intended to assist in improving patient care by efficiently detecting potential DRPs related to commonly occurring medicines and medical conditions in older Australians. These criteria may also contribute to the medication management education of healthcare professionals.

  • drug-related problems
  • prescribing criteria
  • older patients
  • inappropriate drug use

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: and

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Article summary

Article focus

  • Drug-related problems (DRPs) are common in older people. They may result in drug treatment goals not being achieved and/or the occurrence of adverse drug events.

  • The aim of this study was to further develop and validate a previously published list of prescribing appropriateness criteria for use in older people which may be used to improve the quality of the Australian medication review process, and for quality assessment and education in medicine use.

Key messages

  • The use of medication assessment criteria is one method to assist in identifying DRPs. Criteria developed elsewhere may have little or no applicability to the Australian healthcare environment.

  • Validation of proposed Australian prescribing appropriateness criteria for older people was accomplished using a two-round-modified Delphi method, resulting in agreement for all criteria as measured by median panel ratings, and the amount of dispersion of panel ratings, based on the interpercentile range.

  • Use of these criteria, together with other Australian medication review processes, may assist in improving patient care in a variety of settings by efficiently identifying DRPs to common medical conditions and commonly used medicines. They may also contribute to the medication management knowledge of healthcare professionals through education programmes and by use in daily practice, and for the evaluation of the quality of pharmaceutical care in older people.

Strengths and limitations of this study

  • A validated consensus method was used involving an expert medication management panel of varied specialisation. Criteria were based on established evidence–practice gaps and degree of disease burden imposed on the healthcare system, and were written with the aim of conciseness and clarity.

  • Further developmental work is required to assess the usefulness of these criteria, which only included commonly occurring medicines and medical conditions.


Drug-related problems (DRPs) in older people (≥65 years old) are common.1–4 They may result in drug treatment goals not being achieved and/or disproportionately high numbers of serious adverse medication events due to polypharmacy.5–7 DRPs can occur for many reasons such as undertreatment, inadequate monitoring of medicines, poor medicine or dose selection, duplication of medicines or factors to do with the way the patient uses the medicine.2 ,3 ,8–12 Methods to identify and reduce DRPs include healthcare professional-directed educational interventions,13 comprehensive geriatric assessment,14 discontinuation of multiple medications,15 ,16 electronic health record clinical decision support targeted towards certain diseases or drugs,17 ,18 and the use of medication assessment criteria, which usually consist of explicit (ie, criterion-based rather than implicit or judgement-based) lists of prescribing recommendations for various drugs and/or disease states.13 ,19–22

In Australia, identification and resolution of DRPs are intended to be considered when patients are interviewed by an accredited pharmacist as part of the Home Medicines Review programme.23 This programme aims to provide the sophistication lacking in the application of explicit measures alone, as it takes into account other issues such as the patients history and personal preferences, and is targeted towards patients who may be (among other reasons) currently taking ≥5 regular medicines, attending a number of different doctors, or have recently been discharged from hospital.24

In 2008, we proposed a list of 48 prescribing appropriateness criteria (45 explicit and three implicit) aimed at improving detection of DRPs as part of the Australian medication review process.25 These criteria were intended to be applied alongside the patient interview in order to prompt appropriate history taking, particularly with respect to commonly occurring medical conditions and medicines. Similar criteria derived outside Australia have been found to have application in a variety of settings and for a variety of uses, such as in the training of healthcare professionals and in the evaluation of the quality of healthcare.19 ,26–29 Our criteria were based on the most frequent medicines prescribed to Australians, and the most frequent medical conditions for which older Australians (≥65 years old) consult medical practitioners. Australian medication and disease state resources and guidelines were used to provide content validity.25 However, unlike our criteria, other prescribing criteria or tools have combined evidence with expert opinion to provide face validity.

The aim of this study was to further develop our list of criteria, supplementing it with recommendations for comorbidity and the oldest old where possible, and adding new criteria where necessary through expert consensus. In older patients, the importance of traditional outcomes, such as discrete clinical events or mortality, may be secondary to maintaining physical or cognitive function or relief of symptoms.30 Because of this, optimal care requires clinical decision support tools that consider issues such as patient preferences, frailty, cost and comordidities.31 Additionally, few criteria target the oldest old32 (generally regarded as people older than 85 years), where evidence may be poor, and preventive interventions may be encouraged in patients who have already exceeded an average lifespan.33 ,34

To further develop and validate our criteria list, we identified a panel of medication management experts, and chose the RAND/UCLA appropriateness method, which has been described as the best method for systematically combining recommendations from clinical guidelines, with the opinion of healthcare providers.35



Ethics approval was obtained from the Human Research Ethics Committee of the University of Sydney.

Criteria development

In 2008, we identified the 50 highest-volume Australian Pharmaceutical Benefits Scheme (PBS) medicines prescribed, and the 40 most common reasons for older Australians to seek or receive healthcare. Healthcare information was obtained using the BEACH (Bettering The Evaluation and Care of Health) programme, which continuously collects information about the clinical activities in general practice in Australia.36 We then used Australian medication information sources to identify both optimal and inappropriate medication management of these common conditions.25 In Australia, medication availability and use are largely determined by the PBS.37 In October 2011, commonly used medications and medical conditions were checked and updated using the BEACH programme to ensure that criteria content was current. Changes in evidence, product information, Australian consensus documents, evidence-based publication recommendations or clinical practice guidelines relating to our criteria were noted for evaluation by an expert medication management panel. The criteria were designed to provide guidance on the process of care wherever it occurred—community, hospital, residential home, care home or nursing home. Major considerations in their development were likely accessibility of data from the patient, their medical notes and/or their healthcare professional(s), conciseness and clarity of wording, and provision of a practical number of criteria. Most were explicit to enable consistent application, with additional notes provided for interpretation where necessary. They were written as a statement of the kind of medication management that should or should not occur, to simplify comprehension and facilitate uptake.25

Validation of criteria: participants

We recruited a multidisciplinary group of medication management experts to review, update and rate the criteria, consisting of geriatrician/pharmacologists, clinical pharmacists and disease management advisors to organisations that produce Australian evidence-based therapeutic publications. This resulted in a round-one panel of 15 members. The geriatricians consisted of two professors of geriatric medicine; an associate professor of clinical pharmacology and aged care; a research fellow in geriatric medicine and a hospital staff geriatrician. Clinical pharmacists consisted of a residential medication management review pharmacist; a home medicines review pharmacist; four hospital-based pharmacists (two team leaders, one director and one education and training pharmacist) and a professor of aged care (pharmacy). Disease management advisors to Australian evidence-based therapeutic organisations consisted of Therapeutic Guidelines,38 Australian Medicines Handbook39 and the New South Wales Therapeutic Advisory Group.40

Choice of the RAND/UCLA appropriateness method

We chose the RAND/UCLA appropriateness method, a two-round modified Delphi method41 to select the most appropriate criteria. Unlike the Delphi method, which generally involves multiple questionnaire-driven rounds to obtain convergence of opinion, the RAND method involves an initial individual rating round, and a second face-to-face round. This method has been shown to produce results that have face, construct and predictive validity.42 ,43 Systematically combining available evidence with expert opinion can create quality criteria where best evidence may be lacking.44

While most lists of prescribing criteria are based on expert consensus, this has often been achieved through mail surveys rather than face-to-face meetings.32 ,35 ,45 Although face-to-face meetings restrict panel size, they allow discussion to resolve misinterpretations, introduce new evidence and improve clarity of criteria between rating rounds. We ensured our panel comprised different specialities, as less disagreement has been found among same-specialty panels.46 We addressed concern regarding potential intimidation due to dominant panel personalities by choosing a moderator experienced in the development of these criteria and in facilitating small group discussion. This may also have assisted with conflict-of-interest issues. We used both the median panel rating and the amount of dispersion of panel ratings to identify agreement or disagreement. While it has been acknowledged that discrepancies between these two methods may occur,41 our aim was to achieve agreement for all accepted criteria for both methods after second round discussion.

RAND/UCLA appropriateness method round one

In October 2011, candidate panel members were emailed an explanation of the project and an invitation to participate. After acceptance, they were emailed a rating sheet consisting of 48 criteria, and asked to rate each on a nine-point scale. Ratings of 1–3 were classified as inappropriate, with a rating of one indicating the greatest degree of inappropriateness. Ratings of 7–9 were classified as appropriate, with a rating of nine indicating the greatest degree of appropriateness. Ratings of 4–6 were classified as neither appropriate nor inappropriate. Appropriate was defined as ‘the expected health benefit exceeds the expected negative consequences by a sufficiently wide margin that criteria are worth following, exclusive of cost’. They also received a description of the way in which the criteria had been derived, and a comparison with other prescribing criteria.25 ,32 Panel members were requested to amend the wording or delete, update or identify missing criteria as required. Upon return of the rating sheets, results were tabulated. Agreement was based on four or less panellists rating outside the three-point region containing the median (1–3; 4–6; 7–9), and disagreement was based on five or more panellists rating in each extreme (1–3 and 7–9), as per the RAND/UCLA protocol for a 15-member panel.41

Rand/UCLA appropriateness method round two

In November 2011, a face-to-face meeting of the expert panel, chaired by a panel moderator experienced in facilitating group discussions and criteria development, met to discuss the results of round one and re-rate each of the criteria and any potential additional criteria. One pharmacist, one staff geriatrician and a disease management advisor for a therapeutics publication could not attend, resulting in a 12-member panel. For this meeting, each panel member was provided with a copy of the results from round one. This consisted of the frequency distribution of ratings of all panellists across the nine-point scale, the overall panel median rating for each of the criteria and, for each panellist, an annotation of how they had rated each of the criteria. Scores from other panel members were not revealed. Depending on panellists votes, panel agreement or disagreement was also stated for each of the round one criteria. Additionally, the 30th and 70th percentiles adjusted for symmetry were computed for each of the criteria, as it has been found that when ratings were symmetric with respect to the middle (five on the 1–9 scale), the interpercentile range (IPR) required to label an indication as disagreement was smaller than when they were asymmetric with respect to the middle (values far from five on the 1–9 scale). Agreement after round two occurred when the IPR adjusted for symmetry (IPRAS) was greater than the IPR.41

We used the median method to present data at the face-to-face meeting, as it provided a clear visual interpretation of the ratings for each criterion. By the end of the meeting, our aim was to ensure that there was agreement between the median method and the interpercentile method for all accepted criteria.

Discussion at round two occurred on the level of agreement for each of the criteria. In addition, discussion was facilitated on the wording of each of the criteria to improve clarity and decide whether agreement would be reached. The definitions of agreement and disagreement were adjusted for the smaller second round 12 member panel.41 Agreement was reached when three or less panel members voted outside the three-point region containing the median, or when the IPRAS was greater than the IPR. Disagreement was determined when four or more panellists rated in each extreme (1–3 and 7–9). Each of the criteria were then discussed irrespective of whether there was agreement or disagreement, with panellists having the opportunity of changing their ratings if, for example, misinterpretation had occurred because of the way in which the criteria had been written, or if new evidence had become available, or if criteria had been interpreted in the light of a panellists own clinical experience. Each panel member consented to audio recording of the discussion. Values for the median, IPR and IPRAS41 were computed using SPSS V.20 (SPSS, Chicago,  Illinois, USA).


After round one, there was agreement for the appropriateness of 31 of the 48 criteria, and disagreement for 17 criteria. Of the 31 criteria for which there was agreement, discussion at round two resulted in 17 criteria being amended and retained, 2 criteria being deleted and 12 criteria accepted with no change. Of the 17 criteria for which there was disagreement, discussion at round two resulted in eight criteria being amended and retained, seven criteria being deleted and two criteria accepted with no change. Two new criteria were added, resulting in a total of 41 validated criteria.

An example of how the RAND/UCLA method was applied to each of our criteria is described in table 1 for criterion one. The larger the IPRAS, the less asymmetric are the ratings. For example, 13 of 15 panellists at round one rated indicator 14 with a score of 8 or 9, for which the IPRAS was 8.35.

Table 1

An example of the application of the RAND/UCLA appropriateness method to one criterion (criterion one) from round one

Table 2 lists the median panel ratings, the amount of dispersion of panel ratings, and whether there was agreement or disagreement for the original criteria and the validated criteria. It also lists the amendments made by the panel to the original criteria, and the reasons for these amendments. There was 100% agreement for both median panel ratings and dispersion of panel ratings for the validated criteria. Table 3 contains the final list of validated criteria, arranged according to disease states. Table 4 lists usage information judged to be necessary for certain criteria.

Table 2

Changes made to original criteria according to agreement, disagreement and panel discussion

Table 3

Validated prescribing appropriateness criteria for older Australians (≥65 years) for commonly used medications and medical conditions*,†,‡ (*for usage information for certain criteria, see table 4)

Table 4

Criteria usage information


This study identified a panel of medication management experts to discuss and validate a set of 41 prescribing appropriateness criteria for commonly used medicines and medical conditions in older (≥65 years) Australians. Panel discussion resulted in retention of 39 of the originally proposed 48 criteria, with 25 being reworded, and 14 accepted with no change. These criteria do not simply represent a list of medications to avoid in the elderly, but also address issues such as the need for additional therapy (eg, criteria 23 and 34, table 3), additional tests (eg, criteria 18–20, table 3), ineffective treatment (eg, criteria 22 and 37, table 3) and medication monitoring (eg, criteria 10 and 20, table 3). They were designed to contribute to the Australian quality use of medicines process.94 The information required to apply these criteria may be obtained from the patients or their carer, and patient medical notes and/or their healthcare professional.95 It may also be provided by a Home Medicines Review referral form from the patients’ general practitioner.23 Owing to their currency and the nature of their development, we expect these criteria to make a significant contribution to the detection of DRPs in the Australian healthcare environment. For example, in a review of prescribing indicators for two conditions,36 which are common in older people in Australia—type 2 diabetes and cardiovascular disease96 ,97—disease-oriented and drug-orientated criteria such as ours have shown good content, face, concurrent and predictive validity and operational feasibility, as well as use for internal and external quality assessment in both ambulatory and hospital care.35 Evidence–practice gaps in Australia have been identified in other areas besides diabetes and cardiovascular disease, such as in asthma, pain and vaccination status.9 ,98–101 The existence of these gaps formed part of the developmental process for these criteria.

Prescribing appropriateness tools in Australia

Appropriateness of prescribing has been assessed by measures that are explicit or implicit, in an effort to identify and reduce DRPs.102 In Australia, both types of measures have been used.103–107 However, they have been imported into the Australian healthcare environment, with consequent shortcomings related to both the intrinsic nature of the measure, as well as environment compatibility issues. For example, in a study evaluating the impact of Home Medicine Reviews on appropriateness of prescribing, a significant number of recommendations made regarding the need for monitoring and addition of missing therapy were found to have no impact on explicitly derived scores using the Medication Appropriateness Index,103 due to the intrinsic shortcomings of this tool. This is not a tool that gives precise guidance in relation to specific medicines.13

The Beers criteria,108 perhaps the tool most widely used to assess inappropriate prescribing in older people, has been used in Australia, but requires modification to exclude medicines not listed for government subsidy.107 This is because medicine availability and use in Australia is largely determined by the Australian Pharmaceutical Benefits Scheme37. Other Australian studies have found that some medicines listed as inappropriate by Beers may be appropriate for certain older people according to Australian practice;105 many medicines listed by Beers are not available in Australia; and that some medicines considered inappropriate in Australia are not listed by Beers.106 Disagreement between Beers and other criteria, such as the improving prescribing in the elderly tool, have been identified.109

The Beers criteria was recently updated,22 with approximately half the medicines listed being unavailable in Australia. Further, almost three quarters of the diseases or syndromes listed are not among the 40 problems most frequently managed in patients over 65 years of age by Australian general practitioners.97 Beers still contains recommendations to avoid some medicines that are recommended for certain older people in Australia such as amiodarone, and it has recently been shown that rhythm control in older patients with atrial fibrillation may be more effective than rate control in reducing mortality over the long term.110 Reviews of explicit and implicit criteria have identified these and other problems such as failure to address drug–drug interactions and drug duplication, errors in recommendations, underrepresentation of certain drug categories, inclusion of infrequently prescribed drugs, criteria that are inapplicable for all situations, disagreement between criteria and lack of organisation of criteria.45 ,102 ,111

This has resulted in the development by others of criteria more suited to their own particular healthcare environment.112 ,113 Nationally based criteria have been described as the most desirable type of criteria, as they do not necessitate adaptation to local guidelines or national formularies before they can be used with confidence.32 In 2008, we therefore sought to construct and validate a set of prescribing appropriateness criteria relevant to the Australian healthcare environment. Our development process differed from most other tools21 ,108 ,112–117 as it did not initially involve a consensus panel, which has now been addressed. This development process also resulted in criteria unavailable in other tools such as monitoring, underprescribing, need for additional tests, evaluation of smoking and vaccination status, and certain drug interactions.32 ,45 ,102 Because we have generally named drug classes rather than specific drugs (table 3), and targeted common medical conditions found in older patients,118 ,119 we anticipate that our work may have some international usefulness.

Despite a desire in Australia to develop decision support tools to improve healthcare quality,120 progress has consisted of the development of a limited number of non-age specific structure and process indicator lists for use in hospitals and general practice.40 ,121–123 Many of these lists require updating.32 ,113 ,124 Currently, there is no Australian prescribing appropriateness criteria list to assist in improving medication management in older people. The usefulness of such an approach has been acknowledged, together with other approaches such as medication review.125


Over 80% of older Australians have three or more chronic conditions,96 with Australian general practitioners shown to be dealing more frequently with patients presenting with three or four problems in the year 2009–2010 compared with 2000–2001.126 Comorbidity is associated with poor quality of life, physical disability, high healthcare use, multiple medicines with consequent increased risk of adverse drug events, fragmentation of care and increased mortality.119 ,127 Yet most Australian guidelines for chronic diseases do not modify or discuss the applicability of their recommendations to older patients with multiple comorbid conditions.34 This situation is not restricted to Australia.127 ,128 Because the risk of harm in older patients increases in proportion to the number of treatments prescribed, prioritisation of therapeutic goals is necessary. For example, coronary heart disease (CHD) is an important morbidity in Australia77 ,96 for which treatment with ACE inhibitors or angiotensin 2 antagonists has been recommended to reduce the risk of cardiovascular events.70 ,71 Other criteria derived outside Australia such as STOPP/START do not include this recommendation.21 However, the presence of comorbidity in CHD (commonly arthritis or respiratory disease) or other clinical factors (such as dizziness, falls or patient preference) may mean that medicines such as these are never started, due to consideration of other factors. While we wished to identify problems such as these, the ultimate decision regarding medicine use should always be made on a case-by-case basis based on clinical experience, a discussion between the healthcare professional and the patient, and best available evidence.72 Issues such as these may run counter to recommendations of disease-specific, evidence-based guidelines.34 Addition of our criteria with this associated usage information (table 4) to the implicit processes of Australian medication review may assist in addressing the problem of comorbidity.

The oldest old

Knowledge about the state of health and function of the oldest old is limited,129 with research on their drug use being scarce, and often based on small and selected samples without comparison with other age groups.130 ,131 We know that older patients in general are underrepresented in clinical trials, so that disease-specific guideline recommendations based on evidence may not apply to older cohorts.34 For example, undertreatment with antiosteoporotic medicines has been identified as a significant evidence–practice gap in Australia.98 While STOPP/START criteria recommend calcium and vitamin D supplements,21 no recommendations for more specific medicines are made. Further, evidence available for fracture risk reduction has been reported to differ with age.90 Similarly, blood pressure targets appropriate for older patients may not be appropriate for the oldest old,50 with adverse effects for antihypertensives found to be among the most frequent in centenarians.132 Issues regarding the oldest old appear in table 4, criteria 1, 2, 9, 18 and 39. We have attempted to achieve the advantages of using mostly explicit criteria, such as ease of application, with the addition of application information (tables 2 and 4) unavailable in our previous criteria set.

Rationale for the use of the RAND/UCLA appropriateness method

The RAND/UCLA appropriateness method has been used to rate lists ranging up to over 3000 indications, where panellists have been asked to use the clinical literature and their best clinical judgement to assess the appropriateness of performing a procedure. To do this, they have rated various clinical scenarios.46 While the number and type of our criteria may differ to this, similar criteria have been developed using the RAND/UCLA method. For example, in the development of indicators for patients undergoing total hip or total knee replacement, 1 of the 68 indicators stated that for such patients, ‘deep venous thrombosis prophylaxis should be provided for a minimum of 2 weeks after hospital discharge’.43 In the development of indicators for hazardous prescribing for general physicians (GPs) using this method, 1 of the 34 indicators identified the hazardous use of ‘NSAID in a patient with heart failure’.44 We therefore followed a similar protocol.

Nature of decision support tools

Panel members emphasised that criteria may not provide definitive answers, instead indicating potential problems that might need addressing, due to a perceived unacceptable variation in care.133 While performance indicators are designed to measure the result of statements made in clinical practice guidelines, these guidelines often provide recommendations for care independent of other considerations such as multiple comorbidities, advanced age, frailty, patient preferences, disease burden or limited life expectancy.134–136 In such cases, less stringent goals, deprescribing or non-prescription may be more appropriate.15 ,81 ,137 For example, a frail older patient with multiple comorbidities and one or more functional impairments may have a life expectancy of approximately 2 years or less.75 This raises the question of whether failure to intensify treatment81 or to underuse evidence-based therapies138 reflects appropriate clinical judgement or an inappropriate care gap. The panel felt strongly that use of indicators, guidelines or criteria providing clinical decision support should never replace critical thinking in patient care.139

Strengths and weaknesses

We have followed a recommended approach120 by suggesting criteria for which high-quality evidence exists linking best practice with improved outcomes; where there are established evidence–practice gaps98 ,99; and where the health conditions impose the greatest burden on the healthcare system. We used a validated consensus method, an expert panel of varied specialisation, and criteria written with the aim of conciseness and clarity.

In addition to face and content validity, these validated criteria, much like performance indicators, will require further developmental work to provide evidence of their acceptability, operational feasibility, reliability and degree of predictive validity.35 ,133 Some of this work has already started with the original criteria.95 Further, these criteria only cover commonly occurring medicines and medical conditions. In addition, judgements made by an expert panel may not be representative of all healthcare professionals.

Intended use

These validated criteria are intended for use by healthcare providers to enhance the quality of the Australian medication review process, for quality improvement, educational purposes and internal audit. They are also intended for external quality assessment, such as use by policy makers and for public reporting. Stakeholder involvement will be critical to facilitate local uptake and encourage further research into the effects on health outcomes.125


This study validated 41 prescribing appropriateness criteria to assist in identifying DRPs in older (≥65 years) Australians. These criteria are intended to represent an addition to the medication management skill set that includes consideration of limited life expectancy, evidence base in the oldest old, drug burden and care coordination, patient and care-giver education, empowerment for self management, and shared decision-making. These skills are far from a ‘do everything for everyone’ philosophy, where aggressive treatment may encourage more care, not more appropriate care.31 ,135 Despite the presence of clinical decision support tools, healthcare providers need to know how to think about clinical problems, not just what to think.139



  • Contributors BJB designed and organised the study, analysed and interpreted the data and drafted the manuscript. TFC and RJM made substantial contributions to the conception, design, analysis and interpretation of the data and to critically revising the draft. All authors take responsibility for the accuracy and integrity of the study. All authors have given final approval of the version to be published.

  • Competing interests None.

  • Ethics approval Human Research Ethics Committee of the University of Sydney.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement No additional data are available.