Background  

Health services have a complex decision-making environment and there are many opportunities for innovation.1 As identifying high value in healthcare becomes increasingly important, administrators must choose between the many ideas presented to them.2–4 Different initiatives such as new diagnostic hardware and service redesign compete for limited funds, but objectively choosing between initiatives is challenging.5 The preferences of multiple stakeholders need to be accounted for, including payers, clinicians and patients, who all influence healthcare provision.1

While payers desire value, other stakeholders have heterogeneous preferences. Physicians have the best interests of the patient in mind, but can cause conflict with hospital cost containment measures when preference items such as prosthetics vary substantially in price.6 Patient preferences may often directly oppose both cost-containment and physician preferences in an effort to obtain what patients perceive as optimal care. ‘Doctor shopping’ is one such example where patients will often circumvent value-based care by seeing multiple providers until they get the treatment they want.7 However, not all patient and physician preferences impact negatively on value-based care.

Rationale

Innovations to health services might provide benefits in many ways, yet these may not be comparable across projects. For example, a project to reduce medication mismanagement is not directly comparable to investing in an imaging device, yet these initiatives may find themselves in direct competition for funding. Considering every decision as an economic analysis is one method of evaluating projects using the same criteria, but this approach has been considered contentious and simplistic.8–10 Multiple perspectives often need to be represented and this creates a need for an objective multicriteria decision analysis (MCDA) framework.

MCDA is a method of evaluating the performance and relative importance of different adoption decisions in a holistic manner. MCDA tool has two aspects: quantifying performance and quantifying the weight of each performance category on the overall decision.11 12 This enables direct comparison across competing projects with multiple objectives.13 MCDA can provide decision-makers with visual representations of their preferences and how to prioritise high value care.12 13

The aim for this paper is to describe a simple and transparent MCDA framework for competing projects. It was developed in partnership with the leadership of a 900 bed public teaching hospital in Brisbane, Australia. The hospital employs more than 6000 staff, admits over 100 000 patients each year and is heavily involved in research. The hospital’s executive board used the decision tool over multiple iterations to improve its applicability and relevance. The value of the tool is that it is simple to use, employs a transparent methodology and provides visual summary of the outcomes to aid comparison and decision-making. We also provide some examples and outputs that show what the tool can do and how it can be used in the health system decision-making process.

Patient and public involvement

No patients were recruited for this paper.

Six parameters used for scoring projects

The net cost of implementing the programme is measured against the total benefits of its implementation. This represents the accounting cost to the provider of running the programme, rather than the opportunity cost of one programme over another.

Cash return on investment

Cash return on cash investment (ROI) is expressed as a ratio of dollars returned for every dollar invested. We used a linear transformation to keep scores between 0 and 2. The highest ROI is used as the denominator and given a score of 2, while each subsequent ROI is scored as a proportion of the maximum and multiplied by 2. Calculating ROI is shown in table 1. This field calculates returns in terms of cash only. Any financial benefits gained by improvements to capacity or safety, such as bed days avoided or lower medical costs from falls reductions, must not be included.

For example, if a project’s 5-year profit was $1 500 000 and cost $5 000 000, its ROI would be ($1 500 000/$5 000 000)=0.3. If the highest ROI was 4.0, this would receive a score of 2×(0.3/4.0)=0.15.

Capacity changes

The second parameter is whether the innovation releases capacity. These costs are prepaid or fixed in that the funding has already been allocated to keeping beds open regardless of occupancy, and increasing bed availability will not free up any cash.15 We developed the ‘c-score’ to measure change to capacity in terms of the length of stay (LOS) of the relevant clinical unit. The c-score is outlined in table 2. This formula adds the total change in throughput (change in LOS×number of patients) to the weighted impact of the added capacity (bed days added/LOS). This allows flexibility in how capacity is measured to include both changes to LOS and the impact of available beds in different wards. The weighted bed days component reflects the relative impact of freeing 24 hours of Emergency Department (ED) time compared with 24 hours of a standard ward. This field also uses linear transformation as per the ROI score.

For example, a project reduces LOS by 3 hours (0.125 days), affecting 40 patients per week on average. Average LOS is 5 hours, but no beds have been added, meaning the total c-score is (0.125*40)=5. This is the maximum c-score of any project and receives a score of 2× (5/5)=2.

Patient benefits

Scores to signal improvements to quality of life and patient satisfaction vary from 0 to 2. A score of 2 is assigned for quality of life when the literature shows, through meta-analysis or multisite Randomised Control Trial (RCT), improvements in health utility or health-related quality of life. A score of 1 would be given to projects with no available or reputable evidence in the literature, and a score of 0 for projects which have clear evidence of harm or ineffectiveness. Patient satisfaction must follow the same criteria. Access, defined as a reduction in waiting or travel time and patient costs, is a binary 0 or 1 for ‘no improvement’ or ‘improvement’, respectively, as measured in meta-analysis or primary data collection tools such as the Research and Development Corporation (RAND) Patient Satisfaction Questionnaire (PSQ)-18.

Due to the potential for bias, it is important that objective parties conduct a brief transparent and reproducible literature review to provide sound judgement. Advocates may overlook negative findings and focus on positive ones. As with the ROI and capacity fields, any purported improvements to quality of life must not arise from improvements in patient safety or reductions in length of stay, to avoid multicollinearity. An initiative with an evidence basis for all three parameters would receive a maximum score of 5, then scaled by the linear transformation method used for ROI and capacity. The score breakdowns are shown in table 3.

For example, a programme to install laminar airflow devices in all operating theatres displayed evidence in the literature of no improvement and, in some cases, even harm, scoring a 0 on quality of life. Patients were unaware of the development and did not see reduced wait times, scoring a 0 on both satisfaction and access. The initiative scored 0. The project with the highest outcomes scored 4, so the laminar airflow project was given a score of 2× (0/4)=0.

Patient safety

Patient safety indicators for Australian hospitals arise from the Australian Commission on Safety and Quality in Health Care which monitors 16 hospital-acquired conditions.16 The number of indicators addressed is summed and linearly transformed as per the above criteria. As with outcomes, there must be a clear body of evidence, either on balance or through meta-analysis, showing an improvement in the reported safety outcomes. For example, a project that incorporated daily, assisted walks with long stay patients addressed three conditions, including pressure ulcers, falls and venous thromboembolisms. The project addressing the most conditions scored a 6, giving the assisted walks project a 2× (3/6)=1.

Staff training and research

Staff training and research is required for many clinical personnel and represents an important part of professional development. It can take the form of sanctioned continuing professional development hours, scientific publication or simply improved skill sets and job satisfaction. Table 4 outlines the breakdown of training and research outcomes. These scoring outcomes must be determined by an objective party. As the field is scored from 0 to 2, no linear transformation is necessary.

For example,a project resulted in an academic publication qualifying it as research, but provided no additional training, scoring 1/2.

Organisational risk

Organisational risk weighs the impact of potential risks against the probability of their occurrence. Examples of possible risks include delayed implementation or a negative news article around the project. Classifying impact and probability depends on the organisation’s priorities and characteristics. For example, an organisation may deem probabilities below 10% as low and above 60% as high, and any project requiring debt financing as high impact. The intersection of risk impact and probability creates a matrix with a scoring system from 0 to 2. Ideal projects would have both a low impact and low probability of occurrence, scoring a 2. As with training and research, no scaling factor is required. Figure 1 displays the risk classification matrix.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Organisational risk scale for impact and probability assessments.

For example, a project was estimated to have a 1 in 3 chance of stalling when its main sponsor went on long service leave, classified by the board as a medium probability, low impact risk and scoring 1.5/2.

Weighting and discounting for outcomes in future time periods

Discounting is a standard procedure in financial valuation to account for time preference.17 We used the widely accepted 3% discount rate for ROI. Weighting is a more complicated system of establishing the priorities of the hospital decision-makers.11 17 Each value is multiplied by a weight and summed for the total project score. It is important that weights are agreed collaboratively ahead of time to account for heterogeneous preferences and to recognise the needs of different groups.18

We created weights by asking executives to rank fields by priority, in which two priorities could be of equal rank. Table 5 below shows a ranking system in which decision-makers prioritised ROI and outcomes, followed by safety, capacity, then training and risk as joint bottom. Ranks were summed, with the total then divided by each ranking for a score.

Maintaining consistent weights across a single decision window is crucial for objective measurement. Administrators and executives should decide on priorities and only change them once funding has been allocated and a new decision time period has begun. If weights are determined during or after projects are measured by the decision framework, it will introduce bias towards initiatives offering more points in the selected criteria. Rankings should be achieved through a discussion among key decision-makers and left unchanged for the remainder of the funding window.

Return on investment

EPICentre costs a flat $1.1 million each year, with no cash returned and an ROI of 0. ES Pod costs $3.75 m over 5 years, and generates $4.91 m in that time. ES Pod’s ROI after 5 years is $1.15 m divided by its cost of $3.75 m, or 0.31. Scaled as per the methods to the highest scoring project with an ROI of 1.21, ES Pod scored 2 (0.31/1.21), or 0.51 out of 2.

Capacity

EPICentre decreases ward LOS by 5 hours by reducing wait times for specialist consults. The project affects 39.3 patients per week, but adds no beds. Its c-score is (39.3×0.22+0)=8.6. Scaled to the highest scoring project of 35.7, it scored 2 (8.6/35.7), or 0.48 out of 2. ES Pod does not decrease LOS, but allows 3.4 extra patients to be admitted to the intensive care unit (ICU) per week, where the average LOS is 3.3 days. The c-score for ES Pod is (0+3.4/3.3)=1.03, scaled to 2× (1.03/35.7) or 0.06 out of 2.

Outcomes

There is no evidence of EPICentre improving patient quality of life outcomes directly, but a pilot survey showed increases in patient satisfaction and throughput, improving access. These outcomes summed to 4, leading all initiatives for a score of 2 out of 2. ES Pod shows no available evidence for changes to quality of life and satisfaction, but improves access through lower wait times. Outcomes summed to 3, receiving a score of 2× (3/4), or 1.5 out of 2.

Safety 

EPICentre was shown in a pilot programme to reduce falls, Hospital Acquired Infections (HAIs), unplanned ICU visits and venous thromboembolisms. The sum of four indicators was 1 short of the highest score of 5, calculated as 2× (4/5) or 1.6 out of 2. ES Pod was shown to be non-inferior to an ICU setting, but offered no additional safety benefits and received a score of 0/2.

Training and research

EPICentre and ES Pod were two of the highest scoring projects, with both offering Continuing Professional Development (CPD) hours and research manuscripts, receiving scores of 2/2.

Risk

Hospital executive board members determined that both EPICentre and ES Pod had medium probability risks (between 10% and 60%) of some financial risk and negative publicity, respectively. Both were allotted a risk score of 1.5/2.

Weighting

Each outcome measure was weighted according to preset preference allocations as shown in table 5. The weighted scores of all competing projects can be seen in table 6, ranked by cost-per-point gained.

A rational approach is to maximise the number of points obtained for a given budget by accepting all cost-saving and no cost projects (Chemotherapy funding change (CF2), insourcing chemotherapy supply (APHS), ES Pod, functional independence measurement (FIM), clinical nurse/clinical facilitator (CN/CF)), then selecting projects with the cheapest cost-per-point until the budget is exhausted. Figure 2 displays projects with a positive cost-per-point where the x-axis is points gained and the y-axis is cost.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Plotting projects on a plane after cost-saving projects are accepted. APHS, insourcing chemotherapy supply; CEP-CARU, epilepsy surgical centre; CF2, chemotherapy funding change; CN/CF, clinical nurse/clinical facilitator; EPICentre, early patient intervention centre; ES Pod, elective surgery pod; FIM, functional independence measurement; OPAT, outpatient antibiotic therapy; SW, social worker support; TMT, tracheotomy management team; VASE, vascular access education.

Dominated projects are those such as vascular access education (VASE) and epilepsy surgical centre (CEP-CARU) which are both less effective and more expensive than an alternative, and should not be pursued before the dominant strategies. Projects on the cost-effectiveness frontier, such as outpatient antbiotic therapy (OPAT), social worker support (SW), and tracheotomy management team (TMT) should be pursued first, in that order. This choice provides a good example of cost-effectiveness decision-making as there is a clear trade-off between points and costs as projects move up and right. A rational decision-maker might have a budget of $10 m for implementation and could use the decision tool to choose projects with the best cost-per-point starting with OPAT until the $10 m was depleted.

Implementation at the RBWH

The RBWH established a team in 2016 to retrospectively and prospectively evaluate service improvements at various stages of implementation. Needing a transparent and simple method of evaluation, the RBWH initially adopted an evaluation framework already in use at the regional level. This method resulted in large reports that consumed significant amounts of staff time in reporting, requiring significant drafting and editing to convey findings.

The methods used in this paper offered the RBWH a more targeted and concise approach to evaluation. While some written reporting is still necessary to summarise the implementation history and effectiveness of each initiative, adoption of this interface allows the RBWH to perform retrospective and prospective analyses in a shorter timeframe with more concise findings. This facilitates faster funding allocation decisions across competing programmes.

The RBWH has begun using the multicriteria decision-making tool to inform future resource allocation. The use of this decision-making tool gives decision-makers and funding applicants a level of confidence that allocations have been made in a systematic and deliberate manner. This predefined, objective set of criteria allows fair discrimination across competing projects.

Limitations

There are some drawbacks to this approach over non-MCDA methods. A notable downside is that political influences, such as publicly reported deadlines and government pressure, are not quantifiable within the tool. These factors tend to transcend typical decision metrics and distort objective, scientifically supported choices.21 Political pressures may also be reflected in the weighting of different criteria. Occasionally there are situations where a project must be completed for political reasons rather than for the benefit granted to the health system. In cases like this, the tool does not provide a justification for the decision to invest, but instead quantifies impacts of the project and how they compare to other interventions. By plotting politically necessitated projects on the graph in figure 2, it is also possible to show project outcomes to the political bodies demanding certain initiatives, and show why they may or may not be an effective option.

This tool is intended to be a guide, not a mandate, and will not solve healthcare problems overnight. It is also only as good as the data that go into it; gaming through selective interpretation of findings could misrepresent the nature of an intervention, twisting the tool by adding a veneer of credibility to subjective opinions. Similarly, avoiding double-counting is explicitly addressed in the methods, but a potential pitfall that must be avoided by the analyst. MCDA is just a component of an executive decision-making process, to be included in a holistic review of options. Ideally, it would be conducted at the beginning of the funding window as a prospective analysis of the expected benefits of all initiatives competing for funding. This is applicable to hospitals, health systems and hospital-based public policy, as MCDA can be valuable in all of these sectors.5

While many projects are cost-saving, the time horizon for these savings may accrue after several funding windows or years. This may mean that a project will not break even for 4 years, before a large windfall in the fifth. This nuance can be captured through a time factor in the ROI scoring system, where the interests of decision-makers can help choose the year to apply. Similarly, cost-saving projects also may require a significant implementation burden, which can often go beyond the system’s administrative capacity. If a ward or unit only has the staff time to implement four projects of similar scope per year, then funding is not as large a constraint as time and capacity. This highlights the use of the decision tool as part of a larger toolkit. Some decisions are beyond purely objective or quantifiable criteria, and require holistic analysis with input from all parties.

Weighting is determined in isolation from project performance. This removes the impact of a change’s magnitude on the weighting factor, such as might be seen in swing weighting. Instead, weights and performance are synthesised after they have been individually considered. This has some downsides. For example, a major improvement in health outcomes with a slight amount of risk may be scored above a slight improvement in health outcomes with no risk. We have attempted to address this in part through the relative nature of the linear transformation scale. When this does not apply, however, we ultimately must stress that this tool cannot be a full cost-effectiveness analysis on outcomes. As above, it must be considered as a component of an overall decision process, in which cost-effectiveness and equity should play a large part.

Finally, due to the subjective nature of weighting and some of the scoring metrics, we had to sacrifice some scientific rigour for ease of use and transparency. Ideally, each field would have had a methodologically robust scoring system in which each outcome could be objectively defined and multicollinearity avoided through explicit value definitions. Due to the prospective design, short timeframes and need for transparent and easily understood scoring process, true precision was difficult to achieve. Under the circumstances, we believe this tool strikes an acceptable balance between rigour and convenience.