Article Text
PDF
Abstract
Introduction Quality-adjusted life years are derived using health state utility weights which adjust for the relative value of living in each health state compared with living in perfect health. Various techniques are used to estimate health state utility weights including time-trade-off and standard gamble. These methods have exhibited limitations in terms of complexity, validity and reliability. A new composite approach using experimental auctions to value health states is introduced in this protocol.
Methods and analysis A pilot study will test the feasibility and validity of using experimental auctions to value health states in monetary terms. A convenient sample (n=150) from a population of university staff and students will be invited to participate in 30 auction sets with a group of 5 people in each set. The 9 health states auctioned in each auction set will come from the commonly used EQ-5D-3L instrument. At most participants purchase 2 health states, and the participant who acquires the 2 ‘best’ health states on average will keep the amount of money they do not spend in acquiring those health states. The value (highest bid and average bid) of each of the 24 health states will be compared across auctions to test for reliability across auction groups and across auctioneers. A test retest will be conducted for 10% of the sample to assess reliability of responses for health states auctions. Feasibility of conducting experimental auctions to value health states will also be examined. The validity of estimated health states values will be compared with published utility estimates from other methods. This pilot study will explore the feasibility, reliability and validity in using experimental auction for valuing health states.
Ethics and dissemination Ethical clearance was obtained from Griffith University ethics committee. The results will be disseminated in peer-reviewed journals and major international conferences.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Strengths and limitations of this study
This study outlines a new valuation technique to explore more valid and reliable methods for health state valuations.
This will produce monetary values for health states.
This will be the first time experimental auctions are applied to health state valuations.
These methods need to be explored in a variety of combinations in the future.
Introduction
Economic evaluations of healthcare interventions often rely on the summary health measure of quality-adjusted life years (QALYs).1 The use of QALYs as an outcome measure allows comparable benefits across different interventions and diseases where QALYs capture changes in quantity and quality of life. Health-related quality of life is measured with respect to utility, denoting the relative preference of one health state versus another. Utility scores for each health state are generally derived using preference elicitation techniques2 with health states often described from multiattribute utility instruments such as the EQ-5D.3 The concept of expected utility theory as described by von Neumann and Morgenstern4 is used as the theoretical basis for preference elicitation techniques.
Using QALYs, results of economic evaluations are often presented in the form of an incremental cost-effectiveness ratio (ICER)5 where an ICER represents the marginal cost to gain one additional QALY. Decision makers then compare the ICER to a societal willingness to pay (WTP) threshold for acceptable value for money in making funding decisions (either implicitly or explicitly). The WTP threshold is the maximum that society is willing to allocate to a medical intervention which achieves a particular incremental gain in QALYs.6 WTP thresholds used by decision makers have been criticised for their apparent arbitrary nature.7 When the incremental cost per QALY gained for a new intervention is deemed to be lower than the stated WTP threshold, decision makers generally approve reimbursement and subsidisation of the proposed intervention, noting that value for money is one of multiple criteria used for decision-making. However, where WTP threshold estimates are arbitrary and not empirically derived, then the substantial investment decisions which use such WTP thresholds are themselves arbitrary and there may be an overinvestment or underinvestment in health. A recent review paper identified 24 empirical studies containing 383 estimates of WTP for a QALY.8 These estimates, ranged from €1000 to €4 864 167 (median €24 226; in 2010 Euros), underlying the need for a better measurement technique.8
The WTP estimates included in the review were broadly estimated using either revealed or stated preference techniques.8 Revealed preference use real choices that people make between competing alternatives to elicit value. For example, the higher wages needed to entice someone into an inherently unsafe job.9 However, they are disadvantaged by the need for an indirect valuation and assume that people are making rational and informed choices. In contrast, stated preference techniques use hypothetical scenarios. They are flexible but the downside is respondents know they are valuing a hypothetical scenario which can create questionable responses in terms of validity and reliability. Stated preference methods do not provide incentives for respondents to invest sufficient cognitive effort when thinking about their valuation decisions. Over the past decade, evidence has accumulated indicating that people overstate the amount they are willing to pay when hypothetical valuation questions are asked relative to when real money is involved.10 ,11
A new composite approach which combines the advantages of stated and revealed preferences may have merit. An experimental auction is such an approach which has been commonly applied to value goods in other areas.10 ,11 An advantage of experimental auctions is that people are placed within an active market environment where they can incorporate market feedback and where there are real consequences to expressing preferences that differ from what they actually want. In experimental auctions, bids are preferences obtained in a market with real money albeit with hypothetical health states. Therefore, experimental auctions push the preference elicitation into a new level by merging preference and WTP for health states in one measurement. Monetary values for health states can eliminate the arbitrarily set WTP thresholds and facilitate decision makers with direct comparisons of programme values.12 The aim of this article is to describe the protocol for a study to use experimental auctions to derive a monetary value of health states the ‘Health Auctions: a Valuation Experiment (HAVE) study’. After the conclusion of our data collection (anticipated in 2016), we will publish the results of this pilot study.
Methods
The protocol for this health state auction study will set out to test the validity, feasibility and reliability of health state auctions. This study will recruit a convenience sample of 150 university students and staff to auction EQ-5D-3L health states. Thirty auctions, each auction with five participants, will be held.
Health state development and training
This valuation exercise will value 25 health states of the 243 EQ-5D-3L health states; 24 health states were chosen from the 43 health states used in the measurement and valuation of health (MVH) study3 plus full health (11111). The MVH study was the first EQ-5D-3L health state valuation. We randomly selected 24 from these 43 health states and randomly divided these 24 into three groups as shown in table 1. The pits state of worst health (33333) was also added into each group (table 1). Therefore, each auction will be using 10 health states. However, 33333 is used as the reference health state and will not be directly valued in the auction.
Health states groups
Participants and recruitment
A recruitment booth will be set up on campus during the week of student orientation. Students and staff who walk past the booth will be invited to participate. Posters and emails will be distributed to advertise the study. Participant will be paid $10 to compensate them for their time. Recruitment officers will tend the booth and obtain participant consent. Those who provide consent will be sent to the auction room. The participants will be informed about the auction methods, the EQ-5D-3L instrument and the health states defined by this instrument. The participants will then be trained on the specific auction methods used in this study with training following the methodology described by Lusk and Shogren.10 ,11 Participants will take part in the training and auction as a group of five people.
Auction training and data collection
A reference manual for detailed training of instructors was developed, and data collectors and auctioneers will undergo a regime of training to conduct health state auctions and undertake electronic data entry. This will include auctioneers first participating in an auction as a bidder as well as reading materials regarding the theory of auctions and auction mechanisms. Before conducting an auction or entering data, a data collector (auctioneer) will shadow an experienced investigator for at least one auction before conducting an auction or entering data under the supervision of an experienced investigator.
Participants will be trained on the experimental auction mechanisms in six steps. Auction training will ensure participants avoid resorting to heuristics used in other markets or other study environments.10 ,11
Mechanisms of the experimental auction will be carefully explained to the participant as described in the training manual. Written instructions will also be provided in dot point form to provide a quick reference for participants. The term buyer will be used in the auctions instead of winner to reduce ‘auction fever’.
Investigators will provide a numerical example of second price auction as given in the manual.
Investigators will instruct the participants that it is in their best interest to bid truthfully.
A simple quiz will be used to test participants' knowledge of the mechanism. They will be asked about the mechanics and any misperceptions will be corrected.
Participants will be allowed to ask questions pertaining to the auction mechanics. The manual has developed frequently asked questions, and trainers are instructed to use these responses to provide consistency in their answers to the participants.
Investigators will conduct a round of real money practice auction with a snack bar (valued at $1.00) using the same auction mechanism. Participants will not be told the value of the snack bar but will be told that this is a legally binding transaction. Once completed, the highest bidder for the snack bar is asked to pay the amount they bid in exchange for the snack bar using their own money.
Auction design
After the training, participants will complete a questionnaire to collect their demographic data and disease history. Next, they will complete the EQ-5D-3L and visual analogue scale for their current health state. Participant groups will then be allocated to one of the three health states group and presented with 10 printed cards of the health states to be auctioned. They will be asked to read the health state description carefully and arrange them in ascending order according to their preference and record it on the questionnaire. This will familiarise them with the health states being auctioned.
Next, participants will be told that they will only live for 10 more years and then die. These 10 years will be spent in the worst possible health state (33333: I am confined to bed, I am unable to wash or dress myself, I am unable to perform my usual activities, I have extreme pain or discomfort and I am extremely anxious and depressed). They are then told that they have the opportunity, through the forthcoming auction, to buy a maximum of two health states with a duration of 5 years each. They are told that they do not have to buy any health states or they could buy one health state for 5 years and then live the remaining 5 years in the worst health state and then die. There are five bidders in an auction and only nine health states (including the full health, 11111) will be auctioned. Consequently at least one bidder will not be able to purchase two health states even they wished to.
The auction room will be set up with five booths facing the auctioneer. Five participants will be sitting in the booths. They will not be able to see each other. The auctioneer and the data entry personnel will sit in the front of the room with a clear view of the five participants. A multimedia projector will be used to illustrate the health state being auctioned. With each new health state, participants will be reminded that they are living in the worst health state.
Auction mechanism
For each group of participants, one auction for nine health states will be conducted. In each auction, the health states will be randomly selected from the nine health states allocated to the auction group including full health (but not 33333). A health state is randomly drawn and auctioned with replacement such that a health state can be auctioned more than once within an auction. However, only nine health states will be auctioned. Alternatively, there is a possibility that some health states might not come up for auction at all.
Participants will not be allowed to talk to each other after the auction has started. Each participant will be given an opening amount (of real money) as starting capital ($100) in the form of poker chips with one dollar denominations although they are told they can use less than one dollar increments for their bids and will receive change. The minimum increment of the bids will be set at $0.50. Participants will not be made aware that each person has the same amount of money.
The auction will consist of two rounds, a closed bid round and an open (English) round for each health state. The closed bid round will ask participants to provide a bid for the health state on a bid sheet. The participant with the lowest bid from the first round will not participate in the subsequent open round. The second highest bid of the first round will be the starting point of the open auction. The auctioneer will offer ascending bids until only one participant remains. This person purchases the health state at the highest bid value from the open auction.
The first round is designed to ensure that there is no anchoring to a starting bid provided by the auctioneer in the open auction. It will also encourage participants to consider a price high enough to win the lot but for the least amount of money. The participant with lowest bid is prevented from further participating to ensure that participants do not attempt to ‘game’ the auction by providing a low bid in the closed round before seeing what others have bid and settling on a value in the second round. This will encourage everyone to give a first bid as close as to what they are willing to pay (or at least as high as what they think others will be willing to pay). The second round provides feedback to participants regarding what other participants are willing to bid and provide an opportunity to reflect and adjust their bid. The second round facilitates an amount that represents the maximum that any of the participants would be willing to pay using a dynamic market. Even if a participant gives the highest bid in the closed round, he/she can drop out early in the open round.
Anonymity between participants will be kept during the auction. The person buying each health state will be kept from other participants to prevent participants estimating others purchasing capacity. Participants will hold their closed bid towards the front of the room allowing data entry of each bid. When the auctioneer starts the open round by calling out the bids anyone can stay in the round by holding their bid sheet up (except the lowest bidder in the closed bid round, who will be informed) regardless of their original bid. A participant's original closed bid will be updated when he/she is excluded from the opening bid round according to the new bid. The auctioneer will increase the bids until only one participant remains. If no one bids in the open auction, the highest bidder from the closed bid round would buy the offered health state. When a participant has bought two health states, they would no longer qualify for bidding. In the case of only one bidder in the closed round, the health state will be awarded without progressing to the open bid round.
Auction payoff reward
For each auction, there will be an overall winner who is deemed as the person with the ‘best health’ over the possible 10-year period. The ‘best health’ will be based on the average value of the two health states they purchased. Each health state will be valued based on the average bid (of final bids from each person) from that auction. This will ensure that no one participant will be able to influence the overall value of a health state or to over bid on an undesirable lot to increase its value. If two bidders have equivalent health over the 10 years, then the person who paid the least is considered the winner.
The winner of the auction will be rewarded with their unspent money for that auction. This provides an incentive for all participants to not only maximise their health (by winning the more desirable health states) but also to do so for the least amount of money. This encourages realistic bidding where the value they place on a health state represents a realistic opportunity cost (ie, less potential winnings). The ability of auctions to be incentive compatible and to be played out based on the ‘real’ value of money is the striking advantage of this approach compared with previous methods. While the scores are being calculated and before money is paid out, participants will be asked to respond to a Likert-type questionnaire exploring how easily they understood the health state auction.
Plan of analysis
A computerised data entry template in an Excel spreadsheet has been developed to capture bid and auction values for each health state for each participant during the auction. This will help ensure accurate data entry and minimise errors. The template will produce the market price for each health state in each individual auction, select the best purchased health states and the winner for each group of participants.
The primary outcome from this study is the feasibility, reliability and validity of monetary values of health states derived from experimental auctions. Secondary outcomes are the monetary values for the EQ-5D-3L health states, and the value of a QALY. As this is the first time the value for health states is to be determined by experimental auctions, an idea on the feasibility of this process is important. In assessing the feasibility, completion rate, mean time for completing the training and auction will be recorded. Questionnaires will be used to explore whether the participants were able to understand the auction process related to health state valuation. Concordance between data collector and participant rating on the ease of use of the auction process will be estimated.
Using 10% of the original sample, test retest reliability of the pilot study will be measured using Cronbach's α. The test retest will be conducted 2 weeks after the first data collection. Two auctions will be conducted exactly the same way with the same participants, health states set, health state sequence and auctioneer. Correlation coefficients between auction bids for the same health states across bidding rounds for each group of participants will be reported and correlation coefficients for mean bids across groups will also be estimated. The estimated monetary values for the 25 health states that were auctioned will be compared against the EQ-5D-3L Australian utility values to test external validity. Logical inconsistency of each participant's bids relative to how they ranked the health states and bids for auctioned health states will be used to test for internal validity of the values. Effects of participant characteristics on health state valuations will also be examined.
Regression methods will also be explored to develop an approach to derive monetary values for the entire 243 health states described by the EQ-5D-3L. Sample size calculations will be undertaken based on these pilot data to estimate the required sample to estimate monetary values for the full 243 health states with 80% power and 95% significance.
Discussion
Methods described in this study will produce the first ever health state values using an experimental auction approach. Instead of the usual utility derived for the health states based on the preferences of the population, this auction will produce dollar values for the health states. With this protocol, we suggest a new method to value health states using emerging experimental auctions methodology. Auctions are proven to be useful in extracting an accurate value for an item when the price is unknown. Thus, a dollar value for health states, derived from experimental auctions would indicate the WTP by the population for any relevant health state. This approach could eliminate the long-standing stalemate in health state valuation methodology of the inability to provide WTP values for a given health state by the general population.8 Potentially, there are substantial health technology assessment reform implications from the results of this study. For example, instead of using the cost-utility analysis framework, this approach would enable cost-benefit analysis. The outcome of health interventions could be measured by monetary values using the auction values produced from experimental auctions such as that described above. This would enable cost-benefit analysis to be undertaken aiding decision-making in healthcare.
Previous health state valuation studies only ask the participants to compare a given health state with full health to obtain the preference for that health state. Participants' individual attributes are not considered in health state. Similarly, we are standardising the participants of the HAVE study by asking them to consider the funds given as their only income and imagine they live in the worst possible health state. However, in secondary analyses, participant characteristics will be explored with respect to their bidding behaviour.
Previous studies using experimental auctions to elicit preference for health have relied on vignettes focusing on food safety.10 ,13 ,14 The vignettes contained information such as fatality rates and cost of illness. For example,10 ,13 participants bid on a stringently screened sandwich relative to a sandwich with a chance of becoming ill from one of five pathogens. These studies enable an estimate of the WTP to avoid or reduce the risk of a specific health condition and/or a very small risk of death. While these applications of experimental auctions are novel, they do not provide a feasible approach to value the full spectrum of health-related quality of life. Specifically, the approaches to date are disease specific, rely on non-standardised vignette descriptions of health and disease burden, and are restricted to the notion that the item participants are bidding for can render them in the health state described.
As such, these approaches are narrowly limited in their application and unlikely to represent a sufficient evidence base required for healthcare decision-making. For example, earlier studies using WTP methods to value health have been found to inflate valuations of a specific condition or intervention that respondents are asked about, relative to conditions or interventions that respondents are not asked about. This is sometimes known as ‘budget constraint bias’, where valuing items in isolation can lead to sum totals of WTP in excess of the available budget. Consequently, undertaking auctions to value health conditions in isolation is likely to result in a similar bias. However, these novel experiments do provide a significant improvement in purely hypothetical studies by providing a market environment with an incentive compatible mechanism to illicit truthful responses. That is, Neill et al15 and Blumenschein et al16 found hypothetical payment studies result in inflated valuations compared with real payment studies. The experimental auction approaches used in the above studies provide real monetary mechanisms to overcome the hypothetical nature of previous health state valuation studies. The challenge therefore is to expand the experimental auction approach, which encourages truthful valuations, to derive values over the full spectrum of health states.
This study will use a convenient sample and consist mainly of young adults. Hence, the values produced form this study will not be representative of Australian general population. The health state values will only be indicative but provide a useful benchmark which can be used for the design of future studies. The objective of the present study is to test the proposed experimental auction method for health state valuations. Depending on the success of the proposed study, the same methods can then be applied to a larger and more representative sample.
This protocol paves the way for an important juncture in the development of new methodology for health state valuation.
References
Footnotes
Contributors SK and JB initially conceptualised the methodology. PAS and DP aided the development of the methodology. All four authors contributed to writing up the paper. SK and JB are collecting data and undertaking data analysis.
Funding This study is funded by a Menzies Health Institute Queensland project grant.
Competing interests None declared.
Ethics approval The ethical clearance for the study is given by the Griffith University human research ethics committee (MED/61/14/HREC).
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement The results will be disseminated as peer-reviewed journal articles.