Some probabilistic models of best, worst, and best–worst choices

https://doi.org/10.1016/j.jmp.2005.05.003Get rights and content

Abstract

Over the past decade or so, a choice design in which a person is asked to select both the best and the worst option in an available set of options has been gaining favor over more traditional designs, such as where the person is asked, for instance, to: select the best option; select the worst option; rank the options; or rate the options. In this paper, we develop theoretical results for three overlapping classes of probabilistic models for best, worst, and best–worst choices, with the models in each class proposing specific ways in which such choices might be related. The models in these three classes are called random ranking and random utility, joint and sequential, and ratio scale. We include some models that belong to more than one class, with the best known being the maximum-difference (maxdiff) model, summarize estimation issues related to the models, and formulate a number of open theoretical problems.

Introduction

Finn and Louviere (1992) proposed a discrete choice task in which a person is asked to select both the best and the worst option in an available (sub)set of options. They presented an analytical model for data from that task, and considered its evaluation using an experimental design (2j fractional factorial) that ensures that each option, and each pair of distinct options, is presented equally often across the selected subsets of size j. Since the publication of that paper, interest in, and use of, such best-worst choice tasks has been increasing, with two recent empirical applications receiving “best paper” awards (Cohen, 2003, Cohen and Neira, 2003). In principle, best–worst tasks have a number of advantages over traditional discrete choice tasks: (1) a single pair of best–worst choices contains a great deal of information about the person's ranking of options (e.g., if there are 3 items in a set, one obtains the entire ranking of that set; if there are 4 items in a set, one obtains information on the implied best option in 9 of the 11 possible non-empty, non-singleton subsets;1 and if there are 5 items in a set, one obtains information on the implied best option in 18 of the 26 possible non-empty, non-singleton subsets); (2) best–worst tasks take advantage of a person's propensity to identify and respond more consistently to extreme options; and (3) best–worst tasks seem to be easy for people. Despite increasing use of the approach, the underlying models have not been axiomatized, leaving practitioners without clear guidelines on appropriate experimental designs, data analyses, and interpretation of results.

This paper develops theoretical results for three overlapping classes of probabilistic models for best, worst, and best–worst choices, with the models in each class proposing specific relationships between such choices. The model classes are called random ranking and random utility, joint and sequential, and ratio scale. We consider models that belong simultaneously to one or more of these classes, with the best known being the maximum-difference (maxdiff) model, which is introduced later in this section, and we formulate a number of open theoretical problems.

We now illustrate the framework, and the basic models, through the maximum-difference (maxdiff) model of best–worst choice. To do so we require some basic notation. Let T with |T|2 denote the finite set of  potentially available choice options, and for any subset XT, with |X|2, let BX(x) denote the probability that the alternative x is chosen as best in X, WX(y) the probability that the alternative y is chosen as worst in X, and BWX(x,y) the probability that, jointly, the alternative x is chosen as best in X and the alternative yx is chosen as worst in X. Thus 0BX(x),WX(y),BWX(x,y)1and xXBX(x)=yXWX(y)=x,yXxyBWX(x,y)=1.We assume throughout the paper that for each xT, B{x}(x)=W{x}(x)=1.

For motivational purposes only, we assume now that best and worst choices are in some sense more basic than simultaneous best–worst choices, and develop a model of the latter based on the former. So suppose that when asked to choose the best and the worst option in a (finite) set X, the person simultaneously, but independently, chooses the best, respectively, the worst, option in X. If the resulting options are distinct, these are reported as the best–worst pair of options in X, otherwise the person re-samples. As we show later in detail, such a process gives rise to the following representation for the best–worst choice probabilities in terms of the best and the worst choice probabilities:2 for x,yX,xyBWX(x,y)=BX(x)WX(y)r,sXrsBX(r)WX(s).

Now, suppose that the multinomial logit (MNL or Luce's choice) model holds separately for the best and the worst choice probabilities, i.e., there exist ratio scales b and w such that for x,yX,BX(x)=b(x)rXb(r),WX(y)=w(y)sXw(s).Then direct substitution of (2) in (1) yields that for xy,BWX(x,y)=b(x)w(y)r,sXrsb(r)w(s).

Notice that this combined set of representations has three interesting properties—the best–worst choice probabilities are represented in (1) directly in terms of the best and worst choice probabilities, and are represented in (3) in terms of the ratio scale values that determine the best and the worst choice probabilities, with the same functional form in both representations. Thus, this aggregation method is plausible, and interesting, in that it works both at the level of the choice probabilities and at the level of the scale values. We are immediately lead to the first theoretical question, namely, are there other (ratio scale) models that satisfy this type of aggregation property, and, if so, how large is this class. The detailed formulation of this problem requires extensive further notation, and its solution the use of complex functional equation techniques. The conjectured final result is that the class of solutions is large, but in an interesting sense the individual solutions do not differ greatly from the above model.

It is important to note that the above example was for choices from a given fixed set X. Often probabilistic models of choice (or, briefly, probabilistic choice models) are assumed to be consistent over all the subsets of a finite master set T. If that is assumed in the present context, and we consider the binary choice probabilities B{x,y}(x) and W{x,y}(y), then it is reasonable to assume that, and to test whether, B{x,y}(x)=W{x,y}(y)which with (2) gives that3 for each for zT,b(z)=1w(z)and so in particular any scale transform α of b is linked to the scale transform 1α of w. In the following work, we consider separately the cases where b and w are independent ratio scales, and where they are subject to common scale transforms.

The paper has the following structure. Each of the three main sections develops one of three classes of models, gives examples of that class, and presents solved, and open, theoretical problems about that class. Section 2 introduces the terminology and basic conditions, Section 3 presents random ranking and random utility models, Section 4 joint and sequential choice models, and Section 5 ratio scale models. Section 6 summarizes the results and restates the main open theoretical problems.

Section snippets

General terminology and conditions

Given a finite master set T,  and a particular set X, XT, we refer to the set {BX(x),xX}, {WX(y),yX}, {BWX(x,y),x,yX, xy}, respectively, as a set of best, worst, best–worst choice probabilities (on X). We have a complete set of best, worst, best–worst choice probabilities, respectively, (on a master set T) when we have a set of best, worst, best–worst choice probabilities on each X, XT. Unless stated otherwise, we assume that we have a complete set of  best, worst, and best–worst choice

Random ranking and random utility models

We are naturally interested in relations between these various choice and ranking probabilities. Since the focus of this paper is best–worst choice, we concentrate on how ranking probabilities might determine such best–worst choice probabilities, though we do consider one result in the other direction. There is extensive related work on how best choice probabilities determine, or are determined by, ranking probabilities (see Falmagne, 1978, Fishburn, 1994, Fishburn, 2002).

For each zT and rank

Joint and sequential best–worst choice models

We now motivate models through the idea that a person, in selecting the best–worst pair of options, approaches the selection of the best and the worst option independently, and follows up on these choices, as needed, to ensure that the same option is not selected as both best and worst. We first consider models where the best and worst choices are made jointly, then models where these choices are made sequentially.

A referee asked whether such an independence assumption is reasonable and

Ratio scale models

We now return to the maxdiff model, i.e., the example in Section 1, and use it to motivate a theoretical question concerning when a set of best, worst and best–worst choice probabilities are “of the same form” with the best–worst choice probabilities determined by some “functions” of the best and worst choice probabilities. Unfortunately, the notation required for the general formulation is complex, so we use the earlier example to illustrate the ideas, and include details in the Appendix. We

Summary and conclusions

We derived and discussed theoretical results for a number of best, worst and best–worst choice models, with the focus on the latter. Our results include a number of interesting theoretical relationships between these types of models, which in turn suggest a variety of tests to determine which model is most consistent with choice data. For example, the maximum-difference model is of the well-known Luce (1959), equivalently Multinomial Logit (McFadden, 1974), form with ratios of scale values, and

Acknowledgments

This research has been supported by Australian Research Council Discovery Grant DP034343632 to the University of Technology, Sydney, for Louviere, Street and Marley, and Natural Science and Engineering Research Council Discovery Grant 8124-98 to the University of Victoria for Marley. It was begun while Marley was a Visiting Researcher at Systems, Organizations and Management of the University of Groningen and supported by the Netherlands’ Organization for Scientific Research for the period July

References (23)

  • S. Barbera et al.

    Falmagne and the rationalizability of stochastic choices in terms of random orderings

    Econometrica

    (1986)
  • Cited by (455)

    • A variation of the cube model for best–worst choice

      2023, Journal of Mathematical Psychology
    View all citing articles on Scopus
    View full text