Evaluability

Evidence

Hsee (1996) described the evaluability hypothesis, according to which evaluations can reverse depending on whether options are judged separately or jointly. His first example: 116 students named their price for two second-hand music dictionaries — one with 10,000 entries and like new, one with 20,000 entries and a torn cover. Side by side the thicker one was worth more ($27 versus $19); judged alone, the immaculate one was ($24 versus $20). The number of entries cannot be placed on its own; the condition can.

The lever can be switched on and off. For two CD changers, 202 students judged harmonic distortion and capacity. Adding the usual market range for distortion — 0.002 percent at best, 0.012 at worst — made the reversal disappear entirely.

In the upper row the ranking flips (t = 3.32, p < .01 — a difference chance alone would produce in fewer than one case in a hundred); in the lower one it no longer does (t < 1, not significant). This is the most usable finding in the field: giving the usual range alongside a figure that is hard to place makes it judgeable without a comparison.

That prices too can be hard to place, the authors state explicitly. An attribute can be difficult to judge even when its values are given precisely and everyone understands them. Their own example: someone travelling who hears that a hotel room costs $50 a night cannot say whether that is high or low without knowing local price levels. In a further study with 294 students, the evaluation curve for an unfamiliar test score shifted with prior knowledge from a flat line (no information, F(3,96) = 1.51, not significant) to a steep one (range known, F(3,97) = 73.83, p < .001). The F tests whether the groups differ more than chance would lead you to expect.

A second question is closer to shops than the ranking is. Across six studies, Hsee and Leclerc (1998) ask not which of two options wins but whether a single option looks more attractive alone or paired. Their answer turns on the reference point: if both options sit above what the viewer assumes is usual, comparison hurts both. If both sit below it, comparison helps both. In the last two studies the measure was a purchase choice rather than a price; pooled, the interactions were clear (χ² = 8.39 and 15.09). The authors put it tersely themselves: good products are hurt by comparison and should be shown separately, lesser ones gain and should be shown together.

The biggest challenge comes not from the data but from the interpretation. Sher and McKenzie (2014) show that the same findings can be explained without any bias at all. Someone who does not know the market infers from the options shown what the market looks like — and then judges relative to that inference. In their first experiment with 269 students those inferences were enormous: people who saw only the candidate with 10 programs put the market average at 15.2 programs; those who saw only the one with 70 put it at 85.1 (all comparisons p < .001).

The second experiment is the decisive one. There 228 students were given other participants’ market estimates as background and then evaluated one person. The result: the more experienced candidate came out ahead under every information state ($48.6k against $44.9k, and $41.0k against $36.9k). The ranking therefore never flips — only the level of the valuations moves. The authors call this an evaluation reversal without a preference reversal, and they hold the behaviour to be not irrational.

Their third experiment doubles as a partial repetition of Hsee. Using the CD changers from his fourth experiment, the separate evaluation replicated ($205.50 against $187.70, t(109) = 2.27, p = .03). The joint evaluation ran in the expected direction but missed significance ($193.10 against $205.10, t(55) = 1.49, p = .14). Once two markedly worse devices were added, the choice reversed: in the pair 30 percent preferred the higher-capacity device, in the field of four, 77 percent.

The neighbouring hypothesis has been tested preregistered. Vonasch and colleagues (2023) repeated three studies from Hsee (1998) — the paper on the “less is better” effect, not the 1996 one — with 400 respondents.

The separate half holds three times out of three, the joint half once out of two. The authors conclude that the evidence for the evaluability hypothesis is weaker than assumed — precisely because its most surprising half holds so well and its unremarkable half does not.

Limits

The hypothesis has been tested, and it holds only halfway. What shows up repeatedly is the separate evaluation: without a comparison, the easy-to-place attribute carries the judgment. What does not hold reliably is the second half — that in comparison the hard-to-place attribute takes over and reverses the ranking. It missed significance in Sher and McKenzie and failed in one of two studies in Vonasch and colleagues.

The ice cream finding hangs on two responses. Vonasch and colleagues adopted Hsee’s exclusion rule and removed two outliers. Without that step the effect disappears (p = .64 instead of .01). They report both openly; the finding is not dependable.

The usual interpretation is contested. On Sher and McKenzie’s account the reversals are not preference reversals and not a reasoning error but sensible inferences from incomplete market knowledge. That changes nothing about the fact that valuations move — but it changes what to call it. The authors draw an uncomfortable conclusion of their own: whoever shapes the decision context can mislead even perfectly rational buyers. See reactance for the counter-movement.

The 1999 review is a narrative, not a meta-analysis. It gathers findings from hiring decisions, environmental programmes and consumer goods and assigns them to one explanation; it contains no pooled effect sizes.

Hsee and Leclerc tested only two options with two attributes and one reference point. What happens with more options, more attributes or several references they leave explicitly open. They also suspect that experienced buyers do not replace their internal yardstick with the neighbouring option so readily.

Sher and McKenzie limit their own model. It explains no reversals that hang on the form of the response, and none arising from incomparable goods. Their design also assumes that respondents take someone else’s market estimate at face value; in an unpublished variant with two visible options it worked only weakly and erratically.

That percentage and euro discounts differ in how well they can be evaluated, and that this explains the Rule of 100, remains a plausible interpretation. That prices can be hard to place at all is established; that this particular difference between the two discount formats does the work is not.

Nothing is actually bought in any of these papers. The samples are students and online respondents who named prices or ticked a purchase intention. Field evidence exists only as a second-hand pointer: coupons for brands in the same product category worked less well on a shared advertising page than on separate ones.