Difficult Choices: What Influences the Error Variance in a Choice Experiment

In models of choice probability there can be heterogeneity both in individual preferences and in the error in the unobserved portion of utility. The error variance, or its inverse, the scale factor is often assumed to be identically distributed for all individuals and alternatives but this can be an unrealistic assumption. For this study we explicitly model the effect of observed variables on choice reliability through parameterization of the scale factor. We analyse Canterbury region residents’ preferences for water quality in New Zealand’s Hurunui River using a fully-ranked choice experiment with two treatment groups for elicitation format: best-worst and repeated-best ranking. We find that error variance decreases with each level of ranking. The best-worst sequential ranking technique is recommended in the literature but we find in practice it is associated with a higher error variance than an alternative, repeated-best technique. Choices which included one or more alternatives with a negative price (a reduction in local taxes) had a higher error variance and this has implications for estimates of gain/loss asymmetry. Conversely, people who had seen the river, or spent longer on the choice task, or rated their own level of understanding highly had a lower error variance. People who spent more time on a choice task also made more reliable choices, up to a point. We also find that parameterizing the scale factor reduces the standard deviation of random parameters in a mixed logit model. Scale variation confounds the identification of preference heterogeneity and care should therefore be taken to control for expected sources of this variation.