Order Effects in Eliciting Preferences

Having an accurate account of preferences help governments design better policies for their citizens, organizations develop more efficient incentive schemes for their employees and adjust their product to better suit their clients' needs. The plethora of elicitation methods most commonly used can be broadly distinguished between methods that rely on people self-assessing and directly stating their preferences (qualitative) and methods that are indirectly inferring such preferences through choices in some task (quantitative). Alarmingly, the two approaches produce systematically different conclusions about preferences and, therefore, survey designers often include both quantitative and qualitative items. An important methodological question that is hitherto unaddressed is whether the order in which quantitative and qualitative items are encountered affects elicited preferences. We conduct three, pre-registered, studies with a total of 3,000 participants, where we elicit preferences about risk, time-discounting and altruism in variations of two conditions: 'Quantitative First' and 'Qualitative First'. We find significant and systematic order effects. Eliciting preferences through qualitative items first boosts inferred patience and altruism while using quantitative items first increases the cross-method correlation for risk and time preferences. We explore how monetary incentivization and introducing financial context modulates these results and discuss the implications of our findings in the context of nudging interventions as well as our understanding of the nature of preferences.