IDEAS home Printed from
   My bibliography  Save this article

Ordering anomalies in choice experiments


  • Day, Brett
  • Pinto Prades, Jose-Luis


This paper investigates whether responses to choice experiments (CEs) are subject to ordering anomalies. While previous research has focussed on the possibility that such anomalies relate to position in the sequence of choice tasks, our research reveals that the particular order of tasks matters. Using a novel experimental design that allows us to test our hypotheses using simple nonparametric statistics, we observe ordering anomalies in CE data similar to those recorded in the dichotomous choice contingent valuation literature. Those ordering effects operate in both price and commodity dimensions and are observed to compound over a series of choice tasks. Our findings cast serious doubt on the current practice of asking each respondent to undertake several choice tasks in a CE while treating each response as an independent observation on that individual's preferences.

Suggested Citation

  • Day, Brett & Pinto Prades, Jose-Luis, 2010. "Ordering anomalies in choice experiments," Journal of Environmental Economics and Management, Elsevier, vol. 59(3), pages 271-285, May.
  • Handle: RePEc:eee:jeeman:v:59:y:2010:i:3:p:271-285

    Download full text from publisher

    File URL:
    Download Restriction: Full text for ScienceDirect subscribers only

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    1. Clark, Jeremy & Friesen, Lana, 2008. "The causes of order effects in contingent valuation surveys: An experimental investigation," Journal of Environmental Economics and Management, Elsevier, vol. 56(2), pages 195-206, September.
    2. Brooks, Richard AU -, 1996. "EuroQol: the current state of play," Health Policy, Elsevier, vol. 37(1), pages 53-72, July.
    3. DeShazo, J. R., 2002. "Designing Transactions without Framing Effects in Iterative Question Formats," Journal of Environmental Economics and Management, Elsevier, vol. 43(3), pages 360-385, May.
    4. Flachaire, Emmanuel & Hollard, Guillaume, 2007. "Starting point bias and respondent uncertainty in dichotomous choice contingent valuation surveys," Resource and Energy Economics, Elsevier, vol. 29(3), pages 183-194, September.
    5. David Aadland & Arthur Caplan & Owen Phillips, 2007. "A Bayesian examination of information and uncertainty in contingent valuation," Journal of Risk and Uncertainty, Springer, vol. 35(2), pages 149-178, October.
    6. Swait, Joffre & Adamowicz, Wiktor, 2001. " The Influence of Task Complexity on Consumer Choice: A Latent Class Model of Decision Strategy Switching," Journal of Consumer Research, Oxford University Press, vol. 28(1), pages 135-148, June.
    7. Amos Tversky & Daniel Kahneman, 1991. "Loss Aversion in Riskless Choice: A Reference-Dependent Model," The Quarterly Journal of Economics, Oxford University Press, vol. 106(4), pages 1039-1061.
    8. Herriges, Joseph A. & Shogren, Jason F., 1996. "Starting Point Bias in Dichotomous Choice Valuation with Follow-Up Questioning," Journal of Environmental Economics and Management, Elsevier, vol. 30(1), pages 112-131, January.
    9. Bateman, Ian J. & Langford, Ian H. & Jones, Andrew P. & Kerr, Geoffrey N., 2001. "Bound and path effects in double and triple bounded dichotomous choice contingent valuation," Resource and Energy Economics, Elsevier, vol. 23(3), pages 191-213, July.
    10. Ready Richard C. & Whitehead John C. & Blomquist Glenn C., 1995. "Contingent Valuation When Respondents Are Ambivalent," Journal of Environmental Economics and Management, Elsevier, vol. 29(2), pages 181-196, September.
    11. Kevin J. Boyle & Richard C. Bishop & Michael P. Welsh, 1985. "Starting Point Bias in Contingent Valuation Bidding Games," Land Economics, University of Wisconsin Press, vol. 62(2), pages 188-194.
    12. John C. Whitehead, 2002. "Incentive Incompatibility and Starting-Point Bias in Iterative Valuation Questions," Land Economics, University of Wisconsin Press, vol. 78(2), pages 285-297.
    13. Thomas P. Holmes & Kevin J. Boyle, 2005. "Dynamic Learning and Context-Dependence in Sequential, Attribute-Based, Stated-Preference Valuation Questions," Land Economics, University of Wisconsin Press, vol. 81(1).
    14. Flores, Nicholas E. & Strong, Aaron, 2007. "Cost credibility and the stated preference analysis of public goods," Resource and Energy Economics, Elsevier, vol. 29(3), pages 195-205, September.
    15. Cameron Trudy Ann & Quiggin John, 1994. "Estimation Using Contingent Valuation Data from a Dichotomous Choice with Follow-Up Questionnaire," Journal of Environmental Economics and Management, Elsevier, vol. 27(3), pages 218-234, November.
    16. Bateman, Ian J. & Cole, Matthew & Cooper, Philip & Georgiou, Stavros & Hadley, David & Poe, Gregory L., 2004. "On visible choice sets and scope sensitivity," Journal of Environmental Economics and Management, Elsevier, vol. 47(1), pages 71-93, January.
    17. Adamowicz, Wiktor L., 2004. "What's it worth? An examination of historical trends and future directions in environmental valuation," Australian Journal of Agricultural and Resource Economics, Australian Agricultural and Resource Economics Society, vol. 48(3), September.
    18. G. Cornelis van Kooten & Emina Krcmar & Erwin H. Bulte, 2001. "Preference Uncertainty in Non-Market Valuation: A Fuzzy Approach," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 83(3), pages 487-500.
    19. Donald M. McLeod & Olvar Bergland, 1999. "Willingness-to-Pay Estimates Using the Double-Bounded Dichotomous-Choice Contingent Valuation Format: A Test for Validity and Precision in a Bayesian Framework," Land Economics, University of Wisconsin Press, vol. 75(1), pages 115-125.
    20. Powe, N. A. & Bateman, I. J., 2003. "Ordering effects in nested 'top-down' and 'bottom-up' contingent valuation designs," Ecological Economics, Elsevier, vol. 45(2), pages 255-270, June.
    21. Bateman, Ian J. & Brouwer, Roy, 2006. "Consistency and construction in stated WTP for health risk reductions: A novel scope-sensitivity test," Resource and Energy Economics, Elsevier, vol. 28(3), pages 199-214, August.
    Full references (including those not matched with items on IDEAS)


    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.

    Cited by:

    1. Petrolia, Daniel & Interis, Matthew & Hwang, Joonghyun, 2015. "Single-Choice, Repeated-Choice, and Best-Worst Elicitation Formats: Do Results Differ and by How Much?," Working Papers 212479, Mississippi State University, Department of Agricultural Economics.
    2. Bruno Lanz & Allan Provins, 2015. "Using discrete choice experiments to regulate the provision of water services: do status quo choices reflect preferences?," Journal of Regulatory Economics, Springer, vol. 47(3), pages 300-324, June.
    3. McNair, Ben J. & Bennett, Jeff & Hensher, David A., 2011. "A comparison of responses to single and repeated discrete choice questions," Resource and Energy Economics, Elsevier, vol. 33(3), pages 554-571, September.
    4. Daniel R. Petrolia & Matthew G. Interis & Joonghyun Hwang, 2014. "America's Wetland? A National Survey of Willingness to Pay for Restoration of Louisiana's Coastal Wetlands," Marine Resource Economics, University of Chicago Press, vol. 29(1), pages 17-37.
    5. repec:eee:transe:v:106:y:2017:i:c:p:160-177 is not listed on IDEAS
    6. Rocamora, Beatriz & Colombo, Sergio & Glenk, Klaus, 2014. "El impacto de las respuestas inconsistentes en las medidas de bienestar estimadas con el método del experimento de elección," Economia Agraria y Recursos Naturales, Spanish Association of Agricultural Economists, vol. 14(2).
    7. Mikołaj Czajkowski & Marek Giergiczny & William H. Greene, 2012. "Learning and Fatigue Effects Revisited. The Impact of Accounting for Unobservable Preference and Scale Heterogeneity on Perceived Ordering Effects in Multiple Choice Task Discrete Choice Experiments," Working Papers 2012-08, Faculty of Economic Sciences, University of Warsaw.
    8. Nguyen, Thanh Cong & Robinson, Jackie & Kaneko, Shinji & Komatsu, Satoru, 2013. "Estimating the value of economic benefits associated with adaptation to climate change in a developing country: A case study of improvements in tropical cyclone warning services," Ecological Economics, Elsevier, vol. 86(C), pages 117-128.
    9. Balbontin, Camila & Hensher, David A. & Collins, Andrew T., 2017. "Integrating attribute non-attendance and value learning with risk attitudes and perceptual conditioning," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 97(C), pages 172-191.
    10. Carlsson, Fredrik & Kataria, Mitesh & Krupnick, Alan & Lampi, Elina & Löfgren, Åsa & Qin, Ping & Sterner, Thomas, 2013. "The truth, the whole truth, and nothing but the truth—A multiple country test of an oath script," Journal of Economic Behavior & Organization, Elsevier, vol. 89(C), pages 105-121.
    11. Bruno Lanz & Allan Provins, 2013. "Valuing Local Environmental Amenity with Discrete Choice Experiments: Spatial Scope Sensitivity and Heterogeneous Marginal Utility of Income," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 56(1), pages 105-130, September.
    12. McNair, Ben J. & Hensher, David A. & Bennett, Jeff, 2010. "Modelling heterogeneity in response behaviour towards a sequence of discrete choice questions: a latent class approach," MPRA Paper 23427, University Library of Munich, Germany.
    13. Day, Brett & Bateman, Ian J. & Carson, Richard T. & Dupont, Diane & Louviere, Jordan J. & Morimoto, Sanae & Scarpa, Riccardo & Wang, Paul, 2012. "Ordering effects and choice set awareness in repeat-response stated preference studies," Journal of Environmental Economics and Management, Elsevier, vol. 63(1), pages 73-91.
    14. Hoyos, David, 2010. "The state of the art of environmental valuation with discrete choice experiments," Ecological Economics, Elsevier, vol. 69(8), pages 1595-1603, June.
    15. Ben McNair & David Hensher & Jeff Bennett, 2012. "Modelling Heterogeneity in Response Behaviour Towards a Sequence of Discrete Choice Questions: A Probabilistic Decision Process Model," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 51(4), pages 599-616, April.
    16. Meyerhoff, Jürgen & Glenk, Klaus, 2015. "Learning how to choose—effects of instructional choice sets in discrete choice experiments," Resource and Energy Economics, Elsevier, vol. 41(C), pages 122-142.
    17. Leong, Waiyan & Hensher, David A., 2012. "Embedding multiple heuristics into choice models: An exploratory analysis," Journal of choice modelling, Elsevier, vol. 5(3), pages 131-144.
    18. Achyut Kafle & Stephen K. Swallow & Elizabeth C. Smith, 2015. "Does Public Funding Affect Preferred Tradeoffs and Crowd-In or Crowd-Out Willingness to Pay? A Watershed Management Case," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 60(3), pages 471-495, March.
    19. Dekker, T. & Koster, P.R. & Brouwer, R., 2012. "Changing with the tide: Semi-parametric estimation of preference dynamics," Serie Research Memoranda 0005, VU University Amsterdam, Faculty of Economics, Business Administration and Econometrics.
    20. Carlsson, Fredrik & Raun Mørkbak, Morten & Bøye Olsen, Søren, 2010. "The first time is the hardest: A test of ordering effects in choice experiments," Working Papers in Economics 470, University of Gothenburg, Department of Economics.
    21. Gabriela Scheufele & Jeff Bennett, 2012. "Response Strategies and Learning in Discrete Choice Experiments," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 52(3), pages 435-453, July.
    22. Campbell, Danny & Boeri, Marco & Doherty, Edel & George Hutchinson, W., 2015. "Learning, fatigue and preference formation in discrete choice experiments," Journal of Economic Behavior & Organization, Elsevier, vol. 119(C), pages 345-363.


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jeeman:v:59:y:2010:i:3:p:271-285. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Dana Niculescu). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.