IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2506.05996.html
   My bibliography  Save this paper

Statistical significance in choice modelling: computation, usage and reporting

Author

Listed:
  • Stephane Hess
  • Andrew Daly
  • Michiel Bliemer
  • Angelo Guevara
  • Ricardo Daziano
  • Thijs Dekker

Abstract

This paper offers a commentary on the use of notions of statistical significance in choice modelling. We argue that, as in many other areas of science, there is an over-reliance on 95% confidence levels, and misunderstandings of the meaning of significance. We also observe a lack of precision in the reporting of measures of uncertainty in many studies, especially when using p-values and even more so with star measures. The paper provides a precise discussion on the computation of measures of uncertainty and confidence intervals, discusses the use of statistical tests, and also stresses the importance of considering behavioural or policy significance in addition to statistical significance.

Suggested Citation

  • Stephane Hess & Andrew Daly & Michiel Bliemer & Angelo Guevara & Ricardo Daziano & Thijs Dekker, 2025. "Statistical significance in choice modelling: computation, usage and reporting," Papers 2506.05996, arXiv.org, revised Jun 2025.
    • Handle: RePEc:arx:papers:2506.05996
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2506.05996
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Mokhtarian, Patricia L., 2016. "Discrete choice models’ ρ2: A reintroduction to an old friend," Journal of choice modelling, Elsevier, vol. 21(C), pages 60-65.
    2. King, Gary & Roberts, Margaret E., 2015. "How Robust Standard Errors Expose Methodological Problems They Do Not Fix, and What to Do About It," Political Analysis, Cambridge University Press, vol. 23(2), pages 159-179, April.
    3. C. Angelo Guevara, 2024. "Endogeneity in discrete choice models," Chapters, in: Stephane Hess & Andrew Daly (ed.), Handbook of Choice Modelling, chapter 23, pages 668-692, Edward Elgar Publishing.
    4. Ronald L. Wasserstein & Allen L. Schirm & Nicole A. Lazar, 2019. "Moving to a World Beyond “p," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 1-19, March.
    5. Valentin Amrhein & Sander Greenland, 2018. "Remove, rather than redefine, statistical significance," Nature Human Behaviour, Nature, vol. 2(1), pages 4-4, January.
    6. Chiou, Lesley & Walker, Joan L., 2007. "Masking identification of discrete choice models under simulation methods," Journal of Econometrics, Elsevier, vol. 141(2), pages 683-703, December.
    7. Daniel J. Benjamin & James O. Berger & Magnus Johannesson & Brian A. Nosek & E.-J. Wagenmakers & Richard Berk & Kenneth A. Bollen & Björn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Chr, 2018. "Redefine statistical significance," Nature Human Behaviour, Nature, vol. 2(1), pages 6-10, January.
      • Daniel Benjamin & James Berger & Magnus Johannesson & Brian Nosek & E. Wagenmakers & Richard Berk & Kenneth Bollen & Bjorn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Christopher Chambe, 2017. "Redefine Statistical Significance," Artefactual Field Experiments 00612, The Field Experiments Website.
    8. Chiara Calastri & Romain Crastes dit Sourd & Stephane Hess, 2020. "We want it all: experiences from a survey seeking to capture social network structures, lifetime events and short-term travel and activity planning," Transportation, Springer, vol. 47(1), pages 175-201, February.
    9. Krinsky, Itzhak & Robb, A Leslie, 1986. "On Approximating the Statistical Properties of Elasticities," The Review of Economics and Statistics, MIT Press, vol. 68(4), pages 715-719, November.
    10. Ronald L. Wasserstein & Nicole A. Lazar, 2016. "The ASA's Statement on p -Values: Context, Process, and Purpose," The American Statistician, Taylor & Francis Journals, vol. 70(2), pages 129-133, May.
    11. Hess, Stephane & Palma, David, 2019. "Apollo: A flexible, powerful and customisable freeware package for choice model estimation and application," Journal of choice modelling, Elsevier, vol. 32(C), pages 1-1.
    12. Krinsky, Itzhak & Robb, A Leslie, 1990. "On Approximating the Statistical Properties of Elasticities: A Correction," The Review of Economics and Statistics, MIT Press, vol. 72(1), pages 189-190, February.
    13. Stephane Hess & David Bunch & Andrew Daly, 2025. "Get me out of this hole: a profile likelihood approach to identifying and avoiding inferior local optima in choice models," Papers 2506.02722, arXiv.org.
    14. Moshe Ben-Akiva & Joffre Swait, 1986. "The Akaike Likelihood Ratio Index," Transportation Science, INFORMS, vol. 20(2), pages 133-136, May.
    15. Daly, Andrew & Hess, Stephane & de Jong, Gerard, 2012. "Calculating errors for measures derived from choice modelling estimates," Transportation Research Part B: Methodological, Elsevier, vol. 46(2), pages 333-341.
    16. Guevara, C. Angelo, 2018. "Overidentification tests for the exogeneity of instruments in discrete choice models," Transportation Research Part B: Methodological, Elsevier, vol. 114(C), pages 241-253.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Isler, Cassiano Augusto & Blumenfeld, Marcelo & Caldeira, Gabriel Pereira & Roberts, Clive, 2024. "Long-Distance railway mode choice in Brazil: Evidence from a discrete choice experiment," Research in Transportation Economics, Elsevier, vol. 104(C).
    2. Daly, Andrew & Hess, Stephane & Ortúzar, Juan de Dios, 2023. "Estimating willingness-to-pay from discrete choice models: Setting the record straight," Transportation Research Part A: Policy and Practice, Elsevier, vol. 176(C).
    3. David A. J. Meester & Stephane Hess & John Buckell & Thomas O. Hancock, 2023. "Can decision field theory enhance our understanding of health‐based choices? Evidence from risky health behaviors," Health Economics, John Wiley & Sons, Ltd., vol. 32(8), pages 1710-1732, August.
    4. Curtis Rollins, 2023. "Investigating cost non‐attendance as a driver of inflated welfare estimates in mixed‐logit models," Journal of Agricultural Economics, Wiley Blackwell, vol. 74(3), pages 921-934, September.
    5. Scaccia, Luisa & Marcucci, Edoardo & Gatta, Valerio, 2023. "Prediction and confidence intervals of willingness-to-pay for mixed logit models," Transportation Research Part B: Methodological, Elsevier, vol. 167(C), pages 54-78.
    6. Kelter, Riko, 2022. "Power analysis and type I and type II error rates of Bayesian nonparametric two-sample tests for location-shifts based on the Bayes factor under Cauchy priors," Computational Statistics & Data Analysis, Elsevier, vol. 165(C).
    7. Carson, Richard T. & Czajkowski, Mikołaj, 2019. "A new baseline model for estimating willingness to pay from discrete choice models," Journal of Environmental Economics and Management, Elsevier, vol. 95(C), pages 57-61.
    8. Hala Abou- Ali, 2012. "Willingness to pay for improving land and water conditions for agriculture in Damietta, Egypt," Chapters, in: Hala Abou-Ali (ed.), Economic Incentives and Environmental Regulation, chapter 3, pages 46-70, Edward Elgar Publishing.
    9. Romain Crastes dit Sourd, 2024. "A new empirical approach for mitigating exploding implicit prices in mixed multinomial logit models," American Journal of Agricultural Economics, John Wiley & Sons, vol. 106(1), pages 76-95, January.
    10. Glenk, Klaus & Colombo, Sergio, 2013. "Modelling Outcome-Related Risk in Choice Experiments," Australian Journal of Agricultural and Resource Economics, Australian Agricultural and Resource Economics Society, vol. 57(4), pages 1-20.
    11. Qari, Salmai & Börger, Tobias & Lohse, Tim & Meyerhoff, Jürgen, 2024. "The value of national defense: Assessing public preferences for defense policy options," European Journal of Political Economy, Elsevier, vol. 85(C).
    12. Stephane Hess & David Bunch & Andrew Daly, 2025. "Get me out of this hole: a profile likelihood approach to identifying and avoiding inferior local optima in choice models," Papers 2506.02722, arXiv.org.
    13. Sergio Colombo & Nick Hanley & Jordan Louviere, 2009. "Modeling preference heterogeneity in stated choice data: an analysis for public goods generated by agriculture," Agricultural Economics, International Association of Agricultural Economists, vol. 40(3), pages 307-322, May.
    14. Daziano, Ricardo A. & Achtnicht, Martin, 2014. "Accounting for uncertainty in willingness to pay for environmental benefits," Energy Economics, Elsevier, vol. 44(C), pages 166-177.
    15. Colombo, Sergio & Christie, Michael & Hanley, Nick, 2013. "What are the consequences of ignoring attributes in choice experiments? Implications for ecosystem service valuation," Ecological Economics, Elsevier, vol. 96(C), pages 25-35.
    16. Brander, Luke & Eppink, Florian & Hof, Christine Madden & Bishop, Joshua & Riskas, Kimberly & Goñi, Victoria Guisado & Teh, Lydia & Teh, Louise, 2024. "Turtle Economic Value: The non-use value of marine turtles in the Asia-Pacific region," Ecological Economics, Elsevier, vol. 219(C).
    17. Joachim Marti & John Buckell & Johanna Catherine Maclean & Jody L. Sindelar, 2016. "To ‘Vape’ or Smoke? A Discrete Choice Experiment Among U.S. Adult Smokers," NBER Working Papers 22079, National Bureau of Economic Research, Inc.
    18. Matthews, Yvonne, 2023. "A hybrid and hierarchical stated preference study of freshwater restoration in Aotearoa New Zealand," Ecological Economics, Elsevier, vol. 203(C).
    19. Bliemer, Michiel C.J. & Rose, John M., 2013. "Confidence intervals of willingness-to-pay for random coefficient logit models," Transportation Research Part B: Methodological, Elsevier, vol. 58(C), pages 199-214.
    20. John Buckell & David A Hensher & Stephane Hess, 2021. "Kicking the habit is hard: A hybrid choice model investigation into the role of addiction in smoking behavior," Health Economics, John Wiley & Sons, Ltd., vol. 30(1), pages 3-19, January.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2506.05996. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.