IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2506.05996.html

Statistical significance in choice modelling: computation, usage and reporting

Author

Listed:
  • Stephane Hess
  • Andrew Daly
  • Michiel Bliemer
  • Angelo Guevara
  • Ricardo Daziano
  • Thijs Dekker

Abstract

This paper offers a commentary on the use of notions of statistical significance in choice modelling. We review the reasons for uncertainty in parameter estimates, provide a precise discussion on the computation of measures of uncertainty and confidence intervals, and discuss the use of statistical tests. We argue that, as in many other areas of science, there is an over-reliance on 95\% confidence levels, and misunderstandings of the meaning of significance. We also observe a lack of precision in the reporting of measures of uncertainty in many studies, especially when using $p$-values and even more so with \emph{star} measures. The paper also stresses the importance of considering behavioural or policy significance in addition to statistical significance. Finally, we stress a number of points that are specific to choice modelling and which require special attention, notably in relation to derived measures such as willingness-to-pay, the treatment of random heterogeneity, and the use of repeated choice data.

Suggested Citation

  • Stephane Hess & Andrew Daly & Michiel Bliemer & Angelo Guevara & Ricardo Daziano & Thijs Dekker, 2025. "Statistical significance in choice modelling: computation, usage and reporting," Papers 2506.05996, arXiv.org, revised Mar 2026.
    • Handle: RePEc:arx:papers:2506.05996
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2506.05996
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Krinsky, Itzhak & Robb, A Leslie, 1990. "On Approximating the Statistical Properties of Elasticities: A Correction," The Review of Economics and Statistics, MIT Press, vol. 72(1), pages 189-190, February.
    2. Mokhtarian, Patricia L., 2016. "Discrete choice models’ ρ2: A reintroduction to an old friend," Journal of choice modelling, Elsevier, vol. 21(C), pages 60-65.
    3. Ronald L. Wasserstein & Nicole A. Lazar, 2016. "The ASA's Statement on p -Values: Context, Process, and Purpose," The American Statistician, Taylor & Francis Journals, vol. 70(2), pages 129-133, May.
    4. C. Angelo Guevara, 2024. "Endogeneity in discrete choice models," Chapters, in: Stephane Hess & Andrew Daly (ed.), Handbook of Choice Modelling, chapter 23, pages 668-692, Edward Elgar Publishing.
    5. Ronald L. Wasserstein & Allen L. Schirm & Nicole A. Lazar, 2019. "Moving to a World Beyond “p," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 1-19, March.
    6. Valentin Amrhein & Sander Greenland, 2018. "Remove, rather than redefine, statistical significance," Nature Human Behaviour, Nature, vol. 2(1), pages 4-4, January.
    7. Hess, Stephane & Palma, David, 2019. "Apollo: A flexible, powerful and customisable freeware package for choice model estimation and application," Journal of choice modelling, Elsevier, vol. 32(C), pages 1-1.
    8. Esther Bekker-Grob & Bas Donkers & Marcel Jonker & Elly Stolk, 2015. "Sample Size Requirements for Discrete-Choice Experiments in Healthcare: a Practical Guide," The Patient: Patient-Centered Outcomes Research, Springer;International Academy of Health Preference Research, vol. 8(5), pages 373-384, October.
    9. Chiou, Lesley & Walker, Joan L., 2007. "Masking identification of discrete choice models under simulation methods," Journal of Econometrics, Elsevier, vol. 141(2), pages 683-703, December.
    10. King, Gary & Roberts, Margaret E., 2015. "How Robust Standard Errors Expose Methodological Problems They Do Not Fix, and What to Do About It," Political Analysis, Cambridge University Press, vol. 23(2), pages 159-179, April.
    11. Stephane Hess & David Bunch & Andrew Daly, 2025. "Get me out of this hole: a profile likelihood approach to identifying and avoiding inferior local optima in choice models," Papers 2506.02722, arXiv.org.
    12. Daniel J. Benjamin & James O. Berger & Magnus Johannesson & Brian A. Nosek & E.-J. Wagenmakers & Richard Berk & Kenneth A. Bollen & Björn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Chr, 2018. "Redefine statistical significance," Nature Human Behaviour, Nature, vol. 2(1), pages 6-10, January.
      • Daniel Benjamin & James Berger & Magnus Johannesson & Brian Nosek & E. Wagenmakers & Richard Berk & Kenneth Bollen & Bjorn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Christopher Chambe, 2017. "Redefine Statistical Significance," Artefactual Field Experiments 00612, The Field Experiments Website.
    13. Moshe Ben-Akiva & Joffre Swait, 1986. "The Akaike Likelihood Ratio Index," Transportation Science, INFORMS, vol. 20(2), pages 133-136, May.
    14. Daly, Andrew & Hess, Stephane & de Jong, Gerard, 2012. "Calculating errors for measures derived from choice modelling estimates," Transportation Research Part B: Methodological, Elsevier, vol. 46(2), pages 333-341.
    15. Guevara, C. Angelo, 2018. "Overidentification tests for the exogeneity of instruments in discrete choice models," Transportation Research Part B: Methodological, Elsevier, vol. 114(C), pages 241-253.
    16. Chiara Calastri & Romain Crastes dit Sourd & Stephane Hess, 2020. "We want it all: experiences from a survey seeking to capture social network structures, lifetime events and short-term travel and activity planning," Transportation, Springer, vol. 47(1), pages 175-201, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. John Buckell & David A Hensher & Stephane Hess, 2021. "Kicking the habit is hard: A hybrid choice model investigation into the role of addiction in smoking behavior," Health Economics, John Wiley & Sons, Ltd., vol. 30(1), pages 3-19, January.
    2. Kelter, Riko, 2022. "Power analysis and type I and type II error rates of Bayesian nonparametric two-sample tests for location-shifts based on the Bayes factor under Cauchy priors," Computational Statistics & Data Analysis, Elsevier, vol. 165(C).
    3. Isler, Cassiano Augusto & Blumenfeld, Marcelo & Caldeira, Gabriel Pereira & Roberts, Clive, 2024. "Long-Distance railway mode choice in Brazil: Evidence from a discrete choice experiment," Research in Transportation Economics, Elsevier, vol. 104(C).
    4. Stephane Hess & David Bunch & Andrew Daly, 2025. "Get me out of this hole: a profile likelihood approach to identifying and avoiding inferior local optima in choice models," Papers 2506.02722, arXiv.org.
    5. Joachim Marti & John Buckell & Johanna Catherine Maclean & Jody L. Sindelar, 2016. "To ‘Vape’ or Smoke? A Discrete Choice Experiment Among U.S. Adult Smokers," NBER Working Papers 22079, National Bureau of Economic Research, Inc.
    6. Joachim Marti & John Buckell & Johanna Catherine Maclean & Jody Sindelar, 2019. "To “Vape” Or Smoke? Experimental Evidence On Adult Smokers," Economic Inquiry, Western Economic Association International, vol. 57(1), pages 705-725, January.
    7. Gunter, Ulrich & Önder, Irem & Smeral, Egon, 2019. "Scientific value of econometric tourism demand studies," Annals of Tourism Research, Elsevier, vol. 78(C), pages 1-1.
    8. Hancock, Thomas O. & Hess, Stephane & Choudhury, Charisma F. & Tsoleridis, Panagiotis, 2024. "Decision field theory: An extension for real-world settings," Journal of choice modelling, Elsevier, vol. 52(C).
    9. Daly, Andrew & Hess, Stephane & Ortúzar, Juan de Dios, 2023. "Estimating willingness-to-pay from discrete choice models: Setting the record straight," Transportation Research Part A: Policy and Practice, Elsevier, vol. 176(C).
    10. David A. J. Meester & Stephane Hess & John Buckell & Thomas O. Hancock, 2023. "Can decision field theory enhance our understanding of health‐based choices? Evidence from risky health behaviors," Health Economics, John Wiley & Sons, Ltd., vol. 32(8), pages 1710-1732, August.
    11. Curtis Rollins, 2023. "Investigating cost non‐attendance as a driver of inflated welfare estimates in mixed‐logit models," Journal of Agricultural Economics, Wiley Blackwell, vol. 74(3), pages 921-934, September.
    12. Erik W. van Zwet & Eric A. Cator, 2021. "The significance filter, the winner's curse and the need to shrink," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 75(4), pages 437-452, November.
    13. John Buckell & Alice Wreford & Matthew Quaife & Thomas O. Hancock, 2025. "A break from the norm? Parametric representations of preference heterogeneity for discrete choice models in health," Papers 2506.14099, arXiv.org.
    14. Fukś Maksymilian & Wiejaczka Łukasz, 2025. "Climatic Determinants of Changes in the Ice Regime of Carpathian Rivers," Quaestiones Geographicae, Sciendo, vol. 44(1), pages 131-143.
    15. Scaccia, Luisa & Marcucci, Edoardo & Gatta, Valerio, 2023. "Prediction and confidence intervals of willingness-to-pay for mixed logit models," Transportation Research Part B: Methodological, Elsevier, vol. 167(C), pages 54-78.
    16. Chorus, Caspar & van Cranenburgh, Sander & Daniel, Aemiro Melkamu & Sandorf, Erlend Dancke & Sobhani, Anae & Szép, Teodóra, 2021. "Obfuscation maximization-based decision-making: Theory, methodology and first empirical evidence," Mathematical Social Sciences, Elsevier, vol. 109(C), pages 28-44.
    17. Nikita Arora & Matthew Quaife & Kara Hanson & Mylene Lagarde & Dorka Woldesenbet & Abiy Seifu & Romain Crastes dit Sourd, 2022. "Discrete choice analysis of health worker job preferences in Ethiopia: Separating attribute non‐attendance from taste heterogeneity," Health Economics, John Wiley & Sons, Ltd., vol. 31(5), pages 806-819, May.
    18. Dr. Olivier Gatete, 2025. "Statistics: The Art and Science of Learning from Data," International Journal of Research and Innovation in Social Science, International Journal of Research and Innovation in Social Science (IJRISS), vol. 9(7), pages 285-301, July.
    19. John Buckell & Vrinda Vasavada & Sarah Wordsworth & Dean A. Regier & Matthew Quaife, 2022. "Utility maximization versus regret minimization in health choice behavior: Evidence from four datasets," Health Economics, John Wiley & Sons, Ltd., vol. 31(2), pages 363-381, February.
    20. Robbennolt, Dale & Beliveau, Anna & Bhat, Chandra R., 2025. "An investigation of physical participation dissonance and virtual activity participation in the United States," Transportation Research Part A: Policy and Practice, Elsevier, vol. 201(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2506.05996. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.