IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2506.05996.html
   My bibliography  Save this paper

Statistical significance in choice modelling: computation, usage and reporting

Author

Listed:
  • Stephane Hess
  • Andrew Daly
  • Michiel Bliemer
  • Angelo Guevara
  • Ricardo Daziano
  • Thijs Dekker

Abstract

This paper offers a commentary on the use of notions of statistical significance in choice modelling. We argue that, as in many other areas of science, there is an over-reliance on 95% confidence levels, and misunderstandings of the meaning of significance. We also observe a lack of precision in the reporting of measures of uncertainty in many studies, especially when using p-values and even more so with star measures. The paper provides a precise discussion on the computation of measures of uncertainty and confidence intervals, discusses the use of statistical tests, and also stresses the importance of considering behavioural or policy significance in addition to statistical significance.

Suggested Citation

  • Stephane Hess & Andrew Daly & Michiel Bliemer & Angelo Guevara & Ricardo Daziano & Thijs Dekker, 2025. "Statistical significance in choice modelling: computation, usage and reporting," Papers 2506.05996, arXiv.org, revised Jun 2025.
    • Handle: RePEc:arx:papers:2506.05996
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2506.05996
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Mokhtarian, Patricia L., 2016. "Discrete choice models’ ρ2: A reintroduction to an old friend," Journal of choice modelling, Elsevier, vol. 21(C), pages 60-65.
    2. King, Gary & Roberts, Margaret E., 2015. "How Robust Standard Errors Expose Methodological Problems They Do Not Fix, and What to Do About It," Political Analysis, Cambridge University Press, vol. 23(2), pages 159-179, April.
    3. C. Angelo Guevara, 2024. "Endogeneity in discrete choice models," Chapters, in: Stephane Hess & Andrew Daly (ed.), Handbook of Choice Modelling, chapter 23, pages 668-692, Edward Elgar Publishing.
    4. Ronald L. Wasserstein & Allen L. Schirm & Nicole A. Lazar, 2019. "Moving to a World Beyond “p," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 1-19, March.
    5. Valentin Amrhein & Sander Greenland, 2018. "Remove, rather than redefine, statistical significance," Nature Human Behaviour, Nature, vol. 2(1), pages 4-4, January.
    6. Chiou, Lesley & Walker, Joan L., 2007. "Masking identification of discrete choice models under simulation methods," Journal of Econometrics, Elsevier, vol. 141(2), pages 683-703, December.
    7. Daniel J. Benjamin & James O. Berger & Magnus Johannesson & Brian A. Nosek & E.-J. Wagenmakers & Richard Berk & Kenneth A. Bollen & Björn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Chr, 2018. "Redefine statistical significance," Nature Human Behaviour, Nature, vol. 2(1), pages 6-10, January.
      • Daniel Benjamin & James Berger & Magnus Johannesson & Brian Nosek & E. Wagenmakers & Richard Berk & Kenneth Bollen & Bjorn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Christopher Chambe, 2017. "Redefine Statistical Significance," Artefactual Field Experiments 00612, The Field Experiments Website.
    8. Chiara Calastri & Romain Crastes dit Sourd & Stephane Hess, 2020. "We want it all: experiences from a survey seeking to capture social network structures, lifetime events and short-term travel and activity planning," Transportation, Springer, vol. 47(1), pages 175-201, February.
    9. Krinsky, Itzhak & Robb, A Leslie, 1990. "On Approximating the Statistical Properties of Elasticities: A Correction," The Review of Economics and Statistics, MIT Press, vol. 72(1), pages 189-190, February.
    10. Ronald L. Wasserstein & Nicole A. Lazar, 2016. "The ASA's Statement on p -Values: Context, Process, and Purpose," The American Statistician, Taylor & Francis Journals, vol. 70(2), pages 129-133, May.
    11. Hess, Stephane & Palma, David, 2019. "Apollo: A flexible, powerful and customisable freeware package for choice model estimation and application," Journal of choice modelling, Elsevier, vol. 32(C), pages 1-1.
    12. Stephane Hess & David Bunch & Andrew Daly, 2025. "Get me out of this hole: a profile likelihood approach to identifying and avoiding inferior local optima in choice models," Papers 2506.02722, arXiv.org.
    13. Moshe Ben-Akiva & Joffre Swait, 1986. "The Akaike Likelihood Ratio Index," Transportation Science, INFORMS, vol. 20(2), pages 133-136, May.
    14. Daly, Andrew & Hess, Stephane & de Jong, Gerard, 2012. "Calculating errors for measures derived from choice modelling estimates," Transportation Research Part B: Methodological, Elsevier, vol. 46(2), pages 333-341.
    15. Guevara, C. Angelo, 2018. "Overidentification tests for the exogeneity of instruments in discrete choice models," Transportation Research Part B: Methodological, Elsevier, vol. 114(C), pages 241-253.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kelter, Riko, 2022. "Power analysis and type I and type II error rates of Bayesian nonparametric two-sample tests for location-shifts based on the Bayes factor under Cauchy priors," Computational Statistics & Data Analysis, Elsevier, vol. 165(C).
    2. Isler, Cassiano Augusto & Blumenfeld, Marcelo & Caldeira, Gabriel Pereira & Roberts, Clive, 2024. "Long-Distance railway mode choice in Brazil: Evidence from a discrete choice experiment," Research in Transportation Economics, Elsevier, vol. 104(C).
    3. Stephane Hess & David Bunch & Andrew Daly, 2025. "Get me out of this hole: a profile likelihood approach to identifying and avoiding inferior local optima in choice models," Papers 2506.02722, arXiv.org.
    4. John Buckell & David A Hensher & Stephane Hess, 2021. "Kicking the habit is hard: A hybrid choice model investigation into the role of addiction in smoking behavior," Health Economics, John Wiley & Sons, Ltd., vol. 30(1), pages 3-19, January.
    5. Gunter, Ulrich & Önder, Irem & Smeral, Egon, 2019. "Scientific value of econometric tourism demand studies," Annals of Tourism Research, Elsevier, vol. 78(C), pages 1-1.
    6. Hancock, Thomas O. & Hess, Stephane & Choudhury, Charisma F. & Tsoleridis, Panagiotis, 2024. "Decision field theory: An extension for real-world settings," Journal of choice modelling, Elsevier, vol. 52(C).
    7. Daly, Andrew & Hess, Stephane & Ortúzar, Juan de Dios, 2023. "Estimating willingness-to-pay from discrete choice models: Setting the record straight," Transportation Research Part A: Policy and Practice, Elsevier, vol. 176(C).
    8. David A. J. Meester & Stephane Hess & John Buckell & Thomas O. Hancock, 2023. "Can decision field theory enhance our understanding of health‐based choices? Evidence from risky health behaviors," Health Economics, John Wiley & Sons, Ltd., vol. 32(8), pages 1710-1732, August.
    9. Curtis Rollins, 2023. "Investigating cost non‐attendance as a driver of inflated welfare estimates in mixed‐logit models," Journal of Agricultural Economics, Wiley Blackwell, vol. 74(3), pages 921-934, September.
    10. Erik W. van Zwet & Eric A. Cator, 2021. "The significance filter, the winner's curse and the need to shrink," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 75(4), pages 437-452, November.
    11. Fukś Maksymilian & Wiejaczka Łukasz, 2025. "Climatic Determinants of Changes in the Ice Regime of Carpathian Rivers," Quaestiones Geographicae, Sciendo, vol. 44(1), pages 131-143.
    12. Scaccia, Luisa & Marcucci, Edoardo & Gatta, Valerio, 2023. "Prediction and confidence intervals of willingness-to-pay for mixed logit models," Transportation Research Part B: Methodological, Elsevier, vol. 167(C), pages 54-78.
    13. Chorus, Caspar & van Cranenburgh, Sander & Daniel, Aemiro Melkamu & Sandorf, Erlend Dancke & Sobhani, Anae & Szép, Teodóra, 2021. "Obfuscation maximization-based decision-making: Theory, methodology and first empirical evidence," Mathematical Social Sciences, Elsevier, vol. 109(C), pages 28-44.
    14. Nikita Arora & Matthew Quaife & Kara Hanson & Mylene Lagarde & Dorka Woldesenbet & Abiy Seifu & Romain Crastes dit Sourd, 2022. "Discrete choice analysis of health worker job preferences in Ethiopia: Separating attribute non‐attendance from taste heterogeneity," Health Economics, John Wiley & Sons, Ltd., vol. 31(5), pages 806-819, May.
    15. Stephane Hess & Sander van Cranenburgh, 2025. "Combine and conquer: model averaging for out-of-distribution forecasting," Papers 2506.03693, arXiv.org.
    16. Michaelides, Michael, 2021. "Large sample size bias in empirical finance," Finance Research Letters, Elsevier, vol. 41(C).
    17. Carson, Richard T. & Czajkowski, Mikołaj, 2019. "A new baseline model for estimating willingness to pay from discrete choice models," Journal of Environmental Economics and Management, Elsevier, vol. 95(C), pages 57-61.
    18. Tsoleridis, Panagiotis & Choudhury, Charisma F. & Hess, Stephane, 2022. "Utilising activity space concepts to sampling of alternatives for mode and destination choice modelling of discretionary activities," Journal of choice modelling, Elsevier, vol. 42(C).
    19. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    20. Welling, Malte & Jacobsen, Jette Bredahl & Olsen, Søren Bøye & Lundhede, Thomas, 2025. "Does quantity matter for distance decay? Evidence from two choice experiments on urban green," Resource and Energy Economics, Elsevier, vol. 81(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2506.05996. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.