IDEAS home Printed from https://ideas.repec.org/a/eee/soceco/v73y2018icp86-92.html
   My bibliography  Save this article

Assessing the forecast performance of models of choice

Author

Listed:
  • Stahl, Dale O.

Abstract

We often want to predict human behavior. It is well-known that the model that fits in-sample data best is not necessarily the model that forecasts (i.e. predicts out-of-sample) best, but we lack guidance on how to select a model for the purpose of forecasting. We illustrate the general issues and methods with the case of Rank-Dependent Expected Utility versus Expected Utility, using laboratory data and simulations. We find that poor forecasting performance is a likely outcome for typical laboratory sample sizes due to over-fitting. Finally we derive a decision-theory-based rule for selecting the best model for forecasting depending on the sample size.

Suggested Citation

  • Stahl, Dale O., 2018. "Assessing the forecast performance of models of choice," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 73(C), pages 86-92.
  • Handle: RePEc:eee:soceco:v:73:y:2018:i:c:p:86-92
    DOI: 10.1016/j.socec.2018.02.006
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S2214804318300739
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.socec.2018.02.006?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Harrison, Glenn W, 1989. "Theory and Misbehavior of First-Price Auctions," American Economic Review, American Economic Association, vol. 79(4), pages 749-762, September.
    2. Helga Fehr-Duda & Thomas Epper, 2012. "Probability and Risk: Foundations and Economic Implications of Probability-Dependent Risk Preferences," Annual Review of Economics, Annual Reviews, vol. 4(1), pages 567-593, July.
    3. Quiggin, John, 1982. "A theory of anticipated utility," Journal of Economic Behavior & Organization, Elsevier, vol. 3(4), pages 323-343, December.
    4. Michael P. Clements & David I. Harvey, 2010. "Forecast encompassing tests and probability forecasts," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(6), pages 1028-1062.
    5. Wilcox, Nathaniel T., 2011. "'Stochastically more risk averse:' A contextual theory of stochastic discrete choice under risk," Journal of Econometrics, Elsevier, vol. 162(1), pages 89-104, May.
    6. Glenn Harrison & J. Swarthout, 2014. "Experimental payment protocols and the Bipolar Behaviorist," Theory and Decision, Springer, vol. 77(3), pages 423-438, October.
    7. John D. Hey & Chris Orme, 2018. "Investigating Generalizations Of Expected Utility Theory Using Experimental Data," World Scientific Book Chapters, in: Experiments in Economics Decision Making and Markets, chapter 3, pages 63-98, World Scientific Publishing Co. Pte. Ltd..
    8. Goffe, William L. & Ferrier, Gary D. & Rogers, John, 1994. "Global optimization of statistical functions with simulated annealing," Journal of Econometrics, Elsevier, vol. 60(1-2), pages 65-99.
    9. Harrison, Glenn W, 1992. "Theory and Misbehavior of First-Price Auctions: Reply," American Economic Review, American Economic Association, vol. 82(5), pages 1426-1443, December.
    10. Lattimore, Pamela K. & Baker, Joanna R. & Witte, Ann D., 1992. "The influence of probability on risky choice: A parametric examination," Journal of Economic Behavior & Organization, Elsevier, vol. 17(3), pages 377-400, May.
    11. Glenn Harrison & E. Rutström, 2009. "Expected utility theory and prospect theory: one wedding and a decent funeral," Experimental Economics, Springer;Economic Science Association, vol. 12(2), pages 133-158, June.
    12. Adrian Bruhin & Helga Fehr-Duda & Thomas Epper, 2010. "Risk and Rationality: Uncovering Heterogeneity in Probability Distortion," Econometrica, Econometric Society, vol. 78(4), pages 1375-1412, July.
    13. Drazen Prelec, 1998. "The Probability Weighting Function," Econometrica, Econometric Society, vol. 66(3), pages 497-528, May.
    14. Pamela K. Lattimore & Joanna R. Baker & A. Dryden Witte, 1992. "The Influence Of Probability on Risky Choice: A parametric Examination," NBER Technical Working Papers 0081, National Bureau of Economic Research, Inc.
    15. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Zachary Breig, 2020. "Prediction and Model Selection in Experiments," The Economic Record, The Economic Society of Australia, vol. 96(313), pages 153-176, June.
    2. Dale O. Stahl, 2019. "A Bayesian Method for Characterizing Population Heterogeneity," Games, MDPI, vol. 10(4), pages 1-12, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Galarza, Francisco, 2009. "Choices under Risk in Rural Peru," MPRA Paper 17708, University Library of Munich, Germany.
    2. Kpegli, Yao Thibaut & Corgnet, Brice & Zylbersztejn, Adam, 2023. "All at once! A comprehensive and tractable semi-parametric method to elicit prospect theory components," Journal of Mathematical Economics, Elsevier, vol. 104(C).
    3. Dale O. Stahl, 2019. "A Bayesian Method for Characterizing Population Heterogeneity," Games, MDPI, vol. 10(4), pages 1-12, October.
    4. Ferdinand M. Vieider & Clara Villegas-Palacio & Peter Martinsson & Milagros Mejía, 2016. "Risk Taking For Oneself And Others: A Structural Model Approach," Economic Inquiry, Western Economic Association International, vol. 54(2), pages 879-894, April.
    5. Vincent Laferrière & David Staubli & Christian Thöni, 2023. "Explaining Excess Entry in Winner-Take-All Markets," Management Science, INFORMS, vol. 69(2), pages 1050-1069, February.
    6. Glenn W. Harrison & J. Todd Swarthout, 2016. "Cumulative Prospect Theory in the Laboratory: A Reconsideration," Experimental Economics Center Working Paper Series 2016-04, Experimental Economics Center, Andrew Young School of Policy Studies, Georgia State University.
    7. Luís Santos-Pinto & Adrian Bruhin & José Mata & Thomas Åstebro, 2015. "Detecting heterogeneous risk attitudes with mixed gambles," Theory and Decision, Springer, vol. 79(4), pages 573-600, December.
    8. Jakusch, Sven Thorsten & Meyer, Steffen & Hackethal, Andreas, 2019. "Taming models of prospect theory in the wild? Estimation of Vlcek and Hens (2011)," SAFE Working Paper Series 146, Leibniz Institute for Financial Research SAFE, revised 2019.
    9. Eyal Baharad & Doron Kliger, 2013. "Market failure in light of non-expected utility," Theory and Decision, Springer, vol. 75(4), pages 599-619, October.
    10. Fehr-Duda, Helga & Epper, Thomas & Bruhin, Adrian & Schubert, Renate, 2011. "Risk and rationality: The effects of mood and decision rules on probability weighting," Journal of Economic Behavior & Organization, Elsevier, vol. 78(1-2), pages 14-24, April.
    11. Glenn W. Harrison & Andre Hofmeyr & Don Ross & J. Todd Swarthout, 2018. "Risk Preferences, Time Preferences, and Smoking Behavior," Southern Economic Journal, John Wiley & Sons, vol. 85(2), pages 313-348, October.
    12. Cheremukhin, Anton & Popova, Anna & Tutino, Antonella, 2015. "A theory of discrete choice with information costs," Journal of Economic Behavior & Organization, Elsevier, vol. 113(C), pages 34-50.
    13. Adrian Bruhin & Maha Manai & Luís Santos-Pinto, 2022. "Risk and rationality: The relative importance of probability weighting and choice set dependence," Journal of Risk and Uncertainty, Springer, vol. 65(2), pages 139-184, October.
    14. Adrian Bruhin & Maha Manai & Luis Santos-Pinto, 2019. "Risk and Rationality:The Relative Importance of Probability Weighting and Choice Set Dependence," Cahiers de Recherches Economiques du Département d'économie 19.01new, Université de Lausanne, Faculté des HEC, Département d’économie.
    15. Levon Barseghyan & Francesca Molinari & Ted O'Donoghue & Joshua C. Teitelbaum, 2013. "The Nature of Risk Preferences: Evidence from Insurance Choices," American Economic Review, American Economic Association, vol. 103(6), pages 2499-2529, October.
    16. Nathaniel T. Wilcox, 2023. "Unusual Estimates of Probability Weighting Functions," Research in Experimental Economics, in: Models of Risk Preferences: Descriptive and Normative Challenges, volume 22, pages 69-106, Emerald Group Publishing Limited.
    17. Thomas Epper & Helga Fehr-Duda & Adrian Bruhin, 2011. "Viewing the future through a warped lens: Why uncertainty generates hyperbolic discounting," Journal of Risk and Uncertainty, Springer, vol. 43(3), pages 169-203, December.
    18. Kechagia, Varvara & Drichoutis, Andreas C., 2017. "The effect of olfactory sensory cues on willingness to pay and choice under risk," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 70(C), pages 33-46.
    19. Andreas C. Drichoutis & Varvara Kechagia, 2016. "The effect of olfactory sensory cues on economic decision making," Working Papers 2016-4, Agricultural University of Athens, Department Of Agricultural Economics.
    20. Adrian Bruhin & Maha Manai & Luis Santos-Pinto, 2018. "Risk and Rationality:The Relative Importance of Probability Weighting and Choice Set Dependence," Cahiers de Recherches Economiques du Département d'économie 18.04, Université de Lausanne, Faculté des HEC, Département d’économie.

    More about this item

    Keywords

    Forecast performance; Over-fitting; Cross-validation; Lottery choice;
    All these keywords.

    JEL classification:

    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • D81 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Criteria for Decision-Making under Risk and Uncertainty

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:soceco:v:73:y:2018:i:c:p:86-92. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/inca/620175 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.