IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2504.10636.html
   My bibliography  Save this paper

Who is More Bayesian: Humans or ChatGPT?

Author

Listed:
  • Tianshi Mu
  • Pranjal Rawat
  • John Rust
  • Chengjun Zhang
  • Qixuan Zhong

Abstract

We compare the performance of human and artificially intelligent (AI) decision makers in simple binary classification tasks where the optimal decision rule is given by Bayes Rule. We reanalyze choices of human subjects gathered from laboratory experiments conducted by El-Gamal and Grether and Holt and Smith. We confirm that while overall, Bayes Rule represents the single best model for predicting human choices, subjects are heterogeneous and a significant share of them make suboptimal choices that reflect judgement biases described by Kahneman and Tversky that include the ``representativeness heuristic'' (excessive weight on the evidence from the sample relative to the prior) and ``conservatism'' (excessive weight on the prior relative to the sample). We compare the performance of AI subjects gathered from recent versions of large language models (LLMs) including several versions of ChatGPT. These general-purpose generative AI chatbots are not specifically trained to do well in narrow decision making tasks, but are trained instead as ``language predictors'' using a large corpus of textual data from the web. We show that ChatGPT is also subject to biases that result in suboptimal decisions. However we document a rapid evolution in the performance of ChatGPT from sub-human performance for early versions (ChatGPT 3.5) to superhuman and nearly perfect Bayesian classifications in the latest versions (ChatGPT 4o).

Suggested Citation

  • Tianshi Mu & Pranjal Rawat & John Rust & Chengjun Zhang & Qixuan Zhong, 2025. "Who is More Bayesian: Humans or ChatGPT?," Papers 2504.10636, arXiv.org.
    • Handle: RePEc:arx:papers:2504.10636
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2504.10636
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Yiting Chen & Tracy Xiao Liu & You Shan & Songfa Zhong, 2023. "The emergence of economic rationality of GPT," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 120(51), pages 2316205120-, December.
    2. Heckman, James & Singer, Burton, 1984. "A Method for Minimizing the Impact of Distributional Assumptions in Econometric Models for Duration Data," Econometrica, Econometric Society, vol. 52(2), pages 271-320, March.
    3. Grether, David M, 1978. "Recent Psychological Studies of Behavior under Uncertainty," American Economic Review, American Economic Association, vol. 68(2), pages 70-74, May.
    4. Holt, Charles A. & Smith, Angela M., 2009. "An update on Bayesian updating," Journal of Economic Behavior & Organization, Elsevier, vol. 69(2), pages 125-134, February.
    5. Ali Goli & Amandeep Singh, 2024. "Frontiers: Can Large Language Models Capture Human Preferences?," Marketing Science, INFORMS, vol. 43(4), pages 709-722, July.
    6. Kühl, Niklas & Goutier, Marc & Baier, Lucas & Wolff, Clemens & Martin, Dominik, 2022. "Human vs. supervised machine learning: Who learns patterns faster?," Publications of Darmstadt Technical University, Institute for Business Studies (BWL) 135657, Darmstadt Technical University, Department of Business Administration, Economics and Law, Institute for Business Studies (BWL).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gergely Hajdu & Balázs Krusper, 2023. "Choice-induced Sticky Learning," Department of Economics Working Papers wuwp349, Vienna University of Economics and Business, Department of Economics.
    2. Kai Barron, 2021. "Belief updating: does the ‘good-news, bad-news’ asymmetry extend to purely financial domains?," Experimental Economics, Springer;Economic Science Association, vol. 24(1), pages 31-58, March.
    3. Daniel J. Benjamin, 2018. "Errors in Probabilistic Reasoning and Judgment Biases," NBER Working Papers 25200, National Bureau of Economic Research, Inc.
    4. Patrick Arni & Rafael Lalive & Jan C. Van Ours, 2013. "How Effective Are Unemployment Benefit Sanctions? Looking Beyond Unemployment Exit," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 28(7), pages 1153-1178, November.
    5. Kemptner, Daniel & Tolan, Songül, 2018. "The role of time preferences in educational decision making," Economics of Education Review, Elsevier, vol. 67(C), pages 25-39.
    6. Christelis, Dimitris & Dobrescu, Loretti I. & Motta, Alberto, 2020. "Early life conditions and financial risk-taking in older age," The Journal of the Economics of Ageing, Elsevier, vol. 17(C).
    7. Michelle Sheran Sylvester, 2007. "The Career and Family Choices of Women: A Dynamic Analysis of Labor Force Participation, Schooling, Marriage and Fertility Decisions," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 10(3), pages 367-399, July.
    8. Insoo Cho & Peter F. Orazem, 2021. "How endogenous risk preferences and sample selection affect analysis of firm survival," Small Business Economics, Springer, vol. 56(4), pages 1309-1332, April.
    9. Ali Palali & Jan C. Van ours, 2017. "Love Conquers all but Nicotine: Spousal Peer Effects on the Decision to Quit Smoking," Health Economics, John Wiley & Sons, Ltd., vol. 26(12), pages 1710-1727, December.
    10. Steven M. Lewis & Adrian E. Raftery, 1999. "Bayesian Analysis of Event History Models with Unobserved Heterogeneity via Markov Chain Monte Carlo," Sociological Methods & Research, , vol. 28(1), pages 35-60, August.
    11. Parent, Daniel, 1999. "Wages and Mobility: The Impact of Employer-Provided Training," Journal of Labor Economics, University of Chicago Press, vol. 17(2), pages 298-317, April.
    12. Steven T Berry & Giovanni Compiani, 2023. "An Instrumental Variable Approach to Dynamic Models," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 90(4), pages 1724-1758.
    13. Francesco Devicienti & Valentina Gualtieri & Mariacristina Rossi, 2014. "The Persistence Of Income Poverty And Lifestyle Deprivation: Evidence From Italy," Bulletin of Economic Research, Wiley Blackwell, vol. 66(3), pages 246-278, July.
    14. Elena Casquel & Antoni Cunyat, "undated". "The Welfare Cost of Business Cycles in an Economy with Nonclearing Markets," Working Papers 2005-19, FEDEA.
    15. Brigitte Dormont & Denis Fougère & Ana Prieto, 2001. "L'effet de l'allocation unique dégressive sur la reprise d'emploi," Économie et Statistique, Programme National Persée, vol. 343(1), pages 3-28.
    16. Cockx, Bart & Robin, Stéphane R. & Goebel, Christian, 2006. "Income Support Policies for Part-Time Workers: A Stepping-Stone to Regular Jobs? An Application to Young Long-Term Unemployed Women in Belgium," IZA Discussion Papers 2432, Institute of Labor Economics (IZA).
    17. Hoffman, Mitchell & Burks, Stephen V., 2017. "Worker Overconfidence: Field Evidence and Implications for Employee Turnover and Returns from Training," IZA Discussion Papers 10794, Institute of Labor Economics (IZA).
    18. Pan, Wei & Louis, Thomas A., 1999. "Two semi-parametric empirical Bayes estimators," Computational Statistics & Data Analysis, Elsevier, vol. 30(2), pages 185-196, April.
    19. Fabien Postel-Vinay & Hélène Turon, 2007. "The Public Pay Gap in Britain: Small Differences That (Don't?) Matter," Economic Journal, Royal Economic Society, vol. 117(523), pages 1460-1503, October.
    20. John Cawley & James Heckman & Edward Vytlacil, 1998. "Meritocracy in America: An Examination of Wages Within and Across Occupations," NBER Working Papers 6446, National Bureau of Economic Research, Inc.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2504.10636. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.