Who is More Bayesian: Humans or ChatGPT?

We compare human and artificially intelligent (AI) subjects in classification tasks where the optimal decision rule is given by Bayes’ Rule. Experimental studies reach mixed conclusions about whether human beliefs and decisions accord with Bayes’ Rule. We reanalyze land- mark experiments using a new model of decision making and show that decisions can be nearly optimal even when beliefs are not Bayesian. Using an objective measure of “decision efficiency,” we find that humans are 96% efficient despite the fact that only a minority have Bayesian beliefs. We replicate these same experiments using three generations of ChatGPT as subjects. Using the reasoning provided by GPT responses to understand its “thought process,” we find that GPT-3.5 ignores the prior and is only 75% efficient, whereas GPT-4 and GPT-4o use Bayes’ Rule and are 93% and 99% efficient, respectively. Most errors by GPT-4 and GPT-4o are algebraic mistakes in computing the posterior, but GPT-4o is far less error-prone. GPT performance increased from sub-human to super-human in just 3 years. By version 4o, its beliefs and decision making had become nearly perfectly Bayesian.