Training Novices to Think, or Giving Them LLMs? Evidence from an RCT

We study whether LLMs and cognitive skills influence the evaluation of novices’ intellectual tasks, when these evaluations depend on standard and well-defined performance criteria. This is a common case (junior employees, students’ exams) making our question relevant. In a 2×2 randomized control trial, we assign 1,053 first-year undergraduates in economics, management and finance to one of four conditions: a training on causal reasoning (as a relevant cognitive skill for intellectual tasks), ChatGPT Edu access (GPT-4o), both, or neither. We provide a real-world merchandising business problem and ask experts to evaluate the participants’ solutions according to standard metrics in marketing. We find that causal reasoning changes how students think, leading to mechanism-based solutions and a greater diversity of ideas. However, causal reasoning does not improve evaluations, unlike ChatGPT. Greater textual coherence and a higher number of ideas explain about half of the ChatGPT effect, suggesting that, for novices facing well-defined problems, LLMs produce solutions that resemble expert recommendations not only in textual quality, but also in content. We also find that, when combined with causal training, ChatGPT increases the use of coherent logic, falsification logic, and the attempt to understand why (mechanisms). Overall, we find that human causal reasoning produces effects independently of LLMs, particularly on the diversity of ideas, and therefore it is worth cultivating causal reasoning in a world in which students use LLMs. Probably less appreciated is that the value of diverse ideas also requires an evaluation system that demands diversity rather than standard solutions.