Influence of surprise on reinforcement learning in younger and older adults

Surprise is a key component of many learning experiences, and yet its precise computational role, and how it changes with age, remain debated. One major challenge is that surprise often occurs jointly with other variables, such as uncertainty and outcome probability. To assess how humans learn from surprising events, and whether aging affects this process, we studied choices while participants learned from bandits with either Gaussian or bi-modal outcome distributions, which decoupled outcome probability, uncertainty, and surprise. A total of 102 participants (51 older, aged 50–73; 51 younger, 19–30 years) chose between three bandits, one of which had a bimodal outcome distribution. Behavioral analyses showed that both age-groups learned the average of the bimodal bandit less well. A trial-by-trial analysis indicated that participants performed choice reversals immediately following large absolute prediction errors, consistent with heightened sensitivity to surprise. This effect was stronger in older adults. Computational models indicated that learning rates in younger as well as older adults were influenced by surprise, rather than uncertainty, but also suggested large interindividual variability in the process underlying learning in our task. Our work bridges between behavioral economics research that has focused on how outcomes with low probability affect choice in older adults, and reinforcement learning work that has investigated age differences in the effects of uncertainty and suggests that older adults overly adapt to surprising events, even when accounting for probability and uncertainty effects.Author summary: Learning is a skill that requires a finely adjusted process of extracting just the right information from past experiences to benefit future choices. As we age, this process begins to alter, changing how we react to ambiguity, risk or uncertainty. One challenging aspect of learning is that sometimes we will encounter very surprising consequences of our actions, raising the question whether we should assign more or less weight to these events. We know relatively little about how humans react to these surprises and how age affects learning from surprising outcomes. To learn more about this question, we asked 51 older and 51 younger adults to play a reinforcement learning task that confronted them with surprising outcomes and analyzed their choices. We found that both age groups showed heightened sensitivity to surprising outcomes that resulted in distinctive behavioral adjustments. Notably, older adults weighted these surprising events more than younger adults. Comparing the choices of participants with computational models that incorporated surprises in different ways, we found a model that modulated its learning with the amount of surprise to mimic participants’ choices best. The results help to better understand the role of past surprising events during learning in older and younger adults.