Stimulus uncertainty and relative reward rates determine adaptive responding in perceptual decision-making

In dynamic environments, animals must select actions based on sensory input as well as expected positive and negative consequences. This type of behavior is typically studied using perceptual decision making (PDM) tasks. The arguably most influential framework for describing the cognitive processes underlying PDM is signal detection theory (SDT). One central assumption of SDT is that observers make perceptual decisions by comparing sensory evidence to a static decision criterion. However, mounting evidence suggests that the criterion is in fact highly dynamic and that observers adjust it flexibly according to task demands. Nevertheless, the mechanisms by which observers integrate stimulus and reward information for adaptive criterion learning remain not well understood. Here, we systematically investigated the factors influencing criterion setting at the single-trial level. To that end, we first specified three SDT-based models that learn either from reward, reward omission, or both. Next, by concomitantly manipulating stimulus and reward probabilities, we constructed experimental conditions in which these models make divergent predictions. Finally, we subjected rats and pigeons to a PDM task comprising these conditions. We find that subjects adopted decision criteria that maximize total reward in all experimental conditions. Detailed behavioral analyses reveal that criterion learning is driven by the integration of rewards, not reward omissions, and that reward integration is influenced by two additional factors: first, the degree of stimulus uncertainty, and second, the difference in the relative reward rates (rather than the absolute reward rates) between the choice alternatives. A model incorporating these factors accounts well for criterion dynamics across experimental conditions for both species and links signal detection theory to a learning mechanism operating at the level of single trials which, in the steady state, produces behavior similar to the matching law, a central tenet of learning theory.Author summary: Humans and other animals rely on their senses and experience to categorize objects and pursue their goals. For example, a mushroom hunter uses sight, smell and touch as well as knowledge of the local biota to decide whether to pick a particular mushroom. The consequences of erring can be dire – food intoxication if savoring a poisonous exemplar, or a meager dinner if too many palatable mushrooms are rejected. Also, the hunter’s decision may be influenced by ambient lighting conditions or his estimate of how likely it is to encounter poisonous mushrooms in a particular area at a particular time of year. Our work is concerned with the algorithms that animals use to make such decisions and how they adapt when circumstances, such as stimulus discriminability, change. We show that mathematical models that incorporate the animals’ uncertainty about the type of stimulus currently being perceived make very similar decisions to the animals. Furthermore, we find that the animals balance their choices by considering relative, rather than absolute, reward expectations, reflecting a long-standing principle of animal learning theory. Together, these features collectively allow animals to obtain nearly as many rewards as theoretically possible.