Separating random and deterministic sources of computational noise in explore-exploit decisions

Human decision making is inherently variable. While this variability is often seen as a sign of suboptimal behavior, both theoretical work in machine learning and empirical human studies suggest that variability can actually be adaptive. An example arises when we must choose between exploring unknown options or exploiting options we know well. A little randomness in these ‘explore-exploit’ decisions is remarkably effective as it can encourage us to explore options we might otherwise ignore. In line with this idea, several studies have found evidence that people increase their behavioral variability when it is valuable to explore. A key question, however, is whether this variability in so-called ‘random exploration’ is actually random. That is, is random exploration driven by stochastic processes in the brain or by some unobserved deterministic process that we have failed to account for when measuring behavioral variability? By designing an explore-exploit task in which, unbeknownst to them, participants are presented with the exact same choice twice, we provide a partial answer to this question. By modeling behavior in this task, we were able to estimate a lower bound on the amount of variability that is deterministically driven by the stimulus and an upper bound on the amount of variability that is random. Using this approach, we find evidence that at least 14% of the variability in random exploration in our studied task can be accounted for by deterministic processing of the stimulus. Conversely, this suggests that up to 86% of the variability is truly ‘random’, although it is still possible that this variability is driven by deterministic factors not related to the stimulus. Finally, our results suggest that both deterministic and random sources of variability change proportionally to each other as the value of exploration increases, suggesting that a common noise gating mechanism may be at play in random exploration.Author summary: Human decisions often seem random. Even simple decisions like what food to order at a restaurant can be difficult to predict ahead of time. This randomness in our decisions can be beneficial, effectively allowing us to explore new options. One outstanding question is where the randomness in our decisions comes from. Sometimes, our seemingly random decisions are driven by predictable external factors, like what the guests at the next table order could influence what we order. Other times, our decisions are not driven by external factors but are instead made by random thoughts within our brain. In this work, we developed a computational method that quantifies the extent to which the apparent randomness in our decisions can be explained by deterministic sources of variability in the external stimuli, or random variability unexplained by the stimuli. We found evidence that randomness in exploratory decisions can be explained by both random (up to 86%) and deterministic (more than 14%) sources of variability. Moreover, our results suggest that both sources of variability are adaptive, which enables humans to explore more when it is more beneficial to explore. The joint adaptation of random and deterministic noises also suggests a common noise-gating mechanism for exploration.