Giving AI agents a sense of control facilitates reinforcement learning in multitasking scenarios

Having to control multiple tasks in parallel poses challenges for humans and artificial agents alike. In artificial intelligence, specific forms of reinforcement learning (RL), most notably hierarchical and model-based RL, have shown promising results in scenarios where tasks or skills need to be switched adaptively. However, RL agents still encounter difficulties when faced with serial multitasking that involves switching control between continuously running subtasks, such as changing the radio station while driving in traffic. Inspired by human cognitive processes, we hypothesize that maintaining a sense of control is a key mechanism facilitating such task-switching decisions. We propose a mathematical formulation of a situational sense of control that consists of two components: an evaluative indicator of the predictability of action outcomes and a predictive indicator of a need for control in individual subtasks. We integrate this model of a sense of control into a hierarchical RL agent and evaluate its performance in a Collect Asteroids game environment, in which one must alternate between navigating two spaceships to collect as many asteroids as possible. Comparing RL agents with and without a sense of control, as well as with human participants, shows that equipping RL agents with a sense of control results in significant performance improvements. Our findings indicate that agents equipped with a sense of control prioritize more complex tasks, exhibit increased switching behavior, and make switches at strategically optimal times, leading to superior overall performance. The incorporation of cognitive mechanisms, inspired by human behavior, into RL agents thus appears to yield considerable enhancements in performance when acting in complex and dynamic environments.