Strategic exploitation in Boltzmann Q-learning in the prisoners dilemma

Reinforcement learning is one of the most promising techniques of uncovering near-optimal solutions in various domains of artificial intelligence. For the case of single-agent learning in stationary environment plentiful theoretical and empirical results are available in the literature. However, a consistent treatment of the case of multiagent learning appears to be one of central challenges in the field. The reason for such a difficulty is in the non-stationary nature of the multiagent learning process due to interrelated effects of agent’s decisions. A natural framework for describing interdependent decisions of agents is provided by game theory. In this study we apply two most popular reinforcement learning algorithms, SARSA and Q-learning, to the analysis of the iterated Prisoner’s Dilemma. The main focus is on studying the interplay between the exploration and strategic exploitation in the learning process. The exploration process is parametrised by the Boltzmann choice probabilities. The strategic exploitation is introduced through dependence on discounting parameter in stateless TD(1) SARSA and Q-learning algorithms. We describe three characteristics regimes corresponding to different combinations of exploration and strategic exploitation intensities. In particular, the domination of exploration process leads to the regime characterised by player’s policies fluctuating around quantal response equilibrium for the noisy Prisoner’s Dilemma. Domination of the strategic exploitation leads to an appearance of the strategic trap effect, when both players are for significant time locked in the state numerically indistinguishable from the Nash equilibrium in pure strategies in noiseless Prisoner’s dilemma. In case of moderate intensities of both exploration and exploitation processes the intermediate regime with periodically changing policies does appear.