Emergent Strategic Behaviour in a Macroeconomic Agent-Based Model with Reinforcement Learning

In canonical macroeconomic agent-based model (ABM), firms pursue behavioural (non-optimal) price and quantity strategies, that take the form of heuristics. In this paper we incorporate reinforcement learning (RL) into an otherwise standard ABM by replacing a fraction of heuristic-using firms with RL agents that learn profitmaximizing strategies through repeated interaction with the economic environment. When RL agents adopt a shared Q-function, they endogenously converge to one of three distinct strategic regimes – market power, predatory pricing, or quasi-perfect competition – with the prevailing equilibrium depending on the degree of market competition and the share of RL agents. Under independent Q-functions, agents spontaneously segregate into heterogeneous strategies, yielding higher aggregate market power and producer surplus without explicit coordination. The prevalence of RL agents shapes aggregate output and volatility in a non-monotonic way. To rationalize these findings, we develop a stylized theoretical framework that links the competition intensity between RL and non-RL agents with the prevailing optimal pricing strategies.