Facilitating cooperative behavior through reinforcement learning with age-driven state transitions in structured populations

The Q-learning algorithm in reinforcement learning closely parallels human experience based decision making, and has been widely applied to evolutionary game theory to study the emergence of cooperation. Previous studies have expanded the definition of the state space to some extent, but they have not captured the continuity of states in the temporal dimension. To address this gap, we propose a multistate transition mechanism driven by strategy age, where an agent’s state is defined by the time steps it persists with the same strategy. As the strategy age increases, the agent transitions into higher states until reaching the maximum threshold. For comparison, we also design a bistate mechanism that distinguishes states only between ages below and above the threshold. Simulation results show that both multistate and bistate mechanisms promote cooperation significantly better than memoryless and self-regarding Q-learning, with the multistate mechanism performing best. The key reason is that high age defectors see their Q-value of choosing defection drop below that of choosing cooperation and thus occasionally switch to cooperate under prolonged exposure to defectors. These switches periodically seed new low age cooperators, continually replenishing the cooperative pool. Raising the strategy age threshold expands the state space, giving defectors more chances to switch to cooperation and further boosting cooperation. By contrast, the bistate mechanism partitions the space too coarsely, limiting such transitions and yielding weaker outcomes. We also find that cooperation is most likely to emerge under moderate learning rates α and higher discount factors γ.