Cooperation dynamics driven by reinforcement learning with interactive diversity in structured populations

In reality, individuals tend to make different decisions based on differences in relationships and behaviors with their neighbors. Based on this observation, the paper explores the evolution of cooperative behavior when agents develop separated actions for each neighbor by the reinforcement learning approach. Through simulation experiments, it is shown that our model improves the cooperative level compared to results that only consider the agent’s own behavior. This is because agents tend to adopt cooperative strategies toward their neighbors while avoiding exploitation, thus promoting the steady expansion of cooperation. Notably, we find that agents do not always choose the action with the highest expected rewards. Therefore, we classify the behavior strategies of the agents into 16 types, corresponding to all possible combinations of actions selected in different states. We observe that agents adopting a specific behavior strategy tend to dominate the evolutionary process: when they choose to cooperate, they switch to defection in the next round regardless of the opponent’s action; conversely, when they defect, they switch to cooperation in the next round, again independent of the opponent’s behavior. These agents are typically distributed among others with different strategy types, playing a bridging and buffering role. By facilitating the expansion of neighboring agents, they contribute to the spread of cooperative behavior and ultimately enhance the overall level of cooperation in the population. Similar phenomena are also observed under initial specific distributions (e.g., ALLC, ALLD). Next, the hyperparameters of reinforcement learning are analyzed, and the results show that cooperation is easier to maintain and expand when agents make decisions based on past experiences and fully consider potential future rewards. We also compare this model with a control model that adopted the assumption of interactive homogeneity, and further examine the impact of different network structures on the cooperative evolution. Finally, we introduce the memory mechanism of agents as an extended analysis of the model.