IDEAS home Printed from https://ideas.repec.org/a/eee/chsofr/v178y2024ics0960077923012602.html

Exploring cooperative evolution with tunable payoff’s loners using reinforcement learning

Author

Listed:
  • Zhang, Huizhen
  • An, Tianbo
  • Yan, Pingping
  • Hu, Kaipeng
  • An, Jinjin
  • Shi, Lijuan
  • Zhao, Jian
  • Wang, Jingrui

Abstract

Imitation and replication have emerged as a paradigm in numerous studies that explore the evolution of cooperative behavior. Since they embrace the essence of natural selection, it is widely recognized in exploring the evolution of biological behaviors. However, it is not easy to express the way individuals select and optimize in these simple and elegant ways in the complex and variable interactive environments. Currently, reinforcement learning is widely used in the study of strategy updating dynamics and agent learning processes in game theory. Therefore, we introduce the Q-learning algorithms into the voluntary public goods game to explore the impact of cooperative evolution. Simulation results demonstrate that when the synergy factor is large and the adjust loner payoff’s multiply factor is smaller, the number of cooperators becomes gradually consistent. As the synergy factor increases, the evolution of the proportion of defectors become nonlinear. Furthermore, we further explore the Q-table and strategy updating processes of agents in the steady state under a smaller multiply factor that adjusts the loner’s payoff. The results find inconsistency between the average Q-values and the steady state population strategy distribution. Subsequently, we explain the reason for the inconsistency by analyzing strategy sequence, namely that there are a number of agents who constantly change strategies in the population, and the Q-values of these agents have an impact on the overall Q-values. In addition, evolutionary snapshots of agent strategy sequences are observed. The results find that the agent’s strategic selection shows greater instability when the proportions of cooperators, defectors, and loners in the population are relatively balanced. Finally, the effect of parameters in the Q-learning algorithm on cooperative behavior is analyzed. This study hopes to provide valuable insights into understanding the dynamics of cooperation in complex social interactions.

Suggested Citation

  • Zhang, Huizhen & An, Tianbo & Yan, Pingping & Hu, Kaipeng & An, Jinjin & Shi, Lijuan & Zhao, Jian & Wang, Jingrui, 2024. "Exploring cooperative evolution with tunable payoff’s loners using reinforcement learning," Chaos, Solitons & Fractals, Elsevier, vol. 178(C).
    • Handle: RePEc:eee:chsofr:v:178:y:2024:i:c:s0960077923012602
    DOI: 10.1016/j.chaos.2023.114358
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0960077923012602
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.chaos.2023.114358?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Haozheng Xu & Yiwen Zhang & Xing Jin & Jingrui Wang & Zhen Wang, 2023. "The Evolution of Cooperation in Multigames with Uniform Random Hypergraphs," Mathematics, MDPI, vol. 11(11), pages 1-11, May.
    2. Pan, Jianchen & Zhang, Lan & Han, Wenchen & Huang, Changwei, 2023. "Heterogeneous investment promotes cooperation in spatial public goods game on hypergraphs," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 609(C).
    3. Jingrui Wang & Xing Jin & Yixuan Yang & Qingfang Chen & Zhen Wang & Hong Ding, 2021. "The spread of epidemic under voluntary vaccination with heterogeneous infection rates," International Journal of Modern Physics C (IJMPC), World Scientific Publishing Co. Pte. Ltd., vol. 32(03), pages 1-14, March.
    4. Wang, Jingrui & Zhang, Huizhen & Jin, Xing & Ma, Leyu & Chen, Yueren & Wang, Chao & Zhao, Jian & An, Tianbo, 2023. "Subsidy policy with punishment mechanism can promote voluntary vaccination behaviors in structured populations," Chaos, Solitons & Fractals, Elsevier, vol. 174(C).
    5. Jin, Xing & Tao, Yuchen & Wang, Jingrui & Wang, Chao & Wang, Yongheng & Zhang, Zhouyang & Wang, Zhen, 2023. "Strategic use of payoff information in k-hop evolutionary Best-shot networked public goods game," Applied Mathematics and Computation, Elsevier, vol. 459(C).
    6. Gao, Bo & Liu, Xuan & Lan, Zhong-Zhou & Hong, Jie & Zhang, Wenguang, 2021. "The evolution of cooperation with preferential selection in voluntary public goods game," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 584(C).
    7. Cui, Guang-Hai & Wang, Zhen & Li, Jun-Li & Jin, Xing & Zhang, Zhi-Wang, 2021. "Influence of precaution and dynamic post-indemnity based insurance policy on controlling the propagation of epidemic security risks in networks," Applied Mathematics and Computation, Elsevier, vol. 392(C).
    8. Yang, Zhengzhi & Zheng, Lei & Perc, Matjaž & Li, Yumeng, 2024. "Interaction state Q-learning promotes cooperation in the spatial prisoner's dilemma game," Applied Mathematics and Computation, Elsevier, vol. 463(C).
    9. Lee, Hsuan-Wei & Cleveland, Colin & Szolnoki, Attila, 2022. "Mercenary punishment in structured populations," Applied Mathematics and Computation, Elsevier, vol. 417(C).
    10. Huang, Changwei & Hou, Yongzhao & Han, Wenchen, 2023. "Coevolution of consensus and cooperation in evolutionary Hegselmann–Krause dilemma with the cooperation cost," Chaos, Solitons & Fractals, Elsevier, vol. 168(C).
    11. Jia, Danyang & Li, Tong & Zhao, Yang & Zhang, Xiaoqin & Wang, Zhen, 2022. "Empty nodes affect conditional cooperation under reinforcement learning," Applied Mathematics and Computation, Elsevier, vol. 413(C).
    12. Gabriela Koľveková & Manuela Raisová & Martin Zoričak & Vladimír Gazda, 2021. "Endogenous Shared Punishment Model in Threshold Public Goods Games," Computational Economics, Springer;Society for Computational Economics, vol. 58(1), pages 57-81, June.
    13. Quan, Ji & Yu, Junyu & Li, Xia & Wang, Xianjia, 2023. "Conditional switching between social excluders and loners promotes cooperation in spatial public goods game," Chaos, Solitons & Fractals, Elsevier, vol. 169(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Zhang, Yongqiang & Zheng, Zehao & Zhang, Xiaoming & Ma, Jinlong, 2025. "Dynamic punishment-reputation synergy drives cooperation in spatial public goods game," Applied Mathematics and Computation, Elsevier, vol. 506(C).
    2. Xie, Kai & Szolnoki, Attila, 2025. "Reputation in public goods cooperation under double Q-learning protocol," Chaos, Solitons & Fractals, Elsevier, vol. 196(C).
    3. Li, Yipeng & Hu, Xiangyue & Jin, Xing & Zhang, Huizhen & Yang, Jiajia & Wang, Zhen, 2025. "Environmental information perception enhances cooperation in stochastic public goods games via Q-learning," Applied Mathematics and Computation, Elsevier, vol. 504(C).
    4. An, Tianbo & Zhang, Huizhen & Zhang, Zhanshuo & Liu, Guanghui & Li, Jiayu & Chen, Liangyu & Wang, Zhen, 2025. "Cooperation dynamics driven by reinforcement learning with interactive diversity in structured populations," Chaos, Solitons & Fractals, Elsevier, vol. 201(P2).
    5. Mangold, Gustavo C. & Vainstein, Mendeli H. & Fernandes, Heitor C.M., 2025. "Dilution, diffusion and symbiosis in spatial prisoner’s dilemma with reinforcement learning," Chaos, Solitons & Fractals, Elsevier, vol. 201(P3).
    6. Xie, Kai & Szolnoki, Attila, 2026. "Reinforcement learning in evolutionary game theory: A brief review of recent developments," Applied Mathematics and Computation, Elsevier, vol. 510(C).
    7. Qian, Yinuo & Zhao, Dawei & Xia, Chengyi, 2025. "Disease transmission in dynamic social networks constructed by reinforcement learning-driven preventive game," Chaos, Solitons & Fractals, Elsevier, vol. 199(P1).
    8. Wang, Jiaoyuan & Yang, Yanlong, 2026. "Research on the resilience of reputation mechanism to the cooperative environment in the face of external shocks," Applied Mathematics and Computation, Elsevier, vol. 510(C).
    9. Huang, Yijie, 2025. "The evolution of cooperation in multi-games with reinforcement learning," Chaos, Solitons & Fractals, Elsevier, vol. 201(P2).
    10. Zheng, Guozhong & Zhang, Jiqiang & Deng, Shengfeng & Cai, Weiran & Chen, Li, 2024. "Evolution of cooperation in the public goods game with Q-learning," Chaos, Solitons & Fractals, Elsevier, vol. 188(C).
    11. Shen, Shaofei & Zhang, Xuejun & Xu, Aobo & Duan, Taisen, 2024. "An adaptive exploration mechanism for Q-learning in spatial public goods games," Chaos, Solitons & Fractals, Elsevier, vol. 189(P1).
    12. Zhang, Wei & Zhao, Dongkai & Jin, Xing & Zhang, Huizhen & An, Tianbo & Cui, Guanghai & Wang, Zhen, 2025. "Q-learning facilitates norm emergence in metanorm game model with topological structures," Chaos, Solitons & Fractals, Elsevier, vol. 195(C).
    13. Wang, Weining & Shang, Lihui & Wu, Yipeng & Hu, Mingjian & Wang, Weiyu, 2025. "Bio-inspired mechanism promotes cooperation in spatial public goods games," Chaos, Solitons & Fractals, Elsevier, vol. 200(P3).
    14. Zhao, Jian & Zhang, Ran & An, Tianbo & Zhang, Huizhen & Tong, Daqun & Luo, Xun & An, Jinjin & Wang, Jingrui, 2025. "The effect of heterogeneous perceptions with environmental feedback on spatial social dilemmas," Chaos, Solitons & Fractals, Elsevier, vol. 192(C).
    15. Yuanguo Lin & Fan Lin & Guorong Cai & Hong Chen & Linxin Zou & Yunxuan Liu & Pengcheng Wu, 2025. "Evolutionary Reinforcement Learning: A Systematic Review and Future Directions," Mathematics, MDPI, vol. 13(5), pages 1-33, March.
    16. Wu, Binjie & Shen, Shaofei & Wang, Jiafeng & Wan, Haibin, 2025. "Q-learning promotes the evolution of fairness and generosity in the ultimatum game," Chaos, Solitons & Fractals, Elsevier, vol. 200(P2).
    17. Yan, Zeyuan & Zhao, Hui & Li, Li, 2025. "Dynamic role-switching in hypergraphs: Enhancing cooperation via adaptive punishment and reinforcement learning," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 677(C).
    18. Yang, Yujin & Zhao, Dawei & Wang, Juan, 2025. "Evolution of cooperation in spatial public goods games driven by reinforcement learning and environmental feedback," Chaos, Solitons & Fractals, Elsevier, vol. 199(P1).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wang, Jingrui & Zhang, Huizhen & Jin, Xing & Ma, Leyu & Chen, Yueren & Wang, Chao & Zhao, Jian & An, Tianbo, 2023. "Subsidy policy with punishment mechanism can promote voluntary vaccination behaviors in structured populations," Chaos, Solitons & Fractals, Elsevier, vol. 174(C).
    2. Lee, Hsuan-Wei & Weng, Yi-Ning, 2025. "Granular Q-learning adaptation boosts collective welfare in multi-agent Prisoner’s Dilemma," Chaos, Solitons & Fractals, Elsevier, vol. 199(P1).
    3. Huang, Yijie, 2025. "The evolution of cooperation in multi-games with reinforcement learning," Chaos, Solitons & Fractals, Elsevier, vol. 201(P2).
    4. Wang, Chengjie & Deng, Juan & Zhao, Hui & Li, Li, 2024. "Effect of Q-learning on the evolution of cooperation behavior in collective motion: An improved Vicsek model," Applied Mathematics and Computation, Elsevier, vol. 482(C).
    5. Wang, Xianjia & Yang, Zhipeng & Liu, Yanli & Chen, Guici, 2023. "A reinforcement learning-based strategy updating model for the cooperative evolution," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 618(C).
    6. Cui, Guang-Hai & Li, Jun-Li & Dong, Kun-Xiang & Jin, Xing & Yang, Hong-Yong & Wang, Zhen, 2024. "Influence of subsidy policies against insurances on controlling the propagation of epidemic security risks in networks," Applied Mathematics and Computation, Elsevier, vol. 476(C).
    7. Bai, Pengzhou & Qiang, Bingzhuang & Zou, Kuan & Huang, Changwei, 2024. "Preferential selection based on adaptive attractiveness induce by reinforcement learning promotes cooperation," Chaos, Solitons & Fractals, Elsevier, vol. 180(C).
    8. Lv, Shaojie & Li, Jiaying & Zhao, Changheng, 2025. "Reinforcement learning in spatial public goods games with environmental feedbacks," Chaos, Solitons & Fractals, Elsevier, vol. 195(C).
    9. Zhang, Yao & Hao, Qing-Yi & Qian, Jia-Li & Wu, Chao-Yun & Bi, Yan, 2024. "The influence of the heterogeneities of social institutions and individuals’ tendency to establish social institutions on cooperation," Chaos, Solitons & Fractals, Elsevier, vol. 186(C).
    10. Takesue, Hirofumi, 2026. "Hinderance of cooperation by individual solutions: Evolutionary dynamics of three-strategy games combining the prisoner’s dilemma and stag hunt," Applied Mathematics and Computation, Elsevier, vol. 510(C).
    11. Takesue, Hirofumi, 2025. "Evolution of cooperation in a three-strategy game combining snowdrift and stag hunt games," Applied Mathematics and Computation, Elsevier, vol. 495(C).
    12. Xie, Kai & Szolnoki, Attila, 2026. "Reinforcement learning in evolutionary game theory: A brief review of recent developments," Applied Mathematics and Computation, Elsevier, vol. 510(C).
    13. Zhang, Lan & Li, Yuqin & Xie, Yuan & Feng, Yuee & Huang, Changwei, 2025. "The combined effects of conformity and reinforcement learning on the evolution of cooperation in public goods games," Chaos, Solitons & Fractals, Elsevier, vol. 193(C).
    14. Zhang, Yongqiang & Zheng, Zehao & Zhang, Xiaoming & Ma, Jinlong, 2025. "A layered strategy updating mechanism for spatial public goods game with punishment," Chaos, Solitons & Fractals, Elsevier, vol. 201(P2).
    15. Liu, Zhuo & Wang, Juan & Li, Xiaopeng, 2024. "Evolutionary dynamics of networked N-player trust games with exclusion strategy," Chaos, Solitons & Fractals, Elsevier, vol. 186(C).
    16. Zhou, Meng & Yang, Yanlong & Xiang, Shuwen, 2025. "Effect of community learning mechanism on cooperation in conflict societies," Chaos, Solitons & Fractals, Elsevier, vol. 192(C).
    17. He, Jialu & Lan, Xue & Zheng, Yupeng & Zhang, Han & Liu, Chunhe, 2025. "The impact of preference selection based on influence in spatial evolutionary vaccination game," Chaos, Solitons & Fractals, Elsevier, vol. 199(P3).
    18. Lee, Hsuan-Wei & Cleveland, Colin & Szolnoki, Attila, 2024. "Supporting punishment via taxation in a structured population," Chaos, Solitons & Fractals, Elsevier, vol. 178(C).
    19. Yang, Zhengzhi & Zheng, Lei & Perc, Matjaž & Li, Yumeng, 2024. "Interaction state Q-learning promotes cooperation in the spatial prisoner's dilemma game," Applied Mathematics and Computation, Elsevier, vol. 463(C).
    20. Lee, Hsuan-Wei & Cleveland, Colin & Szolnoki, Attila, 2024. "Suppressing defection by increasing temptation: The impact of smart cooperators on a social dilemma situation," Applied Mathematics and Computation, Elsevier, vol. 479(C).

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:chsofr:v:178:y:2024:i:c:s0960077923012602. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Thayer, Thomas R. (email available below). General contact details of provider: https://www.journals.elsevier.com/chaos-solitons-and-fractals .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.