IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v389y2025ics0306261925005197.html
   My bibliography  Save this article

Reliability assessment of multi-agent reinforcement learning algorithms for hybrid local electricity market simulation

Author

Listed:
  • Zhang, Haoyang
  • Qiu, Dawei
  • Kok, Koen
  • Paterakis, Nikolaos G.

Abstract

The reliability of data-driven multi-agent reinforcement learning (MARL) algorithms is a critical concern, particularly for complex, large-scale multi-agent decision-making problems. This paper aims to assess the reliability of various MARL algorithms in supporting decision-making for prosumer participants in a hybrid local electricity market (LEM) that combines community-based markets and a peer-to-peer (P2P) market. Specifically, it compares the performance of three MARL algorithms: the multi-agent deep deterministic policy gradient (MADDPG) algorithm and two advanced variants incorporating mean-field approximation and attention mechanisms. To evaluate the reliability of these data-driven MARL algorithms, a model-based bi-level optimization problem is introduced for each agent to assess convergence speed and the proximity of results to the ε-Nash equilibrium, as indicated by the no-regret index. The no-regret index is calculated within a mathematical program with equilibrium constraints (MPEC) by fixing the other agents’ behavior generated from the MARL algorithms. Simulation results demonstrate that the attention-MADDPG algorithm achieves the highest no-regret index (0.81), indicating convergence closest to equilibrium, and the greatest total cost reduction (983€), outperforming the other MARL algorithms. The mean-field-MADDPG algorithm is the most balanced, exhibiting robust convergence with the second-highest no-regret index (0.78) and cost reduction (958.8€) under the lowest computational burden (5.4 seconds per episode).

Suggested Citation

  • Zhang, Haoyang & Qiu, Dawei & Kok, Koen & Paterakis, Nikolaos G., 2025. "Reliability assessment of multi-agent reinforcement learning algorithms for hybrid local electricity market simulation," Applied Energy, Elsevier, vol. 389(C).
    • Handle: RePEc:eee:appene:v:389:y:2025:i:c:s0306261925005197
    DOI: 10.1016/j.apenergy.2025.125789
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261925005197
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2025.125789?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. May, Ross & Huang, Pei, 2023. "A multi-agent reinforcement learning approach for investigating and optimising peer-to-peer prosumer energy markets," Applied Energy, Elsevier, vol. 334(C).
    2. Steffen Meinecke & Džanan Sarajlić & Simon Ruben Drauz & Annika Klettke & Lars-Peter Lauven & Christian Rehtanz & Albert Moser & Martin Braun, 2020. "SimBench—A Benchmark Dataset of Electric Power Systems to Compare Innovative Solutions Based on Power Flow Analysis," Energies, MDPI, vol. 13(12), pages 1-19, June.
    3. Azarova, Valeriya & Cohen, Jed & Friedl, Christina & Reichl, Johannes, 2019. "Designing local renewable energy communities to increase social acceptance: Evidence from a choice experiment in Austria, Germany, Italy, and Switzerland," Energy Policy, Elsevier, vol. 132(C), pages 1176-1183.
    4. Faia, Ricardo & Lezama, Fernando & Soares, João & Pinto, Tiago & Vale, Zita, 2024. "Local electricity markets: A review on benefits, barriers, current trends and future perspectives," Renewable and Sustainable Energy Reviews, Elsevier, vol. 190(PA).
    5. Oprea, Simona-Vasilica & Bâra, Adela, 2021. "Devising a trading mechanism with a joint price adjustment for local electricity markets using blockchain. Insights for policy makers," Energy Policy, Elsevier, vol. 152(C).
    6. Sousa, Tiago & Soares, Tiago & Pinson, Pierre & Moret, Fabio & Baroche, Thomas & Sorin, Etienne, 2019. "Peer-to-peer and community-based markets: A comprehensive review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 104(C), pages 367-378.
    7. Dukovska, Irena & Slootweg, J.G. (Han) & Paterakis, Nikolaos G., 2023. "Introducing user preferences for peer-to-peer electricity trading through stochastic multi-objective optimization," Applied Energy, Elsevier, vol. 338(C).
    8. Qiu, Dawei & Wang, Yi & Hua, Weiqi & Strbac, Goran, 2023. "Reinforcement learning for electric vehicle applications in power systems:A critical review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 173(C).
    9. Zhang, Haoyang & Zhan, Sen & Kok, Koen & Paterakis, Nikolaos G., 2024. "Establishing a hierarchical local market structure using multi-cut Benders decomposition," Applied Energy, Elsevier, vol. 363(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Couraud, Benoit & Andoni, Merlinda & Robu, Valentin & Norbu, Sonam & Chen, Si & Flynn, David, 2023. "Responsive FLEXibility: A smart local energy system," Renewable and Sustainable Energy Reviews, Elsevier, vol. 182(C).
    2. Faia, Ricardo & Lezama, Fernando & Soares, João & Pinto, Tiago & Vale, Zita, 2024. "Local electricity markets: A review on benefits, barriers, current trends and future perspectives," Renewable and Sustainable Energy Reviews, Elsevier, vol. 190(PA).
    3. Hildebrandt, Benjamin & Hurink, Johann & Manitz, Michael, 2024. "Local energy management: A base model for the optimization of virtual economic units," Energy Economics, Elsevier, vol. 129(C).
    4. Petrovich, Beatrice & Kubli, Merla, 2023. "Energy communities for companies: Executives’ preferences for local and renewable energy procurement," Renewable and Sustainable Energy Reviews, Elsevier, vol. 184(C).
    5. Antoine Boche & Clément Foucher & Luiz Fernando Lavado Villa, 2022. "Understanding Microgrid Sustainability: A Systemic and Comprehensive Review," Energies, MDPI, vol. 15(8), pages 1-29, April.
    6. Bauwens, Thomas & Schraven, Daan & Drewing, Emily & Radtke, Jörg & Holstenkamp, Lars & Gotchev, Boris & Yildiz, Özgür, 2022. "Conceptualizing community in energy systems: A systematic review of 183 definitions," Renewable and Sustainable Energy Reviews, Elsevier, vol. 156(C).
    7. Zhang, Haoyang & Zhan, Sen & Kok, Koen & Paterakis, Nikolaos G., 2024. "Establishing a hierarchical local market structure using multi-cut Benders decomposition," Applied Energy, Elsevier, vol. 363(C).
    8. Vernay, Anne-Lorène & Sebi, Carine & Arroyo, Fabrice, 2023. "Energy community business models and their impact on the energy transition: Lessons learnt from France," Energy Policy, Elsevier, vol. 175(C).
    9. Heilmann, Jakob & Wensaas, Marthe & Crespo del Granado, Pedro & Hashemipour, Naser, 2022. "Trading algorithms to represent the wholesale market of energy communities in Norway and England," Renewable Energy, Elsevier, vol. 200(C), pages 1426-1437.
    10. Carattini, Stefano & Gillingham, Kenneth & Meng, Xiangyu & Yoeli, Erez, 2024. "Peer-to-peer solar and social rewards: Evidence from a field experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 219(C), pages 340-370.
    11. Zhang, Chenxi & Yang, Yi & Wang, Yunqi & Qiu, Jing & Zhao, Junhua, 2024. "Auction-based peer-to-peer energy trading considering echelon utilization of retired electric vehicle second-life batteries," Applied Energy, Elsevier, vol. 358(C).
    12. Kirchhoff, Hannes & Strunz, Kai, 2019. "Key drivers for successful development of peer-to-peer microgrids for swarm electrification," Applied Energy, Elsevier, vol. 244(C), pages 46-62.
    13. Lee, Juyong & Cho, Youngsang, 2020. "Estimation of the usage fee for peer-to-peer electricity trading platform: The case of South Korea," Energy Policy, Elsevier, vol. 136(C).
    14. Sward, Jeffrey A. & Nilson, Roberta S. & Katkar, Venktesh V. & Stedman, Richard C. & Kay, David L. & Ifft, Jennifer E. & Zhang, K. Max, 2021. "Integrating social considerations in multicriteria decision analysis for utility-scale solar photovoltaic siting," Applied Energy, Elsevier, vol. 288(C).
    15. Lan, Penghang & Chen, She & Li, Qihang & Li, Kelin & Wang, Feng & Zhao, Yaoxun, 2024. "Intelligent hydrogen-ammonia combined energy storage system with deep reinforcement learning," Renewable Energy, Elsevier, vol. 237(PB).
    16. Ma, Li & Wang, Lingfeng & Liu, Zhaoxi, 2021. "Multi-level trading community formation and hybrid trading network construction in local energy market," Applied Energy, Elsevier, vol. 285(C).
    17. Moret, Fabio & Pinson, Pierre & Papakonstantinou, Athanasios, 2020. "Heterogeneous risk preferences in community-based electricity markets," European Journal of Operational Research, Elsevier, vol. 287(1), pages 36-48.
    18. Marco Galici & Mario Mureddu & Emilio Ghiani & Fabrizio Pilo, 2022. "Blockchain-Based Hardware-in-the-Loop Simulation of a Decentralized Controller for Local Energy Communities," Energies, MDPI, vol. 15(20), pages 1-25, October.
    19. Shahzad, Qaisar & Aruga, Kentaka, 2025. "Trade-off in energy policy: Evidence from a best-worst discrete choice experiment," MPRA Paper 124042, University Library of Munich, Germany.
    20. Nieta, Agustín A. Sánchez de la & Ilieva, Iliana & Gibescu, Madeleine & Bremdal, Bernt & Simonsen, Stig & Gramme, Eivind, 2021. "Optimal midterm peak shaving cost in an electricity management system using behind customers’ smart meter configuration," Applied Energy, Elsevier, vol. 283(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:389:y:2025:i:c:s0306261925005197. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.