IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v389y2025ics0306261925005197.html
   My bibliography  Save this article

Reliability assessment of multi-agent reinforcement learning algorithms for hybrid local electricity market simulation

Author

Listed:
  • Zhang, Haoyang
  • Qiu, Dawei
  • Kok, Koen
  • Paterakis, Nikolaos G.

Abstract

The reliability of data-driven multi-agent reinforcement learning (MARL) algorithms is a critical concern, particularly for complex, large-scale multi-agent decision-making problems. This paper aims to assess the reliability of various MARL algorithms in supporting decision-making for prosumer participants in a hybrid local electricity market (LEM) that combines community-based markets and a peer-to-peer (P2P) market. Specifically, it compares the performance of three MARL algorithms: the multi-agent deep deterministic policy gradient (MADDPG) algorithm and two advanced variants incorporating mean-field approximation and attention mechanisms. To evaluate the reliability of these data-driven MARL algorithms, a model-based bi-level optimization problem is introduced for each agent to assess convergence speed and the proximity of results to the ε-Nash equilibrium, as indicated by the no-regret index. The no-regret index is calculated within a mathematical program with equilibrium constraints (MPEC) by fixing the other agents’ behavior generated from the MARL algorithms. Simulation results demonstrate that the attention-MADDPG algorithm achieves the highest no-regret index (0.81), indicating convergence closest to equilibrium, and the greatest total cost reduction (983€), outperforming the other MARL algorithms. The mean-field-MADDPG algorithm is the most balanced, exhibiting robust convergence with the second-highest no-regret index (0.78) and cost reduction (958.8€) under the lowest computational burden (5.4 seconds per episode).

Suggested Citation

  • Zhang, Haoyang & Qiu, Dawei & Kok, Koen & Paterakis, Nikolaos G., 2025. "Reliability assessment of multi-agent reinforcement learning algorithms for hybrid local electricity market simulation," Applied Energy, Elsevier, vol. 389(C).
    • Handle: RePEc:eee:appene:v:389:y:2025:i:c:s0306261925005197
    DOI: 10.1016/j.apenergy.2025.125789
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261925005197
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2025.125789?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. May, Ross & Huang, Pei, 2023. "A multi-agent reinforcement learning approach for investigating and optimising peer-to-peer prosumer energy markets," Applied Energy, Elsevier, vol. 334(C).
    2. Qiu, Dawei & Wang, Yi & Hua, Weiqi & Strbac, Goran, 2023. "Reinforcement learning for electric vehicle applications in power systems:A critical review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 173(C).
    3. Steffen Meinecke & Džanan Sarajlić & Simon Ruben Drauz & Annika Klettke & Lars-Peter Lauven & Christian Rehtanz & Albert Moser & Martin Braun, 2020. "SimBench—A Benchmark Dataset of Electric Power Systems to Compare Innovative Solutions Based on Power Flow Analysis," Energies, MDPI, vol. 13(12), pages 1-19, June.
    4. Zhang, Haoyang & Zhan, Sen & Kok, Koen & Paterakis, Nikolaos G., 2024. "Establishing a hierarchical local market structure using multi-cut Benders decomposition," Applied Energy, Elsevier, vol. 363(C).
    5. Oprea, Simona-Vasilica & Bâra, Adela, 2021. "Devising a trading mechanism with a joint price adjustment for local electricity markets using blockchain. Insights for policy makers," Energy Policy, Elsevier, vol. 152(C).
    6. Sousa, Tiago & Soares, Tiago & Pinson, Pierre & Moret, Fabio & Baroche, Thomas & Sorin, Etienne, 2019. "Peer-to-peer and community-based markets: A comprehensive review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 104(C), pages 367-378.
    7. Azarova, Valeriya & Cohen, Jed & Friedl, Christina & Reichl, Johannes, 2019. "Designing local renewable energy communities to increase social acceptance: Evidence from a choice experiment in Austria, Germany, Italy, and Switzerland," Energy Policy, Elsevier, vol. 132(C), pages 1176-1183.
    8. Faia, Ricardo & Lezama, Fernando & Soares, João & Pinto, Tiago & Vale, Zita, 2024. "Local electricity markets: A review on benefits, barriers, current trends and future perspectives," Renewable and Sustainable Energy Reviews, Elsevier, vol. 190(PA).
    9. Dukovska, Irena & Slootweg, J.G. (Han) & Paterakis, Nikolaos G., 2023. "Introducing user preferences for peer-to-peer electricity trading through stochastic multi-objective optimization," Applied Energy, Elsevier, vol. 338(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Antoine Boche & Clément Foucher & Luiz Fernando Lavado Villa, 2022. "Understanding Microgrid Sustainability: A Systemic and Comprehensive Review," Energies, MDPI, vol. 15(8), pages 1-29, April.
    2. Bauwens, Thomas & Schraven, Daan & Drewing, Emily & Radtke, Jörg & Holstenkamp, Lars & Gotchev, Boris & Yildiz, Özgür, 2022. "Conceptualizing community in energy systems: A systematic review of 183 definitions," Renewable and Sustainable Energy Reviews, Elsevier, vol. 156(C).
    3. Zhang, Haoyang & Zhan, Sen & Kok, Koen & Paterakis, Nikolaos G., 2024. "Establishing a hierarchical local market structure using multi-cut Benders decomposition," Applied Energy, Elsevier, vol. 363(C).
    4. Couraud, Benoit & Andoni, Merlinda & Robu, Valentin & Norbu, Sonam & Chen, Si & Flynn, David, 2023. "Responsive FLEXibility: A smart local energy system," Renewable and Sustainable Energy Reviews, Elsevier, vol. 182(C).
    5. Vernay, Anne-Lorène & Sebi, Carine & Arroyo, Fabrice, 2023. "Energy community business models and their impact on the energy transition: Lessons learnt from France," Energy Policy, Elsevier, vol. 175(C).
    6. Faia, Ricardo & Lezama, Fernando & Soares, João & Pinto, Tiago & Vale, Zita, 2024. "Local electricity markets: A review on benefits, barriers, current trends and future perspectives," Renewable and Sustainable Energy Reviews, Elsevier, vol. 190(PA).
    7. Thunshirn, Philipp & Mlinaric, Ines & Berg, Jana, 2025. "A qualitative analysis of consumer motivations and barriers towards active smart meter utilization," Energy Policy, Elsevier, vol. 203(C).
    8. Hildebrandt, Benjamin & Hurink, Johann & Manitz, Michael, 2024. "Local energy management: A base model for the optimization of virtual economic units," Energy Economics, Elsevier, vol. 129(C).
    9. Heilmann, Jakob & Wensaas, Marthe & Crespo del Granado, Pedro & Hashemipour, Naser, 2022. "Trading algorithms to represent the wholesale market of energy communities in Norway and England," Renewable Energy, Elsevier, vol. 200(C), pages 1426-1437.
    10. Petrovich, Beatrice & Kubli, Merla, 2023. "Energy communities for companies: Executives’ preferences for local and renewable energy procurement," Renewable and Sustainable Energy Reviews, Elsevier, vol. 184(C).
    11. Carattini, Stefano & Gillingham, Kenneth & Meng, Xiangyu & Yoeli, Erez, 2024. "Peer-to-peer solar and social rewards: Evidence from a field experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 219(C), pages 340-370.
    12. Wang, Dongxue & Fan, Ruguo & Yang, Peiwen & Du, Kang & Xu, Xiaoxia & Chen, Rongkai, 2024. "Research on floating real-time pricing strategy for microgrid operator in local energy market considering shared energy storage leasing," Applied Energy, Elsevier, vol. 368(C).
    13. Zhang, Chenxi & Yang, Yi & Wang, Yunqi & Qiu, Jing & Zhao, Junhua, 2024. "Auction-based peer-to-peer energy trading considering echelon utilization of retired electric vehicle second-life batteries," Applied Energy, Elsevier, vol. 358(C).
    14. Zhou, Yuekuan & Lund, Peter D., 2023. "Peer-to-peer energy sharing and trading of renewable energy in smart communities ─ trading pricing models, decision-making and agent-based collaboration," Renewable Energy, Elsevier, vol. 207(C), pages 177-193.
    15. Mukherjee, Monish & Hardy, Trevor & Fuller, Jason C. & Bose, Anjan, 2022. "Implementing multi-settlement decentralized electricity market design for transactive communities with imperfect communication," Applied Energy, Elsevier, vol. 306(PA).
    16. Yang, Jing & Zhang, Zhiyong & Hong, Ming & Yang, Mingwan & Chen, Jiayu, 2020. "An oligarchy game model for the mobile waste heat recovery energy supply chain," Energy, Elsevier, vol. 210(C).
    17. Doumen, Sjoerd C. & Nguyen, Phuong & Kok, Koen, 2022. "Challenges for large-scale Local Electricity Market implementation reviewed from the stakeholder perspective," Renewable and Sustainable Energy Reviews, Elsevier, vol. 165(C).
    18. Kirchhoff, Hannes & Strunz, Kai, 2019. "Key drivers for successful development of peer-to-peer microgrids for swarm electrification," Applied Energy, Elsevier, vol. 244(C), pages 46-62.
    19. Lee, Juyong & Cho, Youngsang, 2020. "Estimation of the usage fee for peer-to-peer electricity trading platform: The case of South Korea," Energy Policy, Elsevier, vol. 136(C).
    20. Botelho, D.F. & de Oliveira, L.W. & Dias, B.H. & Soares, T.A. & Moraes, C.A., 2022. "Integrated prosumers–DSO approach applied in peer-to-peer energy and reserve tradings considering network constraints," Applied Energy, Elsevier, vol. 317(C).

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:389:y:2025:i:c:s0306261925005197. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.