IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v317y2022ics0306261922005025.html
   My bibliography  Save this article

Multi-agent deep deterministic policy gradient algorithm for peer-to-peer energy trading considering distribution network constraints

Author

Listed:
  • Samende, Cephas
  • Cao, Jun
  • Fan, Zhong

Abstract

In this paper, we investigate an energy cost minimization problem for prosumers participating in peer-to-peer energy trading. Due to (i) uncertainties caused by renewable energy generation and consumption, (ii) difficulties in developing an accurate and efficient energy trading model, and (iii) the need to satisfy distribution network constraints, it is challenging for prosumers to obtain optimal energy trading decisions that minimize their individual energy costs. To address the challenge, we first formulate the above problem as a Markov decision process and propose a multi-agent deep deterministic policy gradient algorithm to learn optimal energy trading decisions. To satisfy the distribution network constraints, we propose distribution network tariffs which we incorporate in the algorithm as incentives to incentivize energy trading decisions that help to satisfy the constraints and penalize the decisions that violate them. The proposed algorithm is model-free and allows the agents to learn the optimal energy trading decisions without having prior information about other agents in the network. Simulation results based on real-world datasets show the effectiveness and robustness of the proposed algorithm.

Suggested Citation

  • Samende, Cephas & Cao, Jun & Fan, Zhong, 2022. "Multi-agent deep deterministic policy gradient algorithm for peer-to-peer energy trading considering distribution network constraints," Applied Energy, Elsevier, vol. 317(C).
    • Handle: RePEc:eee:appene:v:317:y:2022:i:c:s0306261922005025
    DOI: 10.1016/j.apenergy.2022.119123
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261922005025
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2022.119123?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Qiu, Dawei & Ye, Yujian & Papadaskalopoulos, Dimitrios & Strbac, Goran, 2021. "Scalable coordinated management of peer-to-peer energy trading: A multi-cluster deep reinforcement learning approach," Applied Energy, Elsevier, vol. 292(C).
    2. Thomas Morstyn & Niall Farrell & Sarah J. Darby & Malcolm D. McCulloch, 2018. "Using peer-to-peer energy-trading platforms to incentivize prosumers to form federated power plants," Nature Energy, Nature, vol. 3(2), pages 94-101, February.
    3. Jin-Gyeom Kim & Bowon Lee, 2020. "Automatic P2P Energy Trading Model Based on Reinforcement Learning Using Long Short-Term Delayed Reward," Energies, MDPI, vol. 13(20), pages 1-27, October.
    4. Olivella-Rosell, Pol & Bullich-Massagué, Eduard & Aragüés-Peñalba, Mònica & Sumper, Andreas & Ottesen, Stig Ødegaard & Vidal-Clos, Josep-Andreu & Villafáfila-Robles, Roberto, 2018. "Optimization problem for meeting distribution system operator requests in local flexibility markets with distributed energy resources," Applied Energy, Elsevier, vol. 210(C), pages 881-895.
    5. Han, Sekyung & Han, Soohee & Aki, Hirohisa, 2014. "A practical battery wear model for electric vehicle charging applications," Applied Energy, Elsevier, vol. 113(C), pages 1100-1108.
    6. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    7. Long, Chao & Wu, Jianzhong & Zhou, Yue & Jenkins, Nick, 2018. "Peer-to-peer energy sharing through a two-stage aggregated battery control in a community Microgrid," Applied Energy, Elsevier, vol. 226(C), pages 261-276.
    8. Juhar Abdella & Khaled Shuaib, 2018. "Peer to Peer Distributed Energy Trading in Smart Grids: A Survey," Energies, MDPI, vol. 11(6), pages 1-22, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Cephas Samende & Zhong Fan & Jun Cao & Renzo Fabián & Gregory N. Baltas & Pedro Rodriguez, 2023. "Battery and Hydrogen Energy Storage Control in a Smart Energy Network with Flexible Energy Demand Using Deep Reinforcement Learning," Energies, MDPI, vol. 16(19), pages 1-20, September.
    2. Chang, Weiguang & Dong, Wei & Yang, Qiang, 2023. "Day-ahead bidding strategy of cloud energy storage serving multiple heterogeneous microgrids in the electricity market," Applied Energy, Elsevier, vol. 336(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Azim, M. Imran & Tushar, Wayes & Saha, Tapan K. & Yuen, Chau & Smith, David, 2022. "Peer-to-peer kilowatt and negawatt trading: A review of challenges and recent advances in distribution networks," Renewable and Sustainable Energy Reviews, Elsevier, vol. 169(C).
    2. Meritxell Domènech Monfort & César De Jesús & Natapon Wanapinit & Niklas Hartmann, 2022. "A Review of Peer-to-Peer Energy Trading with Standard Terminology Proposal and a Techno-Economic Characterisation Matrix," Energies, MDPI, vol. 15(23), pages 1-29, November.
    3. Zhou, Yuekuan & Lund, Peter D., 2023. "Peer-to-peer energy sharing and trading of renewable energy in smart communities ─ trading pricing models, decision-making and agent-based collaboration," Renewable Energy, Elsevier, vol. 207(C), pages 177-193.
    4. Qiu, Dawei & Ye, Yujian & Papadaskalopoulos, Dimitrios & Strbac, Goran, 2021. "Scalable coordinated management of peer-to-peer energy trading: A multi-cluster deep reinforcement learning approach," Applied Energy, Elsevier, vol. 292(C).
    5. Jin, Xiaolong & Wu, Qiuwei & Jia, Hongjie, 2020. "Local flexibility markets: Literature review on concepts, models and clearing methods," Applied Energy, Elsevier, vol. 261(C).
    6. Tushar, Wayes & Yuen, Chau & Saha, Tapan K. & Morstyn, Thomas & Chapman, Archie C. & Alam, M. Jan E. & Hanif, Sarmad & Poor, H. Vincent, 2021. "Peer-to-peer energy systems for connected communities: A review of recent advances and emerging challenges," Applied Energy, Elsevier, vol. 282(PA).
    7. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    8. Wang, Zibo & Yu, Xiaodan & Mu, Yunfei & Jia, Hongjie, 2020. "A distributed Peer-to-Peer energy transaction method for diversified prosumers in Urban Community Microgrid System," Applied Energy, Elsevier, vol. 260(C).
    9. Hu, Qian & Zhu, Ziqing & Bu, Siqi & Wing Chan, Ka & Li, Fangxing, 2021. "A multi-market nanogrid P2P energy and ancillary service trading paradigm: Mechanisms and implementations," Applied Energy, Elsevier, vol. 293(C).
    10. Liu, Jia & Yang, Hongxing & Zhou, Yuekuan, 2021. "Peer-to-peer trading optimizations on net-zero energy communities with energy storage of hydrogen and battery vehicles," Applied Energy, Elsevier, vol. 302(C).
    11. Tushar, Wayes & Saha, Tapan Kumar & Yuen, Chau & Azim, M. Imran & Morstyn, Thomas & Poor, H. Vincent & Niyato, Dustin & Bean, Richard, 2020. "A coalition formation game framework for peer-to-peer energy trading," Applied Energy, Elsevier, vol. 261(C).
    12. Barbara Antonioli Mantegazzini, 2019. "Business model and market design for local digitalized electricity trading: Several insights from the NEMoGrid project," ECONOMICS AND POLICY OF ENERGY AND THE ENVIRONMENT, FrancoAngeli Editore, vol. 0(2), pages 5-24.
    13. de la Hoz, Jordi & Martín, Helena & Alonso, Alex & Carolina Luna, Adriana & Matas, José & Vasquez, Juan C. & Guerrero, Josep M., 2019. "Regulatory-framework-embedded energy management system for microgrids: The case study of the Spanish self-consumption scheme," Applied Energy, Elsevier, vol. 251(C), pages 1-1.
    14. Andreolli, Francesca & D'Alpaos, Chiara & Kort, Peter, 2023. "Does P2P Trading Favor Investments in PV-Battery Systems?," FEEM Working Papers 330498, Fondazione Eni Enrico Mattei (FEEM).
    15. Siripha Junlakarn & Phimsupha Kokchang & Kulyos Audomvongseree, 2022. "Drivers and Challenges of Peer-to-Peer Energy Trading Development in Thailand," Energies, MDPI, vol. 15(3), pages 1-25, February.
    16. Herenčić, Lin & Kirac, Mislav & Keko, Hrvoje & Kuzle, Igor & Rajšl, Ivan, 2022. "Automated energy sharing in MV and LV distribution grids within an energy community: A case for Croatian city of Križevci with a hybrid renewable system," Renewable Energy, Elsevier, vol. 191(C), pages 176-194.
    17. Wang, Zibo & Yu, Xiaodan & Mu, Yunfei & Jia, Hongjie & Jiang, Qian & Wang, Xiaoyu, 2021. "Peer-to-Peer energy trading strategy for energy balance service provider (EBSP) considering market elasticity in community microgrid," Applied Energy, Elsevier, vol. 303(C).
    18. Kuruseelan S & Vaithilingam C, 2019. "Peer-to-Peer Energy Trading of a Community Connected with an AC and DC Microgrid," Energies, MDPI, vol. 12(19), pages 1-15, September.
    19. Cephas Samende & Zhong Fan & Jun Cao & Renzo Fabián & Gregory N. Baltas & Pedro Rodriguez, 2023. "Battery and Hydrogen Energy Storage Control in a Smart Energy Network with Flexible Energy Demand Using Deep Reinforcement Learning," Energies, MDPI, vol. 16(19), pages 1-20, September.
    20. Zhang, Bidan & Du, Yang & Chen, Xiaoyang & Lim, Eng Gee & Jiang, Lin & Yan, Ke, 2022. "A novel adaptive penalty mechanism for Peer-to-Peer energy trading," Applied Energy, Elsevier, vol. 327(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:317:y:2022:i:c:s0306261922005025. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.