IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v13y2020i19p5027-d418674.html
   My bibliography  Save this article

Agent-Based Energy Sharing Mechanism Using Deep Deterministic Policy Gradient Algorithm

Author

Listed:
  • Yi Kuang

    (School of Electric Engineering, Xi’an Jiaotong University, Xi’an 710000, China)

  • Xiuli Wang

    (School of Electric Engineering, Xi’an Jiaotong University, Xi’an 710000, China)

  • Hongyang Zhao

    (School of Electric Engineering, Xi’an Jiaotong University, Xi’an 710000, China)

  • Yijun Huang

    (State Grid Shanghai Municipal Electric Power Company, Shanghai 200122, China)

  • Xianlong Chen

    (School of Electric Engineering, Xi’an Jiaotong University, Xi’an 710000, China)

  • Xifan Wang

    (School of Electric Engineering, Xi’an Jiaotong University, Xi’an 710000, China)

Abstract

Balancing energy generation and consumption is essential for smoothing the power grids. The mismatch between energy supply and demand would not only increase the cost on both sides, but also has a great impact on the stability of the system. This paper proposes a novel energy sharing mechanism (ESM) to facilitate the consumption of local energy. With the help of the ESM, multiple prosumers have an opportunity to share surplus energy with neighboring prosumers. The problem is formulated as a leader–follower framework based on the Stackelberg game theory. To address the aforementioned problems, a deep deterministic policy gradient (DDPG) is applied to solve the Nash equilibrium (NE). The numerical results demonstrate that the proposed method is more stable than the conventional reinforcement learning (RL) algorithm. Moreover, the proposed method can converge to NE and find a relatively good energy sharing (ES) pricing strategy without knowing the specific system information. In short, it is notable that the proposed ESM can be seen as a win–win strategy for both prosumers and the power system.

Suggested Citation

  • Yi Kuang & Xiuli Wang & Hongyang Zhao & Yijun Huang & Xianlong Chen & Xifan Wang, 2020. "Agent-Based Energy Sharing Mechanism Using Deep Deterministic Policy Gradient Algorithm," Energies, MDPI, vol. 13(19), pages 1-20, September.
    • Handle: RePEc:gam:jeners:v:13:y:2020:i:19:p:5027-:d:418674
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/13/19/5027/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/13/19/5027/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Zamani, Ali Ghahgharaee & Zakariazadeh, Alireza & Jadid, Shahram, 2016. "Day-ahead resource scheduling of a renewable energy based virtual power plant," Applied Energy, Elsevier, vol. 169(C), pages 324-340.
    2. Lu, Renzhi & Hong, Seung Ho & Zhang, Xiongfeng, 2018. "A Dynamic pricing demand response algorithm for smart grid: Reinforcement learning approach," Applied Energy, Elsevier, vol. 220(C), pages 220-230.
    3. Hua, Haochen & Qin, Yuchao & Hao, Chuantong & Cao, Junwei, 2019. "Optimal energy management strategies for energy Internet via deep reinforcement learning approach," Applied Energy, Elsevier, vol. 239(C), pages 598-609.
    4. Liu, Youbo & Zuo, Kunyu & Liu, Xueqin (Amy) & Liu, Junyong & Kennedy, Jason M., 2018. "Dynamic pricing for decentralized energy trading in micro-grids," Applied Energy, Elsevier, vol. 228(C), pages 689-699.
    5. Roth, Alvin E. & Erev, Ido, 1995. "Learning in extensive-form games: Experimental data and simple dynamic models in the intermediate term," Games and Economic Behavior, Elsevier, vol. 8(1), pages 164-212.
    6. Xiong, Rui & Cao, Jiayi & Yu, Quanqing, 2018. "Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle," Applied Energy, Elsevier, vol. 211(C), pages 538-548.
    7. Lu, Qing & Lü, Shuaikang & Leng, Yajun, 2019. "A Nash-Stackelberg game approach in regional energy market considering users’ integrated demand response," Energy, Elsevier, vol. 175(C), pages 456-470.
    8. Lu, Renzhi & Hong, Seung Ho, 2019. "Incentive-based demand response for smart grid with reinforcement learning and deep neural network," Applied Energy, Elsevier, vol. 236(C), pages 937-949.
    9. Rocchetta, R. & Bellani, L. & Compare, M. & Zio, E. & Patelli, E., 2019. "A reinforcement learning framework for optimal operation and maintenance of power grids," Applied Energy, Elsevier, vol. 241(C), pages 291-301.
    10. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    11. Kofinas, P. & Dounis, A.I. & Vouros, G.A., 2018. "Fuzzy Q-Learning for multi-agent decentralized energy management in microgrids," Applied Energy, Elsevier, vol. 219(C), pages 53-67.
    12. Thomas Morstyn & Niall Farrell & Sarah J. Darby & Malcolm D. McCulloch, 2018. "Using peer-to-peer energy-trading platforms to incentivize prosumers to form federated power plants," Nature Energy, Nature, vol. 3(2), pages 94-101, February.
    13. Wang, Jianxiao & Zhong, Haiwang & Wu, Chenye & Du, Ershun & Xia, Qing & Kang, Chongqing, 2019. "Incentivizing distributed energy resource aggregation in energy and capacity markets: An energy sharing scheme and mechanism design," Applied Energy, Elsevier, vol. 252(C), pages 1-1.
    14. Zhang, Chenghua & Wu, Jianzhong & Zhou, Yue & Cheng, Meng & Long, Chao, 2018. "Peer-to-Peer energy trading in a Microgrid," Applied Energy, Elsevier, vol. 220(C), pages 1-12.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lu, Ying & Fang, Sidun & Niu, Tao & Liao, Ruijin, 2023. "Energy-transport scheduling for green vehicles in seaport areas: A review on operation models," Renewable and Sustainable Energy Reviews, Elsevier, vol. 184(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    2. Kong, Xiangyu & Kong, Deqian & Yao, Jingtao & Bai, Linquan & Xiao, Jie, 2020. "Online pricing of demand response based on long short-term memory and reinforcement learning," Applied Energy, Elsevier, vol. 271(C).
    3. Qiu, Dawei & Ye, Yujian & Papadaskalopoulos, Dimitrios & Strbac, Goran, 2021. "Scalable coordinated management of peer-to-peer energy trading: A multi-cluster deep reinforcement learning approach," Applied Energy, Elsevier, vol. 292(C).
    4. Tushar, Wayes & Yuen, Chau & Saha, Tapan K. & Morstyn, Thomas & Chapman, Archie C. & Alam, M. Jan E. & Hanif, Sarmad & Poor, H. Vincent, 2021. "Peer-to-peer energy systems for connected communities: A review of recent advances and emerging challenges," Applied Energy, Elsevier, vol. 282(PA).
    5. Lu, Renzhi & Li, Yi-Chang & Li, Yuting & Jiang, Junhui & Ding, Yuemin, 2020. "Multi-agent deep reinforcement learning based demand response for discrete manufacturing systems energy management," Applied Energy, Elsevier, vol. 276(C).
    6. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    7. Correa-Jullian, Camila & López Droguett, Enrique & Cardemil, José Miguel, 2020. "Operation scheduling in a solar thermal system: A reinforcement learning-based framework," Applied Energy, Elsevier, vol. 268(C).
    8. Nyong-Bassey, Bassey Etim & Giaouris, Damian & Patsios, Charalampos & Papadopoulou, Simira & Papadopoulos, Athanasios I. & Walker, Sara & Voutetakis, Spyros & Seferlis, Panos & Gadoue, Shady, 2020. "Reinforcement learning based adaptive power pinch analysis for energy management of stand-alone hybrid energy storage systems considering uncertainty," Energy, Elsevier, vol. 193(C).
    9. Ibrahim, Muhammad Sohail & Dong, Wei & Yang, Qiang, 2020. "Machine learning driven smart electric power systems: Current trends and new perspectives," Applied Energy, Elsevier, vol. 272(C).
    10. Chen, Yang & Park, Byungkwon & Kou, Xiao & Hu, Mengqi & Dong, Jin & Li, Fangxing & Amasyali, Kadir & Olama, Mohammed, 2020. "A comparison study on trading behavior and profit distribution in local energy transaction games," Applied Energy, Elsevier, vol. 280(C).
    11. Zhu, Ziqing & Hu, Ze & Chan, Ka Wing & Bu, Siqi & Zhou, Bin & Xia, Shiwei, 2023. "Reinforcement learning in deregulated energy market: A comprehensive review," Applied Energy, Elsevier, vol. 329(C).
    12. Pinto, Giuseppe & Deltetto, Davide & Capozzoli, Alfonso, 2021. "Data-driven district energy management with surrogate models and deep reinforcement learning," Applied Energy, Elsevier, vol. 304(C).
    13. Seongwoo Lee & Joonho Seon & Byungsun Hwang & Soohyun Kim & Youngghyu Sun & Jinyoung Kim, 2024. "Recent Trends and Issues of Energy Management Systems Using Machine Learning," Energies, MDPI, vol. 17(3), pages 1-24, January.
    14. Park, Keonwoo & Moon, Ilkyeong, 2022. "Multi-agent deep reinforcement learning approach for EV charging scheduling in a smart grid," Applied Energy, Elsevier, vol. 328(C).
    15. Zhao, Liyuan & Yang, Ting & Li, Wei & Zomaya, Albert Y., 2022. "Deep reinforcement learning-based joint load scheduling for household multi-energy system," Applied Energy, Elsevier, vol. 324(C).
    16. Zheng, Lingwei & Wu, Hao & Guo, Siqi & Sun, Xinyu, 2023. "Real-time dispatch of an integrated energy system based on multi-stage reinforcement learning with an improved action-choosing strategy," Energy, Elsevier, vol. 277(C).
    17. Antonopoulos, Ioannis & Robu, Valentin & Couraud, Benoit & Kirli, Desen & Norbu, Sonam & Kiprakis, Aristides & Flynn, David & Elizondo-Gonzalez, Sergio & Wattam, Steve, 2020. "Artificial intelligence and machine learning approaches to energy demand-side response: A systematic review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 130(C).
    18. Xia, Yuanxing & Xu, Qingshan & Li, Fangxing, 2023. "Grid-friendly pricing mechanism for peer-to-peer energy sharing market diffusion in communities," Applied Energy, Elsevier, vol. 334(C).
    19. Xie, Jiahan & Ajagekar, Akshay & You, Fengqi, 2023. "Multi-Agent attention-based deep reinforcement learning for demand response in grid-responsive buildings," Applied Energy, Elsevier, vol. 342(C).
    20. Yamashita, Daniela Yassuda & Vechiu, Ionel & Gaubert, Jean-Paul, 2020. "A review of hierarchical control for building microgrids," Renewable and Sustainable Energy Reviews, Elsevier, vol. 118(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:13:y:2020:i:19:p:5027-:d:418674. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.