IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v284y2023ics0360544223020352.html
   My bibliography  Save this article

An efficient and privacy-preserving algorithm for multiple energy hubs scheduling with federated and matching deep reinforcement learning

Author

Listed:
  • Chen, Minghao
  • Sun, Yi
  • Xie, Zhiyuan
  • Lin, Nvgui
  • Wu, Peng

Abstract

As a significant paradigm change in reinforcement learning, federated learning (FL) has emerged to address the efficiency bottlenecks and privacy concerns of centralized training. However, the discrepancy of multi-energy demands among energy hubs (EHs) may cause undesirable instability of FL mechanisms. This study proposes a scheduling algorithm for adaptively controlling multiple EHs with combined heat and power, gas boiler, and electrical boiler. We first present the formulations of twin delayed deep deterministic policy gradient (TD3) to schedule the energy conversion under the time-varying price and demands of electricity, heat, and natural gas. The novelty of the proposed algorithm lies in developing a distributed TD3-agents training that consists of FL and matching-based agent-to-agent learning, namely matching learning (ML). Specifically, the FL is first introduced for collaborative training of EH's TD3 agents by sharing their semi-trained scheduling models. The ML innovatively proposed for addressing the training instability of FL is formulated as a matching game in which the poor-trained agents learn the scheduling knowledge from the agent who has analogous demand patterns and better reward. Simulation results display the advantage and feasibility of the proposed algorithm in terms of the speed of training convergence, energy consumption of equipment, and economic benefit.

Suggested Citation

  • Chen, Minghao & Sun, Yi & Xie, Zhiyuan & Lin, Nvgui & Wu, Peng, 2023. "An efficient and privacy-preserving algorithm for multiple energy hubs scheduling with federated and matching deep reinforcement learning," Energy, Elsevier, vol. 284(C).
  • Handle: RePEc:eee:energy:v:284:y:2023:i:c:s0360544223020352
    DOI: 10.1016/j.energy.2023.128641
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544223020352
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.energy.2023.128641?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Seyfi, Mohammad & Mehdinejad, Mehdi & Mohammadi-Ivatloo, Behnam & Shayanfar, Heidarali, 2022. "Deep learning-based scheduling of virtual energy hubs with plug-in hybrid compressed natural gas-electric vehicles," Applied Energy, Elsevier, vol. 321(C).
    2. Mostafavi Sani, Mostafa & Mostafavi Sani, Hossein & Fowler, Michael & Elkamel, Ali & Noorpoor, Alireza & Ghasemi, Amir, 2022. "Optimal energy hub development to supply heating, cooling, electricity and freshwater for a coastal urban area taking into account economic and environmental factors," Energy, Elsevier, vol. 238(PB).
    3. Pinto, Giuseppe & Piscitelli, Marco Savino & Vázquez-Canteli, José Ramón & Nagy, Zoltán & Capozzoli, Alfonso, 2021. "Coordinated energy management for a cluster of buildings through deep reinforcement learning," Energy, Elsevier, vol. 229(C).
    4. Pinto, Giuseppe & Deltetto, Davide & Capozzoli, Alfonso, 2021. "Data-driven district energy management with surrogate models and deep reinforcement learning," Applied Energy, Elsevier, vol. 304(C).
    5. Heidari, Amirreza & Maréchal, François & Khovalyg, Dolaana, 2022. "Reinforcement Learning for proactive operation of residential energy systems by learning stochastic occupant behavior and fluctuating solar energy: Balancing comfort, hygiene and energy use," Applied Energy, Elsevier, vol. 318(C).
    6. Lu, Xinhui & Li, Haobin & Zhou, Kaile & Yang, Shanlin, 2023. "Optimal load dispatch of energy hub considering uncertainties of renewable energy and demand response," Energy, Elsevier, vol. 262(PB).
    7. Bashiri Khouzestani, Leyla & Sheikh-El-Eslami, Mohammad Kazem & Salemi, Amir Hosein & Gerami Moghaddam, Iman, 2023. "Virtual smart energy Hub: A powerful tool for integrated multi energy systems operation," Energy, Elsevier, vol. 265(C).
    8. Lee, Sangyoon & Choi, Dae-Hyun, 2021. "Dynamic pricing and energy management for profit maximization in multiple smart electric vehicle charging stations: A privacy-preserving deep reinforcement learning approach," Applied Energy, Elsevier, vol. 304(C).
    9. Salehi, Javad & Namvar, Amin & Gazijahani, Farhad Samadi & Shafie-khah, Miadreza & Catalão, João P.S., 2022. "Effect of power-to-gas technology in energy hub optimal operation and gas network congestion reduction," Energy, Elsevier, vol. 240(C).
    10. Zou, Juan & Yang, Xu & Liu, Zhongbing & Liu, Jiangyang & Zhang, Ling & Zheng, Jinhua, 2021. "Multiobjective bilevel optimization algorithm based on preference selection to solve energy hub system planning problems," Energy, Elsevier, vol. 232(C).
    11. Mansouri, Seyed Amir & Ahmarinejad, Amir & Javadi, Mohammad Sadegh & Catalão, João P.S., 2020. "Two-stage stochastic framework for energy hubs planning considering demand response programs," Energy, Elsevier, vol. 206(C).
    12. Fernández, Joaquín Delgado & Menci, Sergio Potenciano & Lee, Chul Min & Rieger, Alexander & Fridgen, Gilbert, 2022. "Privacy-preserving federated learning for residential short-term load forecasting," Applied Energy, Elsevier, vol. 326(C).
    13. Ghappani, Seyyed Aliasghar & Karimi, Ali, 2023. "Optimal operation framework of an energy hub with combined heat, hydrogen, and power (CHHP) system based on ammonia," Energy, Elsevier, vol. 266(C).
    14. Li, Chengzhou & Wang, Ningling & Wang, Zhuo & Dou, Xiaoxiao & Zhang, Yumeng & Yang, Zhiping & Maréchal, François & Wang, Ligang & Yang, Yongping, 2022. "Energy hub-based optimal planning framework for user-level integrated energy systems: Considering synergistic effects under multiple uncertainties," Applied Energy, Elsevier, vol. 307(C).
    15. Cai, Qingsen & Luo, XingQi & Wang, Peng & Gao, Chunyang & Zhao, Peiyu, 2022. "Hybrid model-driven and data-driven control method based on machine learning algorithm in energy hub and application," Applied Energy, Elsevier, vol. 305(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Abdulnasser, Ghada & Ali, Abdelfatah & Shaaban, Mostafa F. & Mohamed, Essam E.M., 2024. "Optimal resource allocation and operation for smart energy hubs considering hydrogen storage systems and electric vehicles," Energy, Elsevier, vol. 295(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Azimian, Mahdi & Amir, Vahid & Mohseni, Soheil & Brent, Alan C. & Bazmohammadi, Najmeh & Guerrero, Josep M., 2022. "Optimal Investment Planning of Bankable Multi-Carrier Microgrid Networks," Applied Energy, Elsevier, vol. 328(C).
    2. Mu, Yunfei & Xu, Yurui & Cao, Yan & Chen, Wanqing & Jia, Hongjie & Yu, Xiaodan & Jin, Xiaolong, 2022. "A two-stage scheduling method for integrated community energy system based on a hybrid mechanism and data-driven model," Applied Energy, Elsevier, vol. 323(C).
    3. Liu, Tianhao & Tian, Jun & Zhu, Hongyu & Goh, Hui Hwang & Liu, Hui & Wu, Thomas & Zhang, Dongdong, 2023. "Key technologies and developments of multi-energy system: Three-layer framework, modelling and optimisation," Energy, Elsevier, vol. 277(C).
    4. Chen, Minghao & Xie, Zhiyuan & Sun, Yi & Zheng, Shunlin, 2023. "The predictive management in campus heating system based on deep reinforcement learning and probabilistic heat demands forecasting," Applied Energy, Elsevier, vol. 350(C).
    5. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    6. Bu, Yuntao & Yu, Hao & Ji, Haoran & Song, Guanyu & Xu, Jing & Li, Juan & Zhao, Jinli & Li, Peng, 2024. "Hybrid data-driven operation method for demand response of community integrated energy systems utilizing virtual and physical energy storage," Applied Energy, Elsevier, vol. 366(C).
    7. Charalampos Rafail Lazaridis & Iakovos Michailidis & Georgios Karatzinis & Panagiotis Michailidis & Elias Kosmatopoulos, 2024. "Evaluating Reinforcement Learning Algorithms in Residential Energy Saving and Comfort Management," Energies, MDPI, vol. 17(3), pages 1-33, January.
    8. Nweye, Kingsley & Sankaranarayanan, Siva & Nagy, Zoltan, 2023. "MERLIN: Multi-agent offline and transfer learning for occupant-centric operation of grid-interactive communities," Applied Energy, Elsevier, vol. 346(C).
    9. Hua, Zhihao & Li, Jiayong & Zhou, Bin & Or, Siu Wing & Chan, Ka Wing & Meng, Yunfan, 2022. "Game-theoretic multi-energy trading framework for strategic biogas-solar renewable energy provider with heterogeneous consumers," Energy, Elsevier, vol. 260(C).
    10. Pinto, Giuseppe & Kathirgamanathan, Anjukan & Mangina, Eleni & Finn, Donal P. & Capozzoli, Alfonso, 2022. "Enhancing energy management in grid-interactive buildings: A comparison among cooperative and coordinated architectures," Applied Energy, Elsevier, vol. 310(C).
    11. Dimitrios Vamvakas & Panagiotis Michailidis & Christos Korkas & Elias Kosmatopoulos, 2023. "Review and Evaluation of Reinforcement Learning Frameworks on Smart Grid Applications," Energies, MDPI, vol. 16(14), pages 1-38, July.
    12. He, Shuaijia & Gao, Hongjun & Chen, Zhe & Liu, Junyong & Zhao, Liang & Wu, Gang & Xu, Song, 2022. "Low-carbon distribution system planning considering flexible support of zero-carbon energy station," Energy, Elsevier, vol. 244(PB).
    13. Ayas Shaqour & Aya Hagishima, 2022. "Systematic Review on Deep Reinforcement Learning-Based Energy Management for Different Building Types," Energies, MDPI, vol. 15(22), pages 1-27, November.
    14. Liao, Zitian & Liao, Xiaoqun & Khakichi, Aroos, 2024. "Optimum planning of energy hub with participation in electricity market and heat markets and application of integrated load response program with improved particle swarm algorithm," Energy, Elsevier, vol. 286(C).
    15. Coraci, Davide & Brandi, Silvio & Hong, Tianzhen & Capozzoli, Alfonso, 2023. "Online transfer learning strategy for enhancing the scalability and deployment of deep reinforcement learning control in smart buildings," Applied Energy, Elsevier, vol. 333(C).
    16. Noorollahi, Younes & Golshanfard, Aminabbas & Hashemi-Dezaki, Hamed, 2022. "A scenario-based approach for optimal operation of energy hub under different schemes and structures," Energy, Elsevier, vol. 251(C).
    17. Fan, Linyuan & Ji, Dandan & Lin, Geng & Lin, Peng & Liu, Lixi, 2023. "Information gap-based multi-objective optimization of a virtual energy hub plant considering a developed demand response model," Energy, Elsevier, vol. 276(C).
    18. Qiao, Yiyang & Hu, Fan & Xiong, Wen & Guo, Zihao & Zhou, Xiaoguang & Li, Yajun, 2023. "Multi-objective optimization of integrated energy system considering installation configuration," Energy, Elsevier, vol. 263(PC).
    19. Hunt, Julian David & Nascimento, Andreas & Zakeri, Behnam & Barbosa, Paulo Sérgio Franco, 2022. "Hydrogen Deep Ocean Link: a global sustainable interconnected energy grid," Energy, Elsevier, vol. 249(C).
    20. Chankook Park & Wan Gyu Heo & Myung Eun Lee, 2024. "Study on Consumers’ Perceived Benefits and Risks of Smart Energy System," International Journal of Energy Economics and Policy, Econjournals, vol. 14(3), pages 288-300, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:284:y:2023:i:c:s0360544223020352. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.