IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v14y2021i15p4706-d607709.html
   My bibliography  Save this article

Deep Reinforcement Learning for Hybrid Energy Storage Systems: Balancing Lead and Hydrogen Storage

Author

Listed:
  • Louis Desportes

    (Equipes Traitement de l’Information et Systèmes, UMR 8051, National Center for Scientific Research, ENSEA, CY Cergy Paris University, 95000 Cergy-Pontoise, France
    Current address: 6 avenue du Ponceau, 95000 Cergy-Pontoise, France.)

  • Inbar Fijalkow

    (Equipes Traitement de l’Information et Systèmes, UMR 8051, National Center for Scientific Research, ENSEA, CY Cergy Paris University, 95000 Cergy-Pontoise, France
    Current address: 6 avenue du Ponceau, 95000 Cergy-Pontoise, France.)

  • Pierre Andry

    (Equipes Traitement de l’Information et Systèmes, UMR 8051, National Center for Scientific Research, ENSEA, CY Cergy Paris University, 95000 Cergy-Pontoise, France
    Current address: 6 avenue du Ponceau, 95000 Cergy-Pontoise, France.)

Abstract

We address the control of a hybrid energy storage system composed of a lead battery and hydrogen storage. Powered by photovoltaic panels, it feeds a partially islanded building. We aim to minimize building carbon emissions over a long-term period while ensuring that 35% of the building consumption is powered using energy produced on site. To achieve this long-term goal, we propose to learn a control policy as a function of the building and of the storage state using a Deep Reinforcement Learning approach. We reformulate the problem to reduce the action space dimension to one. This highly improves the proposed approach performance. Given the reformulation, we propose a new algorithm, DDPG α rep , using a Deep Deterministic Policy Gradient (DDPG) to learn the policy. Once learned, the storage control is performed using this policy. Simulations show that the higher the hydrogen storage efficiency, the more effective the learning.

Suggested Citation

  • Louis Desportes & Inbar Fijalkow & Pierre Andry, 2021. "Deep Reinforcement Learning for Hybrid Energy Storage Systems: Balancing Lead and Hydrogen Storage," Energies, MDPI, vol. 14(15), pages 1-22, August.
    • Handle: RePEc:gam:jeners:v:14:y:2021:i:15:p:4706-:d:607709
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/14/15/4706/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/14/15/4706/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    2. Alexander N. Kozlov & Nikita V. Tomin & Denis N. Sidorov & Electo E. S. Lora & Victor G. Kurbatsky, 2020. "Optimal Operation Control of PV-Biomass Gasifier-Diesel-Hybrid Systems Using Reinforcement Learning Techniques," Energies, MDPI, vol. 13(10), pages 1-20, May.
    3. Diagne, Maimouna & David, Mathieu & Lauret, Philippe & Boland, John & Schmutz, Nicolas, 2013. "Review of solar irradiance forecasting methods and a proposition for small-scale insular grids," Renewable and Sustainable Energy Reviews, Elsevier, vol. 27(C), pages 65-76.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Gwanggil Jeon, 2022. "Artificial Intelligence Approaches for Energies," Energies, MDPI, vol. 15(18), pages 1-3, September.
    2. Cephas Samende & Zhong Fan & Jun Cao & Renzo Fabián & Gregory N. Baltas & Pedro Rodriguez, 2023. "Battery and Hydrogen Energy Storage Control in a Smart Energy Network with Flexible Energy Demand Using Deep Reinforcement Learning," Energies, MDPI, vol. 16(19), pages 1-20, September.
    3. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    4. Mudhafar Al-Saadi & Maher Al-Greer & Michael Short, 2021. "Strategies for Controlling Microgrid Networks with Energy Storage Systems: A Review," Energies, MDPI, vol. 14(21), pages 1-45, November.
    5. Ritu Kandari & Neeraj Neeraj & Alexander Micallef, 2022. "Review on Recent Strategies for Integrating Energy Storage Systems in Microgrids," Energies, MDPI, vol. 16(1), pages 1-24, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Feng, Cong & Zhang, Jie & Zhang, Wenqi & Hodge, Bri-Mathias, 2022. "Convolutional neural networks for intra-hour solar forecasting based on sky image sequences," Applied Energy, Elsevier, vol. 310(C).
    2. Voyant, Cyril & Motte, Fabrice & Notton, Gilles & Fouilloy, Alexis & Nivet, Marie-Laure & Duchaud, Jean-Laurent, 2018. "Prediction intervals for global solar irradiation forecasting using regression trees methods," Renewable Energy, Elsevier, vol. 126(C), pages 332-340.
    3. Tulika Saha & Sriparna Saha & Pushpak Bhattacharyya, 2020. "Towards sentiment aided dialogue policy learning for multi-intent conversations using hierarchical reinforcement learning," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-28, July.
    4. Boland, John, 2015. "Spatial-temporal forecasting of solar radiation," Renewable Energy, Elsevier, vol. 75(C), pages 607-616.
    5. Mahmoud Mahfouz & Angelos Filos & Cyrine Chtourou & Joshua Lockhart & Samuel Assefa & Manuela Veloso & Danilo Mandic & Tucker Balch, 2019. "On the Importance of Opponent Modeling in Auction Markets," Papers 1911.12816, arXiv.org.
    6. Javier Borquez & Hector Chavez & Karina A. Barbosa & Marcela Jamett & Rodrigo Acuna, 2020. "A Simple Distribution Energy Tariff under the Penetration of DG," Energies, MDPI, vol. 13(8), pages 1-17, April.
    7. Imen Azzouz & Wiem Fekih Hassen, 2023. "Optimization of Electric Vehicles Charging Scheduling Based on Deep Reinforcement Learning: A Decentralized Approach," Energies, MDPI, vol. 16(24), pages 1-18, December.
    8. Jacob W. Crandall & Mayada Oudah & Tennom & Fatimah Ishowo-Oloko & Sherief Abdallah & Jean-François Bonnefon & Manuel Cebrian & Azim Shariff & Michael A. Goodrich & Iyad Rahwan, 2018. "Cooperating with machines," Nature Communications, Nature, vol. 9(1), pages 1-12, December.
      • Abdallah, Sherief & Bonnefon, Jean-François & Cebrian, Manuel & Crandall, Jacob W. & Ishowo-Oloko, Fatimah & Oudah, Mayada & Rahwan, Iyad & Shariff, Azim & Tennom,, 2017. "Cooperating with Machines," TSE Working Papers 17-806, Toulouse School of Economics (TSE).
      • Abdallah, Sherief & Bonnefon, Jean-François & Cebrian, Manuel & Crandall, Jacob W. & Ishowo-Oloko, Fatimah & Oudah, Mayada & Rahwan, Iyad & Shariff, Azim & Tennom,, 2017. "Cooperating with Machines," IAST Working Papers 17-68, Institute for Advanced Study in Toulouse (IAST).
      • Jacob Crandall & Mayada Oudah & Fatimah Ishowo-Oloko Tennom & Fatimah Ishowo-Oloko & Sherief Abdallah & Jean-François Bonnefon & Manuel Cebrian & Azim Shariff & Michael Goodrich & Iyad Rahwan, 2018. "Cooperating with machines," Post-Print hal-01897802, HAL.
    9. Sun, Alexander Y., 2020. "Optimal carbon storage reservoir management through deep reinforcement learning," Applied Energy, Elsevier, vol. 278(C).
    10. Yassine Chemingui & Adel Gastli & Omar Ellabban, 2020. "Reinforcement Learning-Based School Energy Management System," Energies, MDPI, vol. 13(23), pages 1-21, December.
    11. Ahmed, R. & Sreeram, V. & Mishra, Y. & Arif, M.D., 2020. "A review and evaluation of the state-of-the-art in PV solar power forecasting: Techniques and optimization," Renewable and Sustainable Energy Reviews, Elsevier, vol. 124(C).
    12. John Boland & Sleiman Farah & Lei Bai, 2022. "Forecasting of Wind and Solar Farm Output in the Australian National Electricity Market: A Review," Energies, MDPI, vol. 15(1), pages 1-18, January.
    13. Woo Jae Byun & Bumkyu Choi & Seongmin Kim & Joohyun Jo, 2023. "Practical Application of Deep Reinforcement Learning to Optimal Trade Execution," FinTech, MDPI, vol. 2(3), pages 1-16, June.
    14. Lu, Yu & Xiang, Yue & Huang, Yuan & Yu, Bin & Weng, Liguo & Liu, Junyong, 2023. "Deep reinforcement learning based optimal scheduling of active distribution system considering distributed generation, energy storage and flexible load," Energy, Elsevier, vol. 271(C).
    15. Yuhong Wang & Lei Chen & Hong Zhou & Xu Zhou & Zongsheng Zheng & Qi Zeng & Li Jiang & Liang Lu, 2021. "Flexible Transmission Network Expansion Planning Based on DQN Algorithm," Energies, MDPI, vol. 14(7), pages 1-21, April.
    16. Huang, Ruchen & He, Hongwen & Gao, Miaojue, 2023. "Training-efficient and cost-optimal energy management for fuel cell hybrid electric bus based on a novel distributed deep reinforcement learning framework," Applied Energy, Elsevier, vol. 346(C).
    17. Michelle M. LaMar, 2018. "Markov Decision Process Measurement Model," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 67-88, March.
    18. Zichen Lu & Ying Yan, 2024. "Temperature Control of Fuel Cell Based on PEI-DDPG," Energies, MDPI, vol. 17(7), pages 1-19, April.
    19. Yang, Ting & Zhao, Liyuan & Li, Wei & Zomaya, Albert Y., 2021. "Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning," Energy, Elsevier, vol. 235(C).
    20. Wang, Xuan & Shu, Gequn & Tian, Hua & Wang, Rui & Cai, Jinwen, 2020. "Operation performance comparison of CCHP systems with cascade waste heat recovery systems by simulation and operation optimisation," Energy, Elsevier, vol. 206(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:14:y:2021:i:15:p:4706-:d:607709. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.