IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v328y2022ics0306261922013708.html
   My bibliography  Save this article

Deep reinforcement learning based optimization for a tightly coupled nuclear renewable integrated energy system

Author

Listed:
  • Yi, Zonggen
  • Luo, Yusheng
  • Westover, Tyler
  • Katikaneni, Sravya
  • Ponkiya, Binaka
  • Sah, Suba
  • Mahmud, Sadab
  • Raker, David
  • Javaid, Ahmad
  • Heben, Michael J.
  • Khanna, Raghav

Abstract

New ways to integrate energy systems to maximize efficiency are being sought to meet carbon emissions goals. Nuclear-renewable integrated energy system (NR-IES) concepts are a leading solution that couples a nuclear power plant with renewable energy, hydrogen generation plants, and energy storage systems, such that thermal and electrical power are dispatchable to fulfill grid-flexibility requirements while also producing hydrogen and maximizing revenue. This paper introduces a deep reinforcement learning (DRL)-based framework to address the complex decision-making tasks for NR-IES. The objective is to maximize revenue by generating and selling hydrogen and electricity simultaneously according to their time-varying prices while keeping the energy flow in the subsystems in balance. A Python-based simulator for a NR-IES concept has been developed to integrate with OpenAI Gym and Ray/RLlib to enable an efficient and flexible computational framework for DRL research and development. Three state-of-the-art DRL algorithms have been investigated, including two-delayed deep deterministic policy gradient (TD3), soft-actor critic (SAC), proximal policy optimization (PPO), to illustrate DRL’s superiority for controlling NR-IES by comparing it with a conventional control approach, particle swarm optimization (PSO). In this effort, PPO has shown more-stable performance and also better generalization capability than SAC and TD3. Comparisons with PSO have demonstrated that, on average, PPO can achieve 13.9% more mean episode returns from the training process and 29.4% more mean episode returns from the testing process when different hydrogen-production targets are applied.

Suggested Citation

  • Yi, Zonggen & Luo, Yusheng & Westover, Tyler & Katikaneni, Sravya & Ponkiya, Binaka & Sah, Suba & Mahmud, Sadab & Raker, David & Javaid, Ahmad & Heben, Michael J. & Khanna, Raghav, 2022. "Deep reinforcement learning based optimization for a tightly coupled nuclear renewable integrated energy system," Applied Energy, Elsevier, vol. 328(C).
  • Handle: RePEc:eee:appene:v:328:y:2022:i:c:s0306261922013708
    DOI: 10.1016/j.apenergy.2022.120113
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261922013708
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.apenergy.2022.120113?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Pinto, Giuseppe & Piscitelli, Marco Savino & Vázquez-Canteli, José Ramón & Nagy, Zoltán & Capozzoli, Alfonso, 2021. "Coordinated energy management for a cluster of buildings through deep reinforcement learning," Energy, Elsevier, vol. 229(C).
    2. Xin Zhang & Jianhua Yang & Weizhou Wang & Man Zhang & Tianjun Jing, 2018. "Integrated Optimal Dispatch of a Rural Micro-Energy-Grid with Multi-Energy Stream Based on Model Predictive Control," Energies, MDPI, vol. 11(12), pages 1-23, December.
    3. Zecca, Antonio & Chiari, Luca, 2010. "Fossil-fuel constraints on global warming," Energy Policy, Elsevier, vol. 38(1), pages 1-3, January.
    4. Wang, Haichao & Yin, Wusong & Abdollahi, Elnaz & Lahdelma, Risto & Jiao, Wenling, 2015. "Modelling and optimization of CHP based district heating system with renewable energy production and energy storage," Applied Energy, Elsevier, vol. 159(C), pages 401-421.
    5. Veeraya Imcharoenkul & Surachai Chaitusaney, 2021. "Optimal Variable Renewable Energy Generation Schedules Considering Market Prices and System Operational Constraints," Energies, MDPI, vol. 14(17), pages 1-18, August.
    6. Wang, Luhao & Li, Qiqiang & Ding, Ran & Sun, Mingshun & Wang, Guirong, 2017. "Integrated scheduling of energy supply and demand in microgrids under uncertainty: A robust multi-objective optimization approach," Energy, Elsevier, vol. 130(C), pages 1-14.
    7. Oriol Vinyals & Igor Babuschkin & Wojciech M. Czarnecki & Michaël Mathieu & Andrew Dudzik & Junyoung Chung & David H. Choi & Richard Powell & Timo Ewalds & Petko Georgiev & Junhyuk Oh & Dan Horgan & M, 2019. "Grandmaster level in StarCraft II using multi-agent reinforcement learning," Nature, Nature, vol. 575(7782), pages 350-354, November.
    8. Kim, Jong Suk & Boardman, Richard D. & Bragg-Sitton, Shannon M., 2018. "Dynamic performance analysis of a high-temperature steam electrolysis plant integrated within nuclear-renewable hybrid energy systems," Applied Energy, Elsevier, vol. 228(C), pages 2090-2110.
    9. Bogdan, Željko & Kopjar, Damir, 2006. "Improvement of the cogeneration plant economy by using heat accumulator," Energy, Elsevier, vol. 31(13), pages 2285-2292.
    10. Rongxiang Yuan & Jun Ye & Jiazhi Lei & Timing Li, 2016. "Integrated Combined Heat and Power System Dispatch Considering Electrical and Thermal Energy Storage," Energies, MDPI, vol. 9(6), pages 1-17, June.
    11. Epiney, A. & Rabiti, C. & Talbot, P. & Alfonsi, A., 2020. "Economic analysis of a nuclear hybrid energy system in a stochastic environment including wind turbines in an electricity grid," Applied Energy, Elsevier, vol. 260(C).
    12. Rezaie, Behnaz & Reddy, Bale V. & Rosen, Marc A., 2014. "Energy analysis of thermal energy storages with grid configurations," Applied Energy, Elsevier, vol. 117(C), pages 54-61.
    13. Yang, Ting & Zhao, Liyuan & Li, Wei & Zomaya, Albert Y., 2021. "Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning," Energy, Elsevier, vol. 235(C).
    14. Upadhyay, Subho & Sharma, M.P., 2014. "A review on configurations, control and sizing methodologies of hybrid energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 38(C), pages 47-63.
    15. Silvente, Javier & Papageorgiou, Lazaros G., 2017. "An MILP formulation for the optimal management of microgrids with task interruptions," Applied Energy, Elsevier, vol. 206(C), pages 1131-1146.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Keerthana Sivamayil & Elakkiya Rajasekar & Belqasem Aljafari & Srete Nikolovski & Subramaniyaswamy Vairavasundaram & Indragandhi Vairavasundaram, 2023. "A Systematic Study on Reinforcement Learning Based Applications," Energies, MDPI, vol. 16(3), pages 1-23, February.
    2. Chen, Yan & Zhang, Ruiqian & Lyu, Jiayi & Hou, Yuqi, 2024. "AI and Nuclear: A perfect intersection of danger and potential?," Energy Economics, Elsevier, vol. 133(C).
    3. Zhou, Yanting & Ma, Zhongjing & Shi, Xingyu & Zou, Suli, 2024. "Multi-agent optimal scheduling for integrated energy system considering the global carbon emission constraint," Energy, Elsevier, vol. 288(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rongxiang Yuan & Jun Ye & Jiazhi Lei & Timing Li, 2016. "Integrated Combined Heat and Power System Dispatch Considering Electrical and Thermal Energy Storage," Energies, MDPI, vol. 9(6), pages 1-17, June.
    2. Yang, Ting & Zhao, Liyuan & Li, Wei & Zomaya, Albert Y., 2021. "Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning," Energy, Elsevier, vol. 235(C).
    3. Haoyu Wang & Roberto Ponciroli & Andrea Alfonsi & Paul W. Talbot & Thomas W. Elmer & Aaron S. Epiney & Richard B. Vilim, 2024. "Feasible Actuator Range Modifier (FARM), a Tool Aiding the Solution of Unit Dispatch Problems for Advanced Energy Systems," Energies, MDPI, vol. 17(12), pages 1-36, June.
    4. Nweye, Kingsley & Sankaranarayanan, Siva & Nagy, Zoltan, 2023. "MERLIN: Multi-agent offline and transfer learning for occupant-centric operation of grid-interactive communities," Applied Energy, Elsevier, vol. 346(C).
    5. Wang, Cheng & Liu, Chuang & Lin, Yuzhang & Bi, Tianshu, 2020. "Day-ahead dispatch of integrated electric-heat systems considering weather-parameter-driven residential thermal demands," Energy, Elsevier, vol. 203(C).
    6. Zhang, Youjun & Hao, Junhong & Ge, Zhihua & Zhang, Fuxiang & Du, Xiaoze, 2021. "Optimal clean heating mode of the integrated electricity and heat energy system considering the comprehensive energy-carbon price," Energy, Elsevier, vol. 231(C).
    7. Qin, Yuxiao & Liu, Pei & Li, Zheng, 2022. "Multi-timescale hierarchical scheduling of an integrated energy system considering system inertia," Renewable and Sustainable Energy Reviews, Elsevier, vol. 169(C).
    8. Hang Liu & Yongcheng Wang & Shilin Nie & Yi Wang & Yu Chen, 2022. "Multistage Economic Scheduling Model of Micro-Energy Grids Considering Flexible Capacity Allocation," Sustainability, MDPI, vol. 14(15), pages 1-29, July.
    9. Guo, Chenyu & Wang, Xin & Zheng, Yihui & Zhang, Feng, 2022. "Real-time optimal energy management of microgrid with uncertainties based on deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    10. Fontenot, Hannah & Dong, Bing, 2019. "Modeling and control of building-integrated microgrids for optimal energy management – A review," Applied Energy, Elsevier, vol. 254(C).
    11. Ding, Zeyu & Hou, Hongjuan & Duan, Liqiang & Huang, Chang & Hu, Eric & Yu, Gang & Zhang, Yumeng & Zhang, Nan, 2021. "Simulation study on a novel solar aided combined heat and power system for heat-power decoupling," Energy, Elsevier, vol. 220(C).
    12. Yan, Rujing & Wang, Jiangjiang & Huo, Shuojie & Zhang, Jing & Tang, Saiqiu & Yang, Mei, 2023. "Comparative study for four technologies on flexibility improvement and renewable energy accommodation of combined heat and power system," Energy, Elsevier, vol. 263(PE).
    13. Jimenez-Navarro, Juan-Pablo & Kavvadias, Konstantinos & Filippidou, Faidra & Pavičević, Matija & Quoilin, Sylvain, 2020. "Coupling the heating and power sectors: The role of centralised combined heat and power plants and district heat in a European decarbonised power system," Applied Energy, Elsevier, vol. 270(C).
    14. Liu, Miaomiao & Liu, Ming & Wang, Yu & Chen, Weixiong & Yan, Junjie, 2021. "Thermodynamic optimization of coal-fired combined heat and power (CHP) systems integrated with steam ejectors to achieve heat–power decoupling," Energy, Elsevier, vol. 229(C).
    15. Wang, Jiawei & You, Shi & Zong, Yi & Cai, Hanmin & Træholt, Chresten & Dong, Zhao Yang, 2019. "Investigation of real-time flexibility of combined heat and power plants in district heating applications," Applied Energy, Elsevier, vol. 237(C), pages 196-209.
    16. Wang, Jiangjiang & Deng, Hongda & Qi, Xiaoling, 2022. "Cost-based site and capacity optimization of multi-energy storage system in the regional integrated energy networks," Energy, Elsevier, vol. 261(PA).
    17. Stančin, H. & Mikulčić, H. & Wang, X. & Duić, N., 2020. "A review on alternative fuels in future energy system," Renewable and Sustainable Energy Reviews, Elsevier, vol. 128(C).
    18. Wang, Dongxiao & Qiu, Jing & Reedman, Luke & Meng, Ke & Lai, Loi Lei, 2018. "Two-stage energy management for networked microgrids with high renewable penetration," Applied Energy, Elsevier, vol. 226(C), pages 39-48.
    19. Sadeghian, Omid & Mohammadpour Shotorbani, Amin & Mohammadi-Ivatloo, Behnam & Sadiq, Rehan & Hewage, Kasun, 2021. "Risk-averse maintenance scheduling of generation units in combined heat and power systems with demand response," Reliability Engineering and System Safety, Elsevier, vol. 216(C).
    20. Verda, Vittorio & Colella, Francesco, 2011. "Primary energy savings through thermal storage in district heating networks," Energy, Elsevier, vol. 36(7), pages 4278-4286.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:328:y:2022:i:c:s0306261922013708. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.