IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v407y2026ics0306261925019877.html

Adaptive home energy management based on PI-DRA-PPO for integrated electricity, hydrogen, and heat storage systems

Author

Listed:
  • Liu, Meng
  • Bu, Siqi
  • Zhu, Ziqing
  • Zhou, Bin

Abstract

The Home Energy Management System (HEMS) is becoming increasingly vital for mitigating residential energy expenditures and maximizing the utilization of renewable energy sources through coordinated scheduling of various energy storage devices. Integrating hydrogen storage into HEMS has also garnered growing attention for its ability to enhance household energy management flexibility and reliability and support the long-term goal of net-zero energy homes (NZEHs). However, challenges such as uncertain electricity prices and intermittent photovoltaic (PV) outputs frequently result in inefficient management strategies, leading to PV curtailment, reduced energy efficiency, and unreliable energy supply. To this end, this paper proposes a novel data-informed Deep Reinforcement Learning (DRL) framework, termed Prediction Intervals Constrained Dynamic Reward Adaptive Proximity Policy Optimization (PI-DRA-PPO), to optimize household energy management by enhancing both energy supply reliability and economic efficiency. A probabilistic prediction module combining Bi-directional Gated Recursive Unit with Coded Engineering (BiGRU-CE) and the Kernel Density Estimation (KDE) is employed to construct reliable prediction intervals (PIs), which serve as safety constraints embedded within the state space of the PI-DRA-PPO algorithm. Additionally, the Hydrogen Energy Storage System (HESS) is strategically integrated with the Thermal Energy Storage System (TESS) and Battery Energy Storage System (BESS) within the proposed HEMS for the coordinated management of multi-timescale energy dynamics induced by the intermittency of renewable energy sources. Compared with basic PPO and other DRL methods, the proposed method can achieve up to a 62.86 % reduction in total energy costs over a representative four-month period covering all seasons. The approach improves the effectiveness of energy scheduling and strengthens the temporal and long-term reliability of HEMS in the face of dynamic energy environments.

Suggested Citation

  • Liu, Meng & Bu, Siqi & Zhu, Ziqing & Zhou, Bin, 2026. "Adaptive home energy management based on PI-DRA-PPO for integrated electricity, hydrogen, and heat storage systems," Applied Energy, Elsevier, vol. 407(C).
    • Handle: RePEc:eee:appene:v:407:y:2026:i:c:s0306261925019877
    DOI: 10.1016/j.apenergy.2025.127257
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261925019877
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2025.127257?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Liu, Yanli & Wang, Junyi & Liu, Liqi, 2024. "Physics-informed reinforcement learning for probabilistic wind power forecasting under extreme events," Applied Energy, Elsevier, vol. 376(PA).
    2. Zhou, Yuekuan & Zheng, Siqian, 2020. "Machine-learning based hybrid demand-side controller for high-rise office buildings with high energy flexibilities," Applied Energy, Elsevier, vol. 262(C).
    3. Pang, Zhihong & Niu, Fuxin & O’Neill, Zheng, 2020. "Solar radiation prediction using recurrent neural network and artificial neural network: A case study with comparisons," Renewable Energy, Elsevier, vol. 156(C), pages 279-289.
    4. Pinto, Giuseppe & Piscitelli, Marco Savino & Vázquez-Canteli, José Ramón & Nagy, Zoltán & Capozzoli, Alfonso, 2021. "Coordinated energy management for a cluster of buildings through deep reinforcement learning," Energy, Elsevier, vol. 229(C).
    5. Hua, Haochen & Qin, Yuchao & Hao, Chuantong & Cao, Junwei, 2019. "Optimal energy management strategies for energy Internet via deep reinforcement learning approach," Applied Energy, Elsevier, vol. 239(C), pages 598-609.
    6. Dorokhova, Marina & Martinson, Yann & Ballif, Christophe & Wyrsch, Nicolas, 2021. "Deep reinforcement learning control of electric vehicle charging in the presence of photovoltaic generation," Applied Energy, Elsevier, vol. 301(C).
    7. Benali, L. & Notton, G. & Fouilloy, A. & Voyant, C. & Dizene, R., 2019. "Solar radiation forecasting using artificial neural network and random forest methods: Application to normal beam, horizontal diffuse and global components," Renewable Energy, Elsevier, vol. 132(C), pages 871-884.
    8. Zhou, Heng & Zheng, Peijun & Dong, Jiuqing & Liu, Jiang & Nakanishi, Yosuke, 2024. "Interpretable feature selection and deep learning for short-term probabilistic PV power forecasting in buildings using local monitoring data," Applied Energy, Elsevier, vol. 376(PA).
    9. Lork, Clement & Li, Wen-Tai & Qin, Yan & Zhou, Yuren & Yuen, Chau & Tushar, Wayes & Saha, Tapan K., 2020. "An uncertainty-aware deep reinforcement learning framework for residential air conditioning energy management," Applied Energy, Elsevier, vol. 276(C).
    10. Shi, Tao & Xu, Chang & Dong, Wenhao & Zhou, Hangyu & Bokhari, Awais & Klemeš, Jiří Jaromír & Han, Ning, 2023. "Research on energy management of hydrogen electric coupling system based on deep reinforcement learning," Energy, Elsevier, vol. 282(C).
    11. Maestre, V.M. & Ortiz, A. & Ortiz, I., 2024. "Sustainable and self-sufficient social home through a combined PV‑hydrogen pilot," Applied Energy, Elsevier, vol. 363(C).
    12. Bio Gassi, Karim & Baysal, Mustafa, 2023. "Improving real-time energy decision-making model with an actor-critic agent in modern microgrids with energy storage devices," Energy, Elsevier, vol. 263(PE).
    13. Dai, Yeming & Wang, Yanxin & Leng, Mingming & Yang, Xinyu & Zhou, Qiong, 2022. "LOWESS smoothing and Random Forest based GRU model: A short-term photovoltaic power generation forecasting method," Energy, Elsevier, vol. 256(C).
    14. Makolo, Peter & Zamora, Ramon & Lie, Tek-Tjing, 2021. "The role of inertia for grid flexibility under high penetration of variable renewables - A review of challenges and solutions," Renewable and Sustainable Energy Reviews, Elsevier, vol. 147(C).
    15. Xu, Biao & Luan, Wenpeng & Yang, Jing & Zhao, Bochao & Long, Chao & Ai, Qian & Xiang, Jiani, 2024. "Integrated three-stage decentralized scheduling for virtual power plants: A model-assisted multi-agent reinforcement learning method," Applied Energy, Elsevier, vol. 376(PA).
    16. Du, Yan & Zandi, Helia & Kotevska, Olivera & Kurte, Kuldeep & Munk, Jeffery & Amasyali, Kadir & Mckee, Evan & Li, Fangxing, 2021. "Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning," Applied Energy, Elsevier, vol. 281(C).
    17. Mansour, Shaza H. & Azzam, Sarah M. & Hasanien, Hany M. & Tostado-Véliz, Marcos & Alkuhayli, Abdulaziz & Jurado, Francisco, 2025. "Deep reinforcement learning-based plug-in electric vehicle charging/discharging scheduling in a home energy management system," Energy, Elsevier, vol. 316(C).
    18. Dai, Yeming & Yu, Weijie & Leng, Mingming, 2024. "A hybrid ensemble optimized BiGRU method for short-term photovoltaic generation forecasting," Energy, Elsevier, vol. 299(C).
    19. Zhao, Chunyang & Andersen, Peter Bach & Træholt, Chresten & Hashemi, Seyedmostafa, 2023. "Grid-connected battery energy storage system: a review on application and integration," Renewable and Sustainable Energy Reviews, Elsevier, vol. 182(C).
    20. Crespo, Alicia & Fernández, Cèsar & Vérez, David & Tarragona, Joan & Borri, Emiliano & Frazzica, Andrea & Cabeza, Luisa F. & de Gracia, Alvaro, 2023. "Thermal performance assessment and control optimization of a solar-driven seasonal sorption storage system for residential application," Energy, Elsevier, vol. 263(PA).
    21. Chen, Minghao & Xie, Zhiyuan & Sun, Yi & Zheng, Shunlin, 2023. "The predictive management in campus heating system based on deep reinforcement learning and probabilistic heat demands forecasting," Applied Energy, Elsevier, vol. 350(C).
    22. Kim, Jimin & Obregon, Josue & Park, Hoonseok & Jung, Jae-Yoon, 2024. "Multi-step photovoltaic power forecasting using transformer and recurrent neural networks," Renewable and Sustainable Energy Reviews, Elsevier, vol. 200(C).
    23. Schinke-Nendza, A. & von Loeper, F. & Osinski, P. & Schaumann, P. & Schmidt, V. & Weber, C., 2021. "Probabilistic forecasting of photovoltaic power supply — A hybrid approach using D-vine copulas to model spatial dependencies," Applied Energy, Elsevier, vol. 304(C).
    24. Meng, Anbo & Wang, Peng & Zhai, Guangsong & Zeng, Cong & Chen, Shun & Yang, Xiaoyi & Yin, Hao, 2022. "Electricity price forecasting with high penetration of renewable energy using attention-based LSTM network trained by crisscross optimization," Energy, Elsevier, vol. 254(PA).
    25. Thaker, Jayesh & Höller, Robert, 2024. "Hybrid model for intra-day probabilistic PV power forecast," Renewable Energy, Elsevier, vol. 232(C).
    26. Ren, Kezheng & Liu, Jun & Liu, Xinglei & Nie, Yongxin, 2023. "Reinforcement Learning-Based Bi-Level strategic bidding model of Gas-fired unit in integrated electricity and natural gas markets preventing market manipulation," Applied Energy, Elsevier, vol. 336(C).
    27. Mehrjoo, Mehrdad & Jafari Jozani, Mohammad & Pawlak, Miroslaw, 2020. "Wind turbine power curve modeling for reliable power prediction using monotonic regression," Renewable Energy, Elsevier, vol. 147(P1), pages 214-222.
    28. Jia, Chunchun & Li, Kunang & He, Hongwen & Zhou, Jiaming & Li, Jianwei & Wei, Zhongbao, 2023. "Health-aware energy management strategy for fuel cell hybrid bus considering air-conditioning control based on TD3 algorithm," Energy, Elsevier, vol. 283(C).
    29. Abbasi, Kashif Raza & Shahbaz, Muhammad & Zhang, Jinjun & Irfan, Muhammad & Alvarado, Rafael, 2022. "Analyze the environmental sustainability factors of China: The role of fossil fuel energy and renewable energy," Renewable Energy, Elsevier, vol. 187(C), pages 390-402.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    2. Mahmud, Sakib & Sayed, Aya Nabil & Himeur, Yassine & Nhlabatsi, Armstrong & Bensaali, Faycal, 2026. "A comprehensive review of deep reinforcement learning applications from centralized power generation to modern energy internet frameworks," Renewable and Sustainable Energy Reviews, Elsevier, vol. 226(PE).
    3. Charalampos Rafail Lazaridis & Iakovos Michailidis & Georgios Karatzinis & Panagiotis Michailidis & Elias Kosmatopoulos, 2024. "Evaluating Reinforcement Learning Algorithms in Residential Energy Saving and Comfort Management," Energies, MDPI, vol. 17(3), pages 1-33, January.
    4. Xia, Chenyue & Xu, Yinliang & Tai, Nengling & Sun, Hongbin, 2026. "A dual-layer feature-selection transformer network for transferable probabilistic forecasting of PV power," Applied Energy, Elsevier, vol. 406(C).
    5. Dimitrios Vamvakas & Panagiotis Michailidis & Christos Korkas & Elias Kosmatopoulos, 2023. "Review and Evaluation of Reinforcement Learning Frameworks on Smart Grid Applications," Energies, MDPI, vol. 16(14), pages 1-38, July.
    6. Qiu, Dawei & Dong, Zihang & Zhang, Xi & Wang, Yi & Strbac, Goran, 2022. "Safe reinforcement learning for real-time automatic control in a smart energy-hub," Applied Energy, Elsevier, vol. 309(C).
    7. Panagiotis Michailidis & Iakovos Michailidis & Dimitrios Vamvakas & Elias Kosmatopoulos, 2023. "Model-Free HVAC Control in Buildings: A Review," Energies, MDPI, vol. 16(20), pages 1-45, October.
    8. Zeyue Sun & Mohsen Eskandari & Chaoran Zheng & Ming Li, 2022. "Handling Computation Hardness and Time Complexity Issue of Battery Energy Storage Scheduling in Microgrids by Deep Reinforcement Learning," Energies, MDPI, vol. 16(1), pages 1-20, December.
    9. Ren, Kezheng & Liu, Jun & Wu, Zeyang & Liu, Xinglei & Nie, Yongxin & Xu, Haitao, 2024. "A data-driven DRL-based home energy management system optimization framework considering uncertain household parameters," Applied Energy, Elsevier, vol. 355(C).
    10. Zhao, Liyuan & Yang, Ting & Li, Wei & Zomaya, Albert Y., 2022. "Deep reinforcement learning-based joint load scheduling for household multi-energy system," Applied Energy, Elsevier, vol. 324(C).
    11. Ma, Tao & Zhang, Yijie & Gu, Wenbo & Xiao, Gang & Yang, Hongxing & Wang, Shuxiao, 2022. "Strategy comparison and techno-economic evaluation of a grid-connected photovoltaic-battery system," Renewable Energy, Elsevier, vol. 197(C), pages 1049-1060.
    12. Hasna Hissou & Said Benkirane & Azidine Guezzaz & Mourade Azrour & Abderrahim Beni-Hssane, 2023. "A Novel Machine Learning Approach for Solar Radiation Estimation," Sustainability, MDPI, vol. 15(13), pages 1-21, July.
    13. Zhong, Mingwei & Fan, Jingmin & Luo, Jianqiang & Xiao, Xuanyi & He, Guanglin & Cai, Rui, 2024. "InfoCAVB-MemoryFormer: Forecasting of wind and photovoltaic power through the interaction of data reconstruction and data augmentation," Applied Energy, Elsevier, vol. 371(C).
    14. Zhang, Yijie & Ma, Tao & Yang, Hongxing, 2022. "Grid-connected photovoltaic battery systems: A comprehensive review and perspectives," Applied Energy, Elsevier, vol. 328(C).
    15. Song, Yuguang & Xia, Mingchao & Chen, Qifang & Chen, Fangjian, 2023. "A data-model fusion dispatch strategy for the building energy flexibility based on the digital twin," Applied Energy, Elsevier, vol. 332(C).
    16. Zhou, Xinlei & Du, Han & Xue, Shan & Ma, Zhenjun, 2024. "Recent advances in data mining and machine learning for enhanced building energy management," Energy, Elsevier, vol. 307(C).
    17. Hajialigol, Parisa & Nweye, Kingsley & Aghaei, Mohammadreza & Najafi, Behzad & Moazami, Amin & Nagy, Zoltan, 2025. "Enhancing self-consumption ratio in a smart microgrid by applying a reinforcement learning-based energy management system," Energy, Elsevier, vol. 335(C).
    18. Gong, Jianqiang & Qu, Zhiguo & Zhu, Zhenle & Xu, Hongtao, 2025. "Parallel TimesNet-BiLSTM model for ultra-short-term photovoltaic power forecasting using STL decomposition and auto-tuning," Energy, Elsevier, vol. 320(C).
    19. Liu, Jing & Wu, Yupeng, 2026. "A novel one-diode model for enhanced performance modelling of photovoltaic systems," Renewable Energy, Elsevier, vol. 264(C).
    20. Coraci, Davide & Brandi, Silvio & Hong, Tianzhen & Capozzoli, Alfonso, 2023. "Online transfer learning strategy for enhancing the scalability and deployment of deep reinforcement learning control in smart buildings," Applied Energy, Elsevier, vol. 333(C).

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:407:y:2026:i:c:s0306261925019877. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.