IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v407y2026ics0306261925019877.html

Adaptive home energy management based on PI-DRA-PPO for integrated electricity, hydrogen, and heat storage systems

Author

Listed:
  • Liu, Meng
  • Bu, Siqi
  • Zhu, Ziqing
  • Zhou, Bin

Abstract

The Home Energy Management System (HEMS) is becoming increasingly vital for mitigating residential energy expenditures and maximizing the utilization of renewable energy sources through coordinated scheduling of various energy storage devices. Integrating hydrogen storage into HEMS has also garnered growing attention for its ability to enhance household energy management flexibility and reliability and support the long-term goal of net-zero energy homes (NZEHs). However, challenges such as uncertain electricity prices and intermittent photovoltaic (PV) outputs frequently result in inefficient management strategies, leading to PV curtailment, reduced energy efficiency, and unreliable energy supply. To this end, this paper proposes a novel data-informed Deep Reinforcement Learning (DRL) framework, termed Prediction Intervals Constrained Dynamic Reward Adaptive Proximity Policy Optimization (PI-DRA-PPO), to optimize household energy management by enhancing both energy supply reliability and economic efficiency. A probabilistic prediction module combining Bi-directional Gated Recursive Unit with Coded Engineering (BiGRU-CE) and the Kernel Density Estimation (KDE) is employed to construct reliable prediction intervals (PIs), which serve as safety constraints embedded within the state space of the PI-DRA-PPO algorithm. Additionally, the Hydrogen Energy Storage System (HESS) is strategically integrated with the Thermal Energy Storage System (TESS) and Battery Energy Storage System (BESS) within the proposed HEMS for the coordinated management of multi-timescale energy dynamics induced by the intermittency of renewable energy sources. Compared with basic PPO and other DRL methods, the proposed method can achieve up to a 62.86 % reduction in total energy costs over a representative four-month period covering all seasons. The approach improves the effectiveness of energy scheduling and strengthens the temporal and long-term reliability of HEMS in the face of dynamic energy environments.

Suggested Citation

  • Liu, Meng & Bu, Siqi & Zhu, Ziqing & Zhou, Bin, 2026. "Adaptive home energy management based on PI-DRA-PPO for integrated electricity, hydrogen, and heat storage systems," Applied Energy, Elsevier, vol. 407(C).
    • Handle: RePEc:eee:appene:v:407:y:2026:i:c:s0306261925019877
    DOI: 10.1016/j.apenergy.2025.127257
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261925019877
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2025.127257?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Liu, Yanli & Wang, Junyi & Liu, Liqi, 2024. "Physics-informed reinforcement learning for probabilistic wind power forecasting under extreme events," Applied Energy, Elsevier, vol. 376(PA).
    2. Zhou, Yuekuan & Zheng, Siqian, 2020. "Machine-learning based hybrid demand-side controller for high-rise office buildings with high energy flexibilities," Applied Energy, Elsevier, vol. 262(C).
    3. Pang, Zhihong & Niu, Fuxin & O’Neill, Zheng, 2020. "Solar radiation prediction using recurrent neural network and artificial neural network: A case study with comparisons," Renewable Energy, Elsevier, vol. 156(C), pages 279-289.
    4. Pinto, Giuseppe & Piscitelli, Marco Savino & Vázquez-Canteli, José Ramón & Nagy, Zoltán & Capozzoli, Alfonso, 2021. "Coordinated energy management for a cluster of buildings through deep reinforcement learning," Energy, Elsevier, vol. 229(C).
    5. Hua, Haochen & Qin, Yuchao & Hao, Chuantong & Cao, Junwei, 2019. "Optimal energy management strategies for energy Internet via deep reinforcement learning approach," Applied Energy, Elsevier, vol. 239(C), pages 598-609.
    6. Dorokhova, Marina & Martinson, Yann & Ballif, Christophe & Wyrsch, Nicolas, 2021. "Deep reinforcement learning control of electric vehicle charging in the presence of photovoltaic generation," Applied Energy, Elsevier, vol. 301(C).
    7. Benali, L. & Notton, G. & Fouilloy, A. & Voyant, C. & Dizene, R., 2019. "Solar radiation forecasting using artificial neural network and random forest methods: Application to normal beam, horizontal diffuse and global components," Renewable Energy, Elsevier, vol. 132(C), pages 871-884.
    8. Zhou, Heng & Zheng, Peijun & Dong, Jiuqing & Liu, Jiang & Nakanishi, Yosuke, 2024. "Interpretable feature selection and deep learning for short-term probabilistic PV power forecasting in buildings using local monitoring data," Applied Energy, Elsevier, vol. 376(PA).
    9. Lork, Clement & Li, Wen-Tai & Qin, Yan & Zhou, Yuren & Yuen, Chau & Tushar, Wayes & Saha, Tapan K., 2020. "An uncertainty-aware deep reinforcement learning framework for residential air conditioning energy management," Applied Energy, Elsevier, vol. 276(C).
    10. Shi, Tao & Xu, Chang & Dong, Wenhao & Zhou, Hangyu & Bokhari, Awais & Klemeš, Jiří Jaromír & Han, Ning, 2023. "Research on energy management of hydrogen electric coupling system based on deep reinforcement learning," Energy, Elsevier, vol. 282(C).
    11. Maestre, V.M. & Ortiz, A. & Ortiz, I., 2024. "Sustainable and self-sufficient social home through a combined PV‑hydrogen pilot," Applied Energy, Elsevier, vol. 363(C).
    12. Bio Gassi, Karim & Baysal, Mustafa, 2023. "Improving real-time energy decision-making model with an actor-critic agent in modern microgrids with energy storage devices," Energy, Elsevier, vol. 263(PE).
    13. Dai, Yeming & Wang, Yanxin & Leng, Mingming & Yang, Xinyu & Zhou, Qiong, 2022. "LOWESS smoothing and Random Forest based GRU model: A short-term photovoltaic power generation forecasting method," Energy, Elsevier, vol. 256(C).
    14. Makolo, Peter & Zamora, Ramon & Lie, Tek-Tjing, 2021. "The role of inertia for grid flexibility under high penetration of variable renewables - A review of challenges and solutions," Renewable and Sustainable Energy Reviews, Elsevier, vol. 147(C).
    15. Xu, Biao & Luan, Wenpeng & Yang, Jing & Zhao, Bochao & Long, Chao & Ai, Qian & Xiang, Jiani, 2024. "Integrated three-stage decentralized scheduling for virtual power plants: A model-assisted multi-agent reinforcement learning method," Applied Energy, Elsevier, vol. 376(PA).
    16. Du, Yan & Zandi, Helia & Kotevska, Olivera & Kurte, Kuldeep & Munk, Jeffery & Amasyali, Kadir & Mckee, Evan & Li, Fangxing, 2021. "Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning," Applied Energy, Elsevier, vol. 281(C).
    17. Mansour, Shaza H. & Azzam, Sarah M. & Hasanien, Hany M. & Tostado-Véliz, Marcos & Alkuhayli, Abdulaziz & Jurado, Francisco, 2025. "Deep reinforcement learning-based plug-in electric vehicle charging/discharging scheduling in a home energy management system," Energy, Elsevier, vol. 316(C).
    18. Dai, Yeming & Yu, Weijie & Leng, Mingming, 2024. "A hybrid ensemble optimized BiGRU method for short-term photovoltaic generation forecasting," Energy, Elsevier, vol. 299(C).
    19. Zhao, Chunyang & Andersen, Peter Bach & Træholt, Chresten & Hashemi, Seyedmostafa, 2023. "Grid-connected battery energy storage system: a review on application and integration," Renewable and Sustainable Energy Reviews, Elsevier, vol. 182(C).
    20. Crespo, Alicia & Fernández, Cèsar & Vérez, David & Tarragona, Joan & Borri, Emiliano & Frazzica, Andrea & Cabeza, Luisa F. & de Gracia, Alvaro, 2023. "Thermal performance assessment and control optimization of a solar-driven seasonal sorption storage system for residential application," Energy, Elsevier, vol. 263(PA).
    21. Chen, Minghao & Xie, Zhiyuan & Sun, Yi & Zheng, Shunlin, 2023. "The predictive management in campus heating system based on deep reinforcement learning and probabilistic heat demands forecasting," Applied Energy, Elsevier, vol. 350(C).
    22. Kim, Jimin & Obregon, Josue & Park, Hoonseok & Jung, Jae-Yoon, 2024. "Multi-step photovoltaic power forecasting using transformer and recurrent neural networks," Renewable and Sustainable Energy Reviews, Elsevier, vol. 200(C).
    23. Schinke-Nendza, A. & von Loeper, F. & Osinski, P. & Schaumann, P. & Schmidt, V. & Weber, C., 2021. "Probabilistic forecasting of photovoltaic power supply — A hybrid approach using D-vine copulas to model spatial dependencies," Applied Energy, Elsevier, vol. 304(C).
    24. Meng, Anbo & Wang, Peng & Zhai, Guangsong & Zeng, Cong & Chen, Shun & Yang, Xiaoyi & Yin, Hao, 2022. "Electricity price forecasting with high penetration of renewable energy using attention-based LSTM network trained by crisscross optimization," Energy, Elsevier, vol. 254(PA).
    25. Thaker, Jayesh & Höller, Robert, 2024. "Hybrid model for intra-day probabilistic PV power forecast," Renewable Energy, Elsevier, vol. 232(C).
    26. Ren, Kezheng & Liu, Jun & Liu, Xinglei & Nie, Yongxin, 2023. "Reinforcement Learning-Based Bi-Level strategic bidding model of Gas-fired unit in integrated electricity and natural gas markets preventing market manipulation," Applied Energy, Elsevier, vol. 336(C).
    27. Mehrjoo, Mehrdad & Jafari Jozani, Mohammad & Pawlak, Miroslaw, 2020. "Wind turbine power curve modeling for reliable power prediction using monotonic regression," Renewable Energy, Elsevier, vol. 147(P1), pages 214-222.
    28. Jia, Chunchun & Li, Kunang & He, Hongwen & Zhou, Jiaming & Li, Jianwei & Wei, Zhongbao, 2023. "Health-aware energy management strategy for fuel cell hybrid bus considering air-conditioning control based on TD3 algorithm," Energy, Elsevier, vol. 283(C).
    29. Abbasi, Kashif Raza & Shahbaz, Muhammad & Zhang, Jinjun & Irfan, Muhammad & Alvarado, Rafael, 2022. "Analyze the environmental sustainability factors of China: The role of fossil fuel energy and renewable energy," Renewable Energy, Elsevier, vol. 187(C), pages 390-402.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    2. Charalampos Rafail Lazaridis & Iakovos Michailidis & Georgios Karatzinis & Panagiotis Michailidis & Elias Kosmatopoulos, 2024. "Evaluating Reinforcement Learning Algorithms in Residential Energy Saving and Comfort Management," Energies, MDPI, vol. 17(3), pages 1-33, January.
    3. Xia, Chenyue & Xu, Yinliang & Tai, Nengling & Sun, Hongbin, 2026. "A dual-layer feature-selection transformer network for transferable probabilistic forecasting of PV power," Applied Energy, Elsevier, vol. 406(C).
    4. Mahmud, Sakib & Sayed, Aya Nabil & Himeur, Yassine & Nhlabatsi, Armstrong & Bensaali, Faycal, 2026. "A comprehensive review of deep reinforcement learning applications from centralized power generation to modern energy internet frameworks," Renewable and Sustainable Energy Reviews, Elsevier, vol. 226(PE).
    5. Dimitrios Vamvakas & Panagiotis Michailidis & Christos Korkas & Elias Kosmatopoulos, 2023. "Review and Evaluation of Reinforcement Learning Frameworks on Smart Grid Applications," Energies, MDPI, vol. 16(14), pages 1-38, July.
    6. Qiu, Dawei & Dong, Zihang & Zhang, Xi & Wang, Yi & Strbac, Goran, 2022. "Safe reinforcement learning for real-time automatic control in a smart energy-hub," Applied Energy, Elsevier, vol. 309(C).
    7. Panagiotis Michailidis & Iakovos Michailidis & Dimitrios Vamvakas & Elias Kosmatopoulos, 2023. "Model-Free HVAC Control in Buildings: A Review," Energies, MDPI, vol. 16(20), pages 1-45, October.
    8. Razzano, Giuseppe & Brandi, Silvio & Piscitelli, Marco Savino & Capozzoli, Alfonso, 2025. "Rule extraction from deep reinforcement learning controller and comparative analysis with ASHRAE control sequences for the optimal management of Heating, Ventilation, and Air Conditioning (HVAC) systems in multizone buildings," Applied Energy, Elsevier, vol. 381(C).
    9. Zeyue Sun & Mohsen Eskandari & Chaoran Zheng & Ming Li, 2022. "Handling Computation Hardness and Time Complexity Issue of Battery Energy Storage Scheduling in Microgrids by Deep Reinforcement Learning," Energies, MDPI, vol. 16(1), pages 1-20, December.
    10. Tungom, Chia E. & Wang, Hong & Beata, Kamuya & Niu, Ben, 2024. "SWOAM: Swarm optimized agents for energy management in grid-interactive connected buildings," Energy, Elsevier, vol. 301(C).
    11. Al-Dahidi, Sameer & Alrbai, Mohammad & Rinchi, Bilal & Alahmer, Hussein & Al-Ghussain, Loiy & Hayajneh, Hassan S. & Alahmer, Ali, 2025. "Techno-economic implications and cost of forecasting errors in solar PV power production using optimized deep learning models," Energy, Elsevier, vol. 323(C).
    12. Xu, Xuesong & Xu, Kai & Zeng, Ziyang & Tang, Jiale & He, Yuanxing & Shi, Guangze & Zhang, Tao, 2024. "Collaborative optimization of multi-energy multi-microgrid system: A hierarchical trust-region multi-agent reinforcement learning approach," Applied Energy, Elsevier, vol. 375(C).
    13. Ren, Kezheng & Liu, Jun & Wu, Zeyang & Liu, Xinglei & Nie, Yongxin & Xu, Haitao, 2024. "A data-driven DRL-based home energy management system optimization framework considering uncertain household parameters," Applied Energy, Elsevier, vol. 355(C).
    14. Zhao, Liyuan & Yang, Ting & Li, Wei & Zomaya, Albert Y., 2022. "Deep reinforcement learning-based joint load scheduling for household multi-energy system," Applied Energy, Elsevier, vol. 324(C).
    15. Homod, Raad Z. & Togun, Hussein & Kadhim Hussein, Ahmed & Noraldeen Al-Mousawi, Fadhel & Yaseen, Zaher Mundher & Al-Kouz, Wael & Abd, Haider J. & Alawi, Omer A. & Goodarzi, Marjan & Hussein, Omar A., 2022. "Dynamics analysis of a novel hybrid deep clustering for unsupervised learning by reinforcement of multi-agent to energy saving in intelligent buildings," Applied Energy, Elsevier, vol. 313(C).
    16. Liu, Weican & Gai, Mei, 2025. "PV-MLP: A lightweight patch-based multi-layer perceptron network with time–frequency domain fusion for accurate long-sequence photovoltaic power forecasting," Renewable Energy, Elsevier, vol. 251(C).
    17. Ma, Tao & Zhang, Yijie & Gu, Wenbo & Xiao, Gang & Yang, Hongxing & Wang, Shuxiao, 2022. "Strategy comparison and techno-economic evaluation of a grid-connected photovoltaic-battery system," Renewable Energy, Elsevier, vol. 197(C), pages 1049-1060.
    18. Yu, Weijie & Dai, Yeming & Wang, Wenjie & Ren, Tao & Leng, Mingming, 2026. "Short-term photovoltaic forecasting: A parallel TimesNet and AT-Informer-AT method," Renewable Energy, Elsevier, vol. 258(C).
    19. Hasna Hissou & Said Benkirane & Azidine Guezzaz & Mourade Azrour & Abderrahim Beni-Hssane, 2023. "A Novel Machine Learning Approach for Solar Radiation Estimation," Sustainability, MDPI, vol. 15(13), pages 1-21, July.
    20. Zhong, Mingwei & Fan, Jingmin & Luo, Jianqiang & Xiao, Xuanyi & He, Guanglin & Cai, Rui, 2024. "InfoCAVB-MemoryFormer: Forecasting of wind and photovoltaic power through the interaction of data reconstruction and data augmentation," Applied Energy, Elsevier, vol. 371(C).

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:407:y:2026:i:c:s0306261925019877. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.