IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v415y2026ics0306261926005295.html

Flexible configuration and multi-objective optimization of multi-energy coupling units considering ammonia energy storage based on hierarchical multi-agent TD3

Author

Listed:
  • Zheng, J.H.
  • Su, Y.Y.
  • Liang, Youcai
  • Yao, Shunchun
  • Xue, Jian

Abstract

With the depletion of fossil fuels and increasing environmental concerns, hybrid energy systems (HES) require more advanced energy storage, and ammonia is gaining attention as a potential zero-carbon energy carrier. This paper proposes a multi-energy coupling unit (MECU) centered on ammonia storage, integrating ammonia conversion, thermal storage, and carbon capture, utilization, and storage (CCUS) to enable flexible regulation across electricity, heat, and gas networks. To optimize the capacity configuration and operation of MECU in the HES, a bi-layer optimization framework is developed, in which the upper-layer minimizes annualized investment cost and optimizes operational performance, and the lower-layer optimizes day-ahead dispatch to achieve economic, environmental, and security objectives. A hierarchical multi-agent twin delayed deep deterministic policy gradient (HMATD3) algorithm is proposed to tackle the bi-layer optimization. It employs a hierarchical architecture to decouple the decision layer and reduce computational complexity, while handling complex constraints via action constraint projection and Lagrangian relaxation. Prioritized experience replay and Huber loss are incorporated to achieve stable optimization and rapid convergence. Additionally, a two-stage training strategy and deployment framework are proposed to realize lifecycle optimization. Experiments conducted on the HES validate HMATD3’s superiority over baseline algorithms, achieving faster convergence and balanced multi-objective performance with negligible constraint violations. Compared to hydrogen-based MECU (MECU-H), ammonia-based MECU (MECU-A) reduces total costs by 26.4% through lower investment expenses and additional urea revenue, while simultaneously improving energy efficiency and lowering carbon emissions. Operational analysis verifies the effectiveness of price response strategies and the enhancement of grid security, highlighting ammonia’s potential in the low-carbon energy transition.

Suggested Citation

  • Zheng, J.H. & Su, Y.Y. & Liang, Youcai & Yao, Shunchun & Xue, Jian, 2026. "Flexible configuration and multi-objective optimization of multi-energy coupling units considering ammonia energy storage based on hierarchical multi-agent TD3," Applied Energy, Elsevier, vol. 415(C).
  • Handle: RePEc:eee:appene:v:415:y:2026:i:c:s0306261926005295
    DOI: 10.1016/j.apenergy.2026.127877
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261926005295
    Download Restriction: Full text may be restricted for ScienceDirect subscribers

    File URL: https://libkey.io/10.1016/j.apenergy.2026.127877?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Kiane Kleijne & Mark A. J. Huijbregts & Florian Knobloch & Rosalie Zelm & Jelle P. Hilbers & Heleen Coninck & Steef V. Hanssen, 2024. "Worldwide greenhouse gas emissions of green hydrogen production and transport," Nature Energy, Nature, vol. 9(9), pages 1139-1152, September.
    2. Gao, Mingfei & Han, Zhonghe & Zhang, Ce & Li, Peng & Wu, Di & Li, Peng, 2023. "Optimal configuration for regional integrated energy systems with multi-element hybrid energy storage," Energy, Elsevier, vol. 277(C).
    3. Zheng, J.H. & Su, Y.Y. & Guo, Y.W. & Li, Yuanzheng & Wu, Q.H., 2025. "Preference-based interactive multi-attribute decision-making support for stochastic scheduling of virtual power plants," Energy Economics, Elsevier, vol. 152(C).
    4. Wen, Du & Wei, Xinyi & Bruneau, Antonin & Maroonian, Aris & Maréchal, François & Van herle, Jan, 2025. "Techno-economic analysis of ammonia to hydrogen and power pathways considering the emerging hydrogen purification and fuel cell technologies," Applied Energy, Elsevier, vol. 390(C).
    5. Zhang, Ziyu & Ding, Tao & Zhou, Quan & Sun, Yuge & Qu, Ming & Zeng, Ziyu & Ju, Yuntao & Li, Li & Wang, Kang & Chi, Fangde, 2021. "A review of technologies and applications on versatile energy storage systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 148(C).
    6. Jiang, Kai & Ashworth, Peta & Zhang, Shiyi & Hu, Guoping, 2022. "Print media representations of carbon capture utilization and storage (CCUS) technology in China," Renewable and Sustainable Energy Reviews, Elsevier, vol. 155(C).
    7. Chen, Yuhong & Lyu, Yanfeng & Yang, Xiangdong & Zhang, Xiaohong & Pan, Hengyu & Wu, Jun & Lei, Yongjia & Zhang, Yanzong & Wang, Guiyin & Xu, Min & Luo, Hongbin, 2022. "Performance comparison of urea production using one set of integrated indicators considering energy use, economic cost and emissions’ impacts: A case from China," Energy, Elsevier, vol. 254(PC).
    8. Canizes, Bruno & Soares, João & Lezama, Fernando & Silva, Cátia & Vale, Zita & Corchado, Juan M., 2019. "Optimal expansion planning considering storage investment and seasonal effect of demand and renewable generation," Renewable Energy, Elsevier, vol. 138(C), pages 937-954.
    9. Sánchez, Antonio & Blanco, Elena C. & Martín, Mariano, 2024. "Comparative assessment of methanol and ammonia: Green fuels vs. hydrogen carriers in fuel cell power generation," Applied Energy, Elsevier, vol. 374(C).
    10. Akhtari, Mohammad Reza & Shayegh, Iman & Karimi, Nader, 2020. "Techno-economic assessment and optimization of a hybrid renewable earth - air heat exchanger coupled with electric boiler, hydrogen, wind and PV configurations," Renewable Energy, Elsevier, vol. 148(C), pages 839-851.
    11. Wang, Tianjing & Dong, Zhao Yang, 2024. "Adaptive personalized federated reinforcement learning for multiple-ESS optimal market dispatch strategy with electric vehicles and photovoltaic power generations," Applied Energy, Elsevier, vol. 365(C).
    12. Zheng, J.H. & Guo, J.C. & Deng, Weisi & Li, Zhigang & Wu, Q.H. & Zhou, X.X., 2024. "Flexible allocation and optimal configuration of multi-level energy exploitation units for heterogeneous energy systems considering resource distribution," Renewable Energy, Elsevier, vol. 230(C).
    13. Kiane Kleijne & Mark A. J. Huijbregts & Florian Knobloch & Rosalie Zelm & Jelle P. Hilbers & Heleen Coninck & Steef V. Hanssen, 2024. "Author Correction: Worldwide greenhouse gas emissions of green hydrogen production and transport," Nature Energy, Nature, vol. 9(11), pages 1449-1449, November.
    14. Liu, Jinhui & Xu, Zhanbo & Wu, Jiang & Liu, Kun & Guan, Xiaohong, 2021. "Optimal planning of distributed hydrogen-based multi-energy systems," Applied Energy, Elsevier, vol. 281(C).
    15. Mucci, Simone & Stumm, Marc-Daniel & Rix, Michael J. & Mitsos, Alexander, 2025. "Model-based evaluation of ammonia energy storage concepts at high technological readiness level," Applied Energy, Elsevier, vol. 377(PB).
    16. Zhang, Tong & Qadrdan, Meysam & Wu, Jianzhong & Couraud, Benoit & Stringer, Martin & Walker, Sara & Hawkes, Adam & Allahham, Adib & Flynn, David & Pudjianto, Danny & Dodds, Paul & Strbac, Goran, 2025. "A systematic review of modelling methods for studying the integration of hydrogen into energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 208(C).
    17. Sinha, Sunanda & Chandel, S.S., 2014. "Review of software tools for hybrid renewable energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 32(C), pages 192-205.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wu, Mou & Fang, Jiakun & Yan, Rujing & Zhang, Jing & Cui, Shichang & Zhang, Yihan & Wang, Shiqian & Ai, Xiaomeng & Wen, Jinyu, 2026. "Techno-economic analysis of hydrogen-to-power, ammonia-to-power pathway with biomass-to-power integration," Applied Energy, Elsevier, vol. 407(C).
    2. Pashakolaie, Vahid Ghorbani & Gudlaugsson, Bjarnhedinn & Ahmed, Tariq G., 2026. "From investment to impact: Exploring socio-economic prospect of hydrogen investment in Tees Valley, UK," Socio-Economic Planning Sciences, Elsevier, vol. 103(C).
    3. Vu, Hai & Chang, Daejun, 2025. "Life cycle global warming impact of a green hydrogen supply chain: A case study of Australia to South Korea," Applied Energy, Elsevier, vol. 397(C).
    4. Al-Ghussain, Loiy & Lu, Zifeng & Alrbai, Mohammad & Al-Dahidi, Sameer & He, Xin & Elgowainy, Amgad & Wang, Michael, 2025. "Global techno-economic and life cycle greenhouse gas emissions assessment of solar and wind based renewable hydrogen production," Applied Energy, Elsevier, vol. 401(PA).
    5. Wen, Du & Wei, Xinyi & Bruneau, Antonin & Maroonian, Aris & Maréchal, François & Van herle, Jan, 2025. "Techno-economic analysis of ammonia to hydrogen and power pathways considering the emerging hydrogen purification and fuel cell technologies," Applied Energy, Elsevier, vol. 390(C).
    6. Tom Terlouw & Christian Moretti & Carina Harpprecht & Romain Sacchi & Russell McKenna & Christian Bauer, 2025. "Global greenhouse gas emissions mitigation potential of existing and planned hydrogen projects," Nature Energy, Nature, vol. 10(12), pages 1503-1515, December.
    7. Nan Zhang & Yang Hu & Zhuang Zhang & Chao Wu & Jiamin Zhu & Yuqing Zhang & Wei Shen & Shanshan Wu & Shibo Xi & Li An & Pinxian Xi & Chun-Hua Yan, 2025. "Crystallinity-dependent structural evolution of CoS2 catalysts for enhanced oxygen evolution reaction," Nature Communications, Nature, vol. 16(1), pages 1-12, December.
    8. Chen, Xiaoyu & Zheng, Lin & Cai, Guotian & Qi, Xiaoling & He, Jiaxin, 2025. "Promoting innovation: Policy impacts on global hydrogen value chain," Renewable and Sustainable Energy Reviews, Elsevier, vol. 223(C).
    9. Driemeier, Carlos E. & Tonon, Giovana C. & Chagas, Mateus F. & Petrielli, Gabriel P. & Henzler, Daniele S. & Gomes, Luísa C.M. & Limeira, Bruno E. & Hernandes, Thayse A.D. & Morais, Edvaldo R., 2025. "Electrolytic hydrogen in a large-scale decarbonized grid with energy reservoirs: An assessment of carbon intensity and integrity," Applied Energy, Elsevier, vol. 391(C).
    10. Brand, C. & Marsden, G. & Anable, J.L. & Dixon, J. & Barrett, J., 2025. "Achieving deep transport energy demand reductions in the United Kingdom," Renewable and Sustainable Energy Reviews, Elsevier, vol. 207(C).
    11. Li, Yuhang & Yang, Ruixin & Long, Yan & Zhou, Xiafeng, 2025. "Investigation of the SO2 capture and iodine-sulfur cycle for H2-H2SO4-electricity-heat polygeneration system based on spectral splitting," Energy, Elsevier, vol. 326(C).
    12. Liu, Zongqi & Ren, Jingjie & Sun, Weicheng & Yu, Di & Wang, Yuan & Bi, Mingshu, 2026. "Analyzing gas crossover risk of proton exchange membrane water electrolyzer through a perspective of disease prevention and treatment," Applied Energy, Elsevier, vol. 403(PA).
    13. Jia, Wenhao & Ding, Tao & He, Yuhan, 2026. "Synergistic integration of green hydrogen in renewable power systems: A comprehensive review of key technologies, research landscape, and future perspectives," Renewable and Sustainable Energy Reviews, Elsevier, vol. 226(PD).
    14. Caiafa, Clara & Kratena, Kurt & Arto, Iñaki, 2025. "Macroeconomic and environmental impacts of two decarbonization options for the Dutch steel industry: green relocation versus green hydrogen imports," Energy Policy, Elsevier, vol. 206(C).
    15. Hong, Frank T. & Xue, Xingyu & Khaled, Fethi & AlRamadan, Abdullah S., 2025. "Decarbonizing China's passenger vehicle sector with renewable hydrogen: Lower-carbon gasoline outperforms alternative fuel applications in current transition," Energy, Elsevier, vol. 333(C).
    16. Mili Liu & Yunqi Jia & Jiangwen Liu & Kang Chen & Hao Zhong & Lin Jiang & Hui Liu & Liuzhang Ouyang & Min Zhu, 2025. "Activating silicon for high hydrogen conversion and sustainable anode recovery," Nature Communications, Nature, vol. 16(1), pages 1-10, December.
    17. Rodrigues, Vera & Monteiro, Alexandra & Díaz Reigosa, David & Ferreira, Joana & Wu, Xiao-Yu, 2026. "Ammonia as a fuel for shipping and aviation: Opportunities, environmental benefits and risks," Renewable and Sustainable Energy Reviews, Elsevier, vol. 239(C).
    18. Kaabinejadian, Amirreza & Pozarlik, Artur & Acar, Canan, 2025. "A systematic review of predictive, optimization, and smart control strategies for hydrogen-based building heating systems," Applied Energy, Elsevier, vol. 379(C).
    19. Michaël Lejeune & Sami Kara & Michael Zwicky Hauschild & Sareh Shahrabifarahani & Rahman Daiyan, 2026. "Pathways to global hydrogen production within planetary boundaries," Nature Communications, Nature, vol. 17(1), pages 1-16, December.
    20. Jian You & Jinyu Lu & Chuanli Liu & Wei Wang & Yongzhao Li & Yuanzhong Gao & Longmin Liu & Xiangbo Luo & Xiaojun Bao & Huaiyin Chen & Jianying Huang & Yuekun Lai & Meihua Wu & Weilong Cai, 2025. "A rationally thin composite membrane with differentiated pore structure for industrial-scale alkaline water electrolysis," Nature Communications, Nature, vol. 16(1), pages 1-12, December.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:415:y:2026:i:c:s0306261926005295. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.