Flexible configuration and multi-objective optimization of multi-energy coupling units considering ammonia energy storage based on hierarchical multi-agent TD3

With the depletion of fossil fuels and increasing environmental concerns, hybrid energy systems (HES) require more advanced energy storage, and ammonia is gaining attention as a potential zero-carbon energy carrier. This paper proposes a multi-energy coupling unit (MECU) centered on ammonia storage, integrating ammonia conversion, thermal storage, and carbon capture, utilization, and storage (CCUS) to enable flexible regulation across electricity, heat, and gas networks. To optimize the capacity configuration and operation of MECU in the HES, a bi-layer optimization framework is developed, in which the upper-layer minimizes annualized investment cost and optimizes operational performance, and the lower-layer optimizes day-ahead dispatch to achieve economic, environmental, and security objectives. A hierarchical multi-agent twin delayed deep deterministic policy gradient (HMATD3) algorithm is proposed to tackle the bi-layer optimization. It employs a hierarchical architecture to decouple the decision layer and reduce computational complexity, while handling complex constraints via action constraint projection and Lagrangian relaxation. Prioritized experience replay and Huber loss are incorporated to achieve stable optimization and rapid convergence. Additionally, a two-stage training strategy and deployment framework are proposed to realize lifecycle optimization. Experiments conducted on the HES validate HMATD3’s superiority over baseline algorithms, achieving faster convergence and balanced multi-objective performance with negligible constraint violations. Compared to hydrogen-based MECU (MECU-H), ammonia-based MECU (MECU-A) reduces total costs by 26.4% through lower investment expenses and additional urea revenue, while simultaneously improving energy efficiency and lowering carbon emissions. Operational analysis verifies the effectiveness of price response strategies and the enhancement of grid security, highlighting ammonia’s potential in the low-carbon energy transition.