Optimal scheduling of electric-gas-thermal-hydrogen integrated energy system considering uncertainties and safe guarantee: A TD3-MIP-based approach

The multiple uncertainties arising from sources and loads present substantial challenges to maintaining the supply-demand balance in the integrated energy system (IES). Data-driven deep reinforcement learning (DRL) algorithms have gained attention for handling complex scheduling problems due to their model-free capabilities. However, ensuring safety remains a significant issue for DRL algorithms, as the agent may fail to satisfy all operation constraints simultaneously. To address these challenges, this paper proposes an IES scheduling approach that combines twin delayed deep deterministic policy gradient (TD3) and mixed-integer programming (MIP), termed as TD3-MIP, which considers uncertainties and provides a safety guarantee. Firstly, to improve energy efficiency and promote low-carbon energy transitions, hydrogen energy is introduced into the IES. A framework for joint electric-gas-thermal-hydrogen scheduling is designed, and the scheduling problem is expressed as a reinforcement learning model. Secondly, the TD3 algorithm is employed to train the scheduling model, dealing with the uncertainties in sources and loads. The operation costs, unbalance penalties and out-of-limit penalties are integrated into the reward function to guide the agent in learning safe and cost-effective decisions during the offline training process. Furthermore, a novel safety IES scheduling strategy is proposed to enforce all operation constraints and achieve supply-demand balance rigorously during the online scheduling process. In this strategy, the trained TD3 critic network structure is redefined as an MIP formulation by introducing a binary variable. In addition, all of the operation constraints are added to the MIP model, which is then solved using a mathematical solver. Finally, simulation results under various scenarios demonstrate that the proposed TD3-MIP approach effectively deals with uncertainties in renewable energy generation and demands under strict constraints. It also shows a competitive edge in cost-efficiency compared to other scheduling approaches.