A knowledge-embedded reinforcement learning framework for energy management of planned integrated energy systems

The planning and operation of integrated energy systems (IESs) face challenges from multi-energy coupling and demand variability. Conventional optimisation methods have limitations in handling scheduling under uncertain conditions, whereas reinforcement learning (RL) often suffers from slow convergence and unstable exploration. To address these challenges, this paper proposes a planning-informed knowledge-embedded reinforcement learning (KRL) framework that integrates long-term planning with operational energy scheduling for IESs. First, the planning stage optimises the capacities of key components, thereby defining the action space and feasible operating regime of the subsequent stage. In the operational stage, a dual-learning strategy combining supervised learning and RL is developed to implement KRL. Prior knowledge is extracted from mixed-integer linear programming-based optimal scheduling solutions under diverse scenarios and then embedded into RL training. By balancing knowledge guidance with self-exploration, the proposed method improves RL training efficiency and stability. The proposed framework is evaluated on the IES of a multi-building community in the UK. Results show that the planning stage achieves up to 29.5% long-term cost savings, with a simple payback period of approximately 7 years. In the operational stage, KRL converges faster and achieves higher rewards than the baseline methods across scenarios. Compared with a conventional community energy supply system mainly relying on grid electricity and gas boilers, the proposed method achieves over 36.4% expenditure savings and 10.5% carbon emission reductions. Under renewable-energy-abundant conditions, the cost-saving performance further increases to 86.7%.