Knowledge-guided reinforcement learning for HVAC controls and energy saving through an efficient simulation framework

Improving the energy efficiency of HVAC systems is crucial for both economic and environmental benefits. Research has demonstrated the superior performance of reinforcement learning (RL) in HVAC controls. However, creating environments for RL training is challenging and time-consuming, and there is still considerable room for improving the performance of traditional RL methods, encompassing training efficiency and control effectiveness. This work presented an efficient simulation framework based on EnergyPlus Python API to enhance the efficiency of creating RL simulation environments and accelerate RL training without the dependence on co-simulation middleware. Furthermore, this work proposes a knowledge-guided RL method for HVAC controls. Specifically, expert policy imitation is used to enhance RL, improving control behaviors in the early stage of RL training and accelerating convergence. Afterwards, the RL policy is further refined with expert knowledge, enabling it to handle both discrete and continuous control variables effectively, thereby enhancing the control performance. The simulation framework significantly reduced the RL training time to 27 s in 14688 simulation steps and achieved a 13.3 % reduction in total energy consumption while maintaining the indoor temperature satisfaction. The simulation framework and training method demonstrated great transferability and robustness across similar climates.