Physics-guided deep reinforcement learning for optimized data center cooling and waste heat recovery utilizing aquifer thermal energy storage

A critical challenge in sustainable data center operations lies in resolving the mismatch between escalating cooling demands and waste heat utilization potential. Conventional approaches address cooling and heat recovery as separate processes, incurring systemic inefficiencies. This study develops a physics-guided deep reinforcement learning (DRL) framework that synergistically optimizes Aquifer Thermal Energy Storage (ATES) for data center cooling and waste heat recovery for nearby office buildings via heat pumps. By incorporating domain knowledge into the reward function design, the proposed approach effectively addresses delayed rewards in long-term ATES thermal balance and enables effective agent training with limited datasets. Multiple advanced DRL agents, such as DQN and D3QN, are trained to control the operation of the integrated energy systems, with dual objectives of minimizing energy consumption and maintaining annual ATES thermal balance. Results demonstrate that the both D3QN and Double DQN algorithms perform well, reducing annual energy consumption by approximately 53 % while also maintaining ATES balance within 4 %. Furthermore, the system achieves a remarkable power usage effectiveness (PUE) of 1.177, representing a 9.5 % improvement over conventional systems. Additionally, system validation via Dymola simulations demonstrates that groundwater temperatures return to initial conditions (±0.5 °C) after annual cycling. The developed framework establishes a generalizable methodology for AI-driven optimization of sustainable data center cooling systems integrated with waste heat recovery.