Deep learning-driven heat load prediction: Investigating the impacts of optimizer and learning rate scheduler strategies

Heat load prediction in district heating systems (DHS) presents challenges such as large-scale systems, complex dynamic characteristics with significant time delays, and the inherent randomness and uncertainty in end-user demand. Traditional models often fail to capture the intricate patterns in historical data. In contrast, deep learning models can identify complex nonlinear relationships within extensive historical data, facilitating more accurate predictions. The optimizer fine-tunes model parameters to minimize the loss function, thereby supporting effective learning and generalization, while the scheduler dynamically adjusts hyperparameters (e.g., learning rate) during training to ensure both stability and fast convergence. This study develops seven heat load prediction models based on CNN, LSTM, and KAN architectures and investigates the performance of different combinations of three optimizers (SGD, Adam, AdamW) and three learning rate schedulers (ExponentialLR, StepLR, CosineAnnealingLR). The results indicate that Adam and AdamW outperform SGD, with SGD suffering from underfitting. Compared to SGD, the models under the Adam and AdamW frameworks exhibit significantly lower mean MAE, RMSE, and MAPE. Specifically, for Adam and AdamW, the mean MAE values are 0.9668 and 1.1746, the mean RMSE values are 1.4053 and 1.6154, and the mean MAPE values are 0.0047 and 0.0057, respectively. CosineAnnealingLR performs exceptionally well by smoothly adjusting the learning rate, enabling the model to effectively capture both periodic and non-stationary features, leading to stable and precise predictions. The CNN-KAN model stands out, demonstrating remarkable stability and robustness, with R2 values greater than 0.95 and 0.97 under the Adam and AdamW frameworks, respectively. Therefore, in the DHS heat load forecasting, the combination of Adam/AdamW and CosineAnnealingLR enables the model to maintain high-precision predictions, while the CNN-KAN architecture demonstrates stable and effective performance on complex heat load forecasting tasks. This study provides actionable guidance for the selection of optimizers and learning-rate schedulers in DHS heat-load forecasting.