MMGPT4LF: Leveraging an optimized pre-trained GPT-2 model with multi-modal cross-attention for load forecasting

Accurate load forecasting is crucial for maintaining power system balance. Traditionally, forecasting relies on time series data such as historical loads and corresponding meteorological information. However, non-time-series data like news reports and holiday schedules can also significantly influence outcomes. Existing research primarily focuses on time series data and lacks effective handling of multi-modal inputs. Recent advances in Large Language Models (LLMs) demonstrate inherent advantages in capturing long-term dependencies and complex textual patterns, indicating their potential for load forecasting. Nevertheless, the application of LLMs in this field remains limited. Thus, to fill this gap, we propose MMGPT4LF, a model that combines the pre-trained GPT-2 model with multi-modal data inputs for load forecasting. Specifically, the model designs an additional time series input head to more effectively capture temporal dependencies, particularly the periodicity and long-term trends present in power load data. Furthermore, the model incorporates a Multi-Modal Cross-Attention (MMCA) mechanism, enabling efficient alignment and fusion of high-dimensional feature representations from both time series and textual inputs. Through this framework, MMGPT4LF not only enhances the effectiveness of multi-modal data fusion but also accurately handles the interactions between different modalities, thereby significantly improving load forecasting accuracy and the model’s generalization ability. Extensive experiments on two open-source load forecasting datasets, compared with nine advanced time series forecasting models, validate the effectiveness and accuracy of MMGPT4LF in load forecasting tasks.