A multimodal deep learning approach for very short-term solar forecasts using sky images and historical numerical data

The increased solar energy integration into the power system introduces various issues due to its intermittent nature. As solar power penetration increases, grid management becomes increasingly complex, highlighting the importance of precise solar power forecasts. This paper addresses a critical challenge in solar power integration, highlighting the significance of very short-term solar forecasting (VSTSF) for grid operators. We propose a novel MULSKIN hybrid framework that combines a custom ResNet and a novel targeted feature attention mechanism (TFAM), which dynamically focuses on the most relevant regions in sky images, improving the model's ability to capture essential cloud dynamics and spatiotemporal patterns. Unlike existing methods in literature that either use image (sky images) or numerical (meteorological) data, the proposed MULSKIN approach integrates both data types, allowing for a more holistic understanding of atmospheric conditions. This paper contributes to advancing the field of solar forecasting by presenting a new, effective, and scalable solution suitable for operational deployment in modern energy systems. With six years of high-resolution data, the model outperforms traditional machine learning (ML), deep learning (DL) models, and baseline reference models, achieving a forecast skill score of 32.84 % and a root mean square error (RMSE) of 54.36W/m2. Our Results indicate that incorporating sky images alongside numerical meteorological data significantly enhances forecasting accuracy, with the MULSKIN model demonstrating its superiority in capturing dynamic cloud movements and weather patterns over short time horizons. The findings underscore the potential of combining multimodal data sources to address the challenges of VSTSF, offering promising improvements in solar irradiance prediction for real-time grid operations.