IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v384y2025ics0306261925000881.html
   My bibliography  Save this article

Toward Large Energy Models: A comparative study of Transformers’ efficacy for energy forecasting

Author

Listed:
  • Gu, Yueyan
  • Jazizadeh, Farrokh
  • Wang, Xuan

Abstract

Buildings’ significant contribution to global energy demand and emissions highlights the need for precise energy forecasting for effective management. Existing research on energy forecasting commonly focuses on specific target problems, such as individual buildings or small groups of buildings, leading to current challenges in data-driven forecasting, including dependence on data quality and quantity, limited generalizability, and computational inefficiency. To address these challenges, Generalized Energy Models (GEMs) for energy forecasting can potentially be developed using large-scale datasets. Transformers, known for their scalability, ability to capture long-term dependencies and efficiency in parallel processing of large datasets, are considered good candidates for GEMs. In this study, we tested the hypothesis that GEMs can be efficiently developed to outperform in-situ (i.e., building-specific) models trained solely on data from individual buildings. To this end, we investigated and compared three candidate multivariate Transformer architectures, utilizing both zero-shot and fine-tuning strategies, with data from 1,014 buildings. The results, evaluated across three prediction horizons (24, 72, and 168 h), confirm that GEMs significantly outperform Transformer-based in-situ models. Fine-tuned GEMs showed performance improvements of up to 28% in MSE and reduced training time by 55%. Besides Transformer-based in-situ models, GEMs outperformed several state-of-the-art non-Transformer deep learning baseline models in both effectiveness and efficiency. We further explored a number of questions, including the required data size for effective fine-tuning, as well as the impact of input sub-sequence length and pre-training dataset size on GEMs’ performance. The findings show a statistically significant performance boost from using larger pre-training datasets, highlighting the potential for larger GEMs using web-scale global data to move toward Large Energy Models (LEM).

Suggested Citation

  • Gu, Yueyan & Jazizadeh, Farrokh & Wang, Xuan, 2025. "Toward Large Energy Models: A comparative study of Transformers’ efficacy for energy forecasting," Applied Energy, Elsevier, vol. 384(C).
    • Handle: RePEc:eee:appene:v:384:y:2025:i:c:s0306261925000881
    DOI: 10.1016/j.apenergy.2025.125358
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261925000881
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2025.125358?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Fuchao Yu & Xianchao Xiu & Yunhui Li, 2022. "A Survey on Deep Transfer Learning and Beyond," Mathematics, MDPI, vol. 10(19), pages 1-27, October.
    2. Ahmad, Tanveer & Chen, Huanxin & Huang, Ronggeng & Yabin, Guo & Wang, Jiangyu & Shair, Jan & Azeem Akram, Hafiz Muhammad & Hassnain Mohsan, Syed Agha & Kazim, Muhammad, 2018. "Supervised based machine learning models for short, medium and long-term energy prediction in distinct building environment," Energy, Elsevier, vol. 158(C), pages 17-32.
    3. Jain, Rishee K. & Smith, Kevin M. & Culligan, Patricia J. & Taylor, John E., 2014. "Forecasting energy consumption of multi-family residential buildings using support vector regression: Investigating the impact of temporal and spatial monitoring granularity on performance accuracy," Applied Energy, Elsevier, vol. 123(C), pages 168-178.
    4. Maleki, Neda & Lundström, Oxana & Musaddiq, Arslan & Jeansson, John & Olsson, Tobias & Ahlgren, Fredrik, 2024. "Future energy insights: Time-series and deep learning models for city load forecasting," Applied Energy, Elsevier, vol. 374(C).
    5. Manfren, Massimiliano & James, Patrick AB. & Tronchin, Lamberto, 2022. "Data-driven building energy modelling – An analysis of the potential for generalisation through interpretable machine learning," Renewable and Sustainable Energy Reviews, Elsevier, vol. 167(C).
    6. Zheng, Peijun & Zhou, Heng & Liu, Jiang & Nakanishi, Yosuke, 2023. "Interpretable building energy consumption forecasting using spectral clustering algorithm and temporal fusion transformers architecture," Applied Energy, Elsevier, vol. 349(C).
    7. Deb, Chirag & Zhang, Fan & Yang, Junjing & Lee, Siew Eang & Shah, Kwok Wei, 2017. "A review on time series forecasting techniques for building energy consumption," Renewable and Sustainable Energy Reviews, Elsevier, vol. 74(C), pages 902-924.
    8. Song, Cairong & Yang, Haidong & Cai, Jianyang & Yang, Pan & Bao, Hao & Xu, Kangkang & Meng, Xian-Bing, 2024. "Multi-energy load forecasting via hierarchical multi-task learning and spatiotemporal attention," Applied Energy, Elsevier, vol. 373(C).
    9. Peng, Jieyang & Kimmig, Andreas & Wang, Dongkun & Niu, Zhibin & Liu, Xiufeng & Tao, Xiaoming & Ovtcharova, Jivka, 2024. "Energy consumption forecasting based on spatio-temporal behavioral analysis for demand-side management," Applied Energy, Elsevier, vol. 374(C).
    10. Eren, Yavuz & Küçükdemiral, İbrahim, 2024. "A comprehensive review on deep learning approaches for short-term load forecasting," Renewable and Sustainable Energy Reviews, Elsevier, vol. 189(PB).
    11. Alexandra L’Heureux & Katarina Grolinger & Miriam A. M. Capretz, 2022. "Transformer-Based Model for Electrical Load Forecasting," Energies, MDPI, vol. 15(14), pages 1-23, July.
    12. Junhui Huang & Sakdirat Kaewunruen, 2023. "Forecasting Energy Consumption of a Public Building Using Transformer and Support Vector Regression," Energies, MDPI, vol. 16(2), pages 1-15, January.
    13. Xiao, Tong & Xu, Peng & He, Ruikai & Sha, Huajing, 2022. "Status quo and opportunities for building energy prediction in limited data Context—Overview from a competition," Applied Energy, Elsevier, vol. 305(C).
    14. Canaydin, Ada & Fu, Chun & Balint, Attila & Khalil, Mohamad & Miller, Clayton & Kazmi, Hussain, 2024. "Interpretable domain-informed and domain-agnostic features for supervised and unsupervised learning on building energy demand data," Applied Energy, Elsevier, vol. 360(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wang, Chao-fan & Liu, Kui-xing & Peng, Jieyang & Li, Xiang & Liu, Xiu-feng & Zhang, Jia-wan & Niu, Zhi-bin, 2025. "High-precision energy consumption forecasting for large office building using a signal decomposition-based deep learning approach," Energy, Elsevier, vol. 314(C).
    2. Ahmad, Tanveer & Huanxin, Chen & Zhang, Dongdong & Zhang, Hongcai, 2020. "Smart energy forecasting strategy with four machine learning models for climate-sensitive and non-climate sensitive conditions," Energy, Elsevier, vol. 198(C).
    3. R. Rueda & M. P. Cuéllar & M. Molina-Solana & Y. Guo & M. C. Pegalajar, 2019. "Generalised Regression Hypothesis Induction for Energy Consumption Forecasting," Energies, MDPI, vol. 12(6), pages 1-22, March.
    4. Sunil Kumar Mohapatra & Sushruta Mishra & Hrudaya Kumar Tripathy & Akash Kumar Bhoi & Paolo Barsocchi, 2021. "A Pragmatic Investigation of Energy Consumption and Utilization Models in the Urban Sector Using Predictive Intelligence Approaches," Energies, MDPI, vol. 14(13), pages 1-28, June.
    5. Venkatraj, V. & Dixit, M.K., 2022. "Challenges in implementing data-driven approaches for building life cycle energy assessment: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 160(C).
    6. Kamran Hassanpouri Baesmat & Zeinab Farrokhi & Grzegorz Chmaj & Emma E. Regentova, 2025. "Parallel Multi-Model Energy Demand Forecasting with Cloud Redundancy: Leveraging Trend Correction, Feature Selection, and Machine Learning," Forecasting, MDPI, vol. 7(2), pages 1-18, May.
    7. Wang, Qiang & Li, Shuyu & Li, Rongrong, 2018. "Forecasting energy demand in China and India: Using single-linear, hybrid-linear, and non-linear time series forecast techniques," Energy, Elsevier, vol. 161(C), pages 821-831.
    8. Zhang, Liang & Wen, Jin & Li, Yanfei & Chen, Jianli & Ye, Yunyang & Fu, Yangyang & Livingood, William, 2021. "A review of machine learning in building load prediction," Applied Energy, Elsevier, vol. 285(C).
    9. Peplinski, McKenna & Dilkina, Bistra & Chen, Mo & Silva, Sam J. & Ban-Weiss, George A. & Sanders, Kelly T., 2024. "A machine learning framework to estimate residential electricity demand based on smart meter electricity, climate, building characteristics, and socioeconomic datasets," Applied Energy, Elsevier, vol. 357(C).
    10. Ahmad, Tanveer & Chen, Huanxin, 2019. "Deep learning for multi-scale smart energy forecasting," Energy, Elsevier, vol. 175(C), pages 98-112.
    11. Li, Guannan & Wu, Yubei & Yoon, Sungmin & Fang, Xi, 2024. "Comprehensive transferability assessment of short-term cross-building-energy prediction using deep adversarial network transfer learning," Energy, Elsevier, vol. 299(C).
    12. Yu, Fu Wing & Ho, Wai Tung & Wong, Chak Fung Jeff, 2025. "Integrating time series decomposition and multivariable approaches for enhanced cooling energy management," Energy, Elsevier, vol. 318(C).
    13. Zhong, Hai & Wang, Jiajun & Jia, Hongjie & Mu, Yunfei & Lv, Shilei, 2019. "Vector field-based support vector regression for building energy consumption prediction," Applied Energy, Elsevier, vol. 242(C), pages 403-414.
    14. Kamel, Ehsan & Sheikh, Shaya & Huang, Xueqing, 2020. "Data-driven predictive models for residential building energy use based on the segregation of heating and cooling days," Energy, Elsevier, vol. 206(C).
    15. Federico Divina & Miguel García Torres & Francisco A. Goméz Vela & José Luis Vázquez Noguera, 2019. "A Comparative Study of Time Series Forecasting Methods for Short Term Electric Energy Consumption Prediction in Smart Buildings," Energies, MDPI, vol. 12(10), pages 1-23, May.
    16. Somu, Nivethitha & Raman M R, Gauthama & Ramamritham, Krithi, 2021. "A deep learning framework for building energy consumption forecast," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    17. Xiao, Liye & Shao, Wei & Liang, Tulu & Wang, Chen, 2016. "A combined model based on multiple seasonal patterns and modified firefly algorithm for electrical load forecasting," Applied Energy, Elsevier, vol. 167(C), pages 135-153.
    18. Alexandru Pîrjan & Simona-Vasilica Oprea & George Căruțașu & Dana-Mihaela Petroșanu & Adela Bâra & Cristina Coculescu, 2017. "Devising Hourly Forecasting Solutions Regarding Electricity Consumption in the Case of Commercial Center Type Consumers," Energies, MDPI, vol. 10(11), pages 1-36, October.
    19. Xavier Serrano-Guerrero & Guillermo Escrivá-Escrivá & Santiago Luna-Romero & Jean-Michel Clairand, 2020. "A Time-Series Treatment Method to Obtain Electrical Consumption Patterns for Anomalies Detection Improvement in Electrical Consumption Profiles," Energies, MDPI, vol. 13(5), pages 1-23, February.
    20. Rao, Congjun & Zhang, Yue & Wen, Jianghui & Xiao, Xinping & Goh, Mark, 2023. "Energy demand forecasting in China: A support vector regression-compositional data second exponential smoothing model," Energy, Elsevier, vol. 263(PC).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:384:y:2025:i:c:s0306261925000881. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.