IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v325y2022ics0306261922011667.html
   My bibliography  Save this article

Accuracy indicators for evaluating retrospective performance of energy system models

Author

Listed:
  • Wen, Xin
  • Jaxa-Rozen, Marc
  • Trutnevyte, Evelina

Abstract

Retrospective evaluation of energy system models and scenarios is essential for ensuring their robustness for prospective policy support. However, quantitative evaluations currently lack systematic methods to be more holistic and informative. This paper reviews existing accuracy indicators used for retrospective evaluations of energy models and scenarios with the aim to find a small suite of complementary indicators. We quantify and compare 24 indicators to assess the retrospective performance of D-EXPANSE electricity sector modeling framework, used to model 31 European countries in parallel from 1990–2019. We find that symmetric mean percentage error, symmetric mean absolute percentage error, symmetric median absolute percentage error, root-mean-squared logarithmic error, and growth error together form the most informative suite of indicators. This study is the first step towards developing a model accuracy testbench to assess energy models and scenarios in multiple dimensions retrospectively.

Suggested Citation

  • Wen, Xin & Jaxa-Rozen, Marc & Trutnevyte, Evelina, 2022. "Accuracy indicators for evaluating retrospective performance of energy system models," Applied Energy, Elsevier, vol. 325(C).
    • Handle: RePEc:eee:appene:v:325:y:2022:i:c:s0306261922011667
    DOI: 10.1016/j.apenergy.2022.119906
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261922011667
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2022.119906?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Hillard G. Huntington, 1994. "Oil Price Forecasting in the 1980s: What Went Wrong?," The Energy Journal, International Association for Energy Economics, vol. 0(Number 2), pages 1-22.
    2. Labandeira, Xavier & Labeaga, José M. & López-Otero, Xiral, 2017. "A meta-analysis on the price elasticity of energy demand," Energy Policy, Elsevier, vol. 102(C), pages 549-568.
    3. McConnell, Dylan & Hearps, Patrick & Eales, Dominic & Sandiford, Mike & Dunn, Rebecca & Wright, Matthew & Bateman, Lachlan, 2013. "Retrospective modeling of the merit-order effect on wholesale electricity prices from distributed photovoltaic generation in the Australian National Electricity Market," Energy Policy, Elsevier, vol. 58(C), pages 17-27.
    4. Fujimori, Shinichiro & Dai, Hancheng & Masui, Toshihiko & Matsuoka, Yuzuru, 2016. "Global energy model hindcasting," Energy, Elsevier, vol. 114(C), pages 293-301.
    5. Huntington, Hillard G., 2021. "Model evaluation for policy insights: Reflections on the forum process," Energy Policy, Elsevier, vol. 156(C).
    6. He, Yongxiu & Jiao, Jie & Chen, Qian & Ge, Sifan & Chang, Yan & Xu, Yang, 2017. "Urban long term electricity demand forecast method based on system dynamics of the new economic normal: The case of Tianjin," Energy, Elsevier, vol. 133(C), pages 9-22.
    7. Qudrat-Ullah, Hassan, 2017. "How to enhance the future use of energy policy simulation models through ex post validation," Energy, Elsevier, vol. 120(C), pages 58-66.
    8. van Ruijven, Bas & de Vries, Bert & van Vuuren, Detlef P. & van der Sluijs, Jeroen P., 2010. "A global model for residential energy use: Uncertainty in calibration to regional data," Energy, Elsevier, vol. 35(1), pages 269-282.
    9. Richard Loulou & Maryse Labriet, 2008. "ETSAP-TIAM: the TIMES integrated assessment model Part I: Model structure," Computational Management Science, Springer, vol. 5(1), pages 7-40, February.
    10. Goodwin, Paul & Lawton, Richard, 1999. "On the asymmetry of the symmetric MAPE," International Journal of Forecasting, Elsevier, vol. 15(4), pages 405-408, October.
    11. Li, Francis G.N. & Trutnevyte, Evelina, 2017. "Investment appraisal of cost-optimal and near-optimal pathways for the UK electricity sector transition to 2050," Applied Energy, Elsevier, vol. 189(C), pages 89-109.
    12. DeCarolis, Joseph & Daly, Hannah & Dodds, Paul & Keppo, Ilkka & Li, Francis & McDowall, Will & Pye, Steve & Strachan, Neil & Trutnevyte, Evelina & Usher, Will & Winning, Matthew & Yeh, Sonia & Zeyring, 2017. "Formalizing best practice for energy system optimization modelling," Applied Energy, Elsevier, vol. 194(C), pages 184-198.
    13. Chris Tofallis, 2015. "A better measure of relative prediction accuracy for model selection and model estimation," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 66(8), pages 1352-1362, August.
    14. Fischer, Carolyn & Herrnstadt, Evan & Morgenstern, Richard, 2009. "Understanding errors in EIA projections of energy demand," Resource and Energy Economics, Elsevier, vol. 31(3), pages 198-209, August.
    15. Evan D. Sherwin & Max Henrion & Inês M. L. Azevedo, 2018. "Estimation of the year-on-year volatility and the unpredictability of the United States energy system," Nature Energy, Nature, vol. 3(4), pages 341-346, April.
    16. Abosedra, Salah & Baghestani, Hamid, 2004. "On the predictive accuracy of crude oil futures prices," Energy Policy, Elsevier, vol. 32(12), pages 1389-1393, August.
    17. Linderoth, Hans, 2002. "Forecast errors in IEA-countries' energy consumption," Energy Policy, Elsevier, vol. 30(1), pages 53-61, January.
    18. Liao, Hua & Cai, Jia-Wei & Yang, Dong-Wei & Wei, Yi-Ming, 2016. "Why did the historical energy forecasting succeed or fail? A case study on IEA's projection," Technological Forecasting and Social Change, Elsevier, vol. 107(C), pages 90-96.
    19. Meira, Erick & Cyrino Oliveira, Fernando Luiz & de Menezes, Lilian M., 2021. "Point and interval forecasting of electricity supply via pruned ensembles," Energy, Elsevier, vol. 232(C).
    20. Richard Loulou, 2008. "ETSAP-TIAM: the TIMES integrated assessment model. part II: mathematical formulation," Computational Management Science, Springer, vol. 5(1), pages 41-66, February.
    21. Siddons, Craig & Allan, Grant & McIntyre, Stuart, 2015. "How accurate are forecasts of costs of energy? A methodological contribution," Energy Policy, Elsevier, vol. 87(C), pages 224-228.
    22. Makridakis, Spyros, 1993. "Accuracy measures: theoretical and practical concerns," International Journal of Forecasting, Elsevier, vol. 9(4), pages 527-529, December.
    23. Glotin, David & Bourgeois, Cyril & Giraudet, Louis-Gaëtan & Quirion, Philippe, 2019. "Prediction is difficult, even when it's about the past: A hindcast experiment using Res-IRF, an integrated energy-economy model," Energy Economics, Elsevier, vol. 84(S1).
    24. Steinbuks, Jevgenijs, 2019. "Assessing the accuracy of electricity production forecasts in developing countries," International Journal of Forecasting, Elsevier, vol. 35(3), pages 1175-1185.
    25. Jan-Philipp Sasse & Evelina Trutnevyte, 2020. "Regional impacts of electricity system transition in Central Europe until 2035," Nature Communications, Nature, vol. 11(1), pages 1-14, December.
    26. Hyndman, Rob J. & Koehler, Anne B., 2006. "Another look at measures of forecast accuracy," International Journal of Forecasting, Elsevier, vol. 22(4), pages 679-688.
    27. Katherine Calvin & Marshall Wise & Page Kyle & Leon Clarke & Jae Edmonds, 2017. "A Hindcast Experiment Using The Gcam 3.0 Agriculture And Land-Use Module," Climate Change Economics (CCE), World Scientific Publishing Co. Pte. Ltd., vol. 8(01), pages 1-21, February.
    28. Trutnevyte, Evelina, 2016. "Does cost optimization approximate the real-world energy transition?," Energy, Elsevier, vol. 106(C), pages 182-193.
    29. al Irsyad, Muhammad Indra & Halog, Anthony & Nepal, Rabindra, 2019. "Renewable energy projections for climate change mitigation: An analysis of uncertainty and errors," Renewable Energy, Elsevier, vol. 130(C), pages 536-546.
    30. O'Neill, Brian C. & Desai, Mausami, 2005. "Accuracy of past projections of US energy consumption," Energy Policy, Elsevier, vol. 33(8), pages 979-993, May.
    31. Müller, Jonas & Trutnevyte, Evelina, 2020. "Spatial projections of solar PV installations at subnational level: Accuracy testing of regression models," Applied Energy, Elsevier, vol. 265(C).
    32. Makridakis, Spyros & Hibon, Michele, 2000. "The M3-Competition: results, conclusions and implications," International Journal of Forecasting, Elsevier, vol. 16(4), pages 451-476.
    33. Groissböck, Markus & Pickl, Matthias J., 2016. "An analysis of the power market in Saudi Arabia: Retrospective cost and environmental optimization," Applied Energy, Elsevier, vol. 165(C), pages 548-558.
    34. Gilbert, Alexander Q. & Sovacool, Benjamin K., 2016. "Looking the wrong way: Bias, renewable electricity, and energy modelling in the United States," Energy, Elsevier, vol. 94(C), pages 533-541.
    35. DeCarolis, Joseph F. & Hunter, Kevin & Sreepathi, Sarat, 2012. "The case for repeatable analysis with energy economy optimization models," Energy Economics, Elsevier, vol. 34(6), pages 1845-1853.
    36. Trutnevyte, Evelina & McDowall, Will & Tomei, Julia & Keppo, Ilkka, 2016. "Energy scenario choices: Insights from a retrospective review of UK energy futures," Renewable and Sustainable Energy Reviews, Elsevier, vol. 55(C), pages 326-337.
    37. Bentzen, J. & Linderoth, H., 2001. "Has the accuracy of energy demand projections in the OECD countries improved since the 1970s?," Papers 01-5, Aarhus School of Business - Department of Economics.
    38. Manzoor, Davood & Aryanpur, Vahid, 2017. "Power sector development in Iran: A retrospective optimization approach," Energy, Elsevier, vol. 140(P1), pages 330-339.
    39. Hua Liao & Jia-Wei Cai & Dong-Wei Yang & Yi-Ming Wei, 2016. "Why did the historical energy forecasting succeed or fail? A case study on IEA¡¯s projection," CEEP-BIT Working Papers 92, Center for Energy and Environmental Policy Research (CEEP), Beijing Institute of Technology.
    40. Priesmann, Jan & Nolting, Lars & Praktiknjo, Aaron, 2019. "Are complex energy system models more accurate? An intra-model comparison of power system optimization models," Applied Energy, Elsevier, vol. 255(C).
    41. Wachtmeister, Henrik & Henke, Petter & Höök, Mikael, 2018. "Oil projections in retrospect: Revisions, accuracy and current uncertainty," Applied Energy, Elsevier, vol. 220(C), pages 138-153.
    42. Vaclav Smil, 2008. "Long-range energy forecasts are no more than fairy tales," Nature, Nature, vol. 453(7192), pages 154-154, May.
    43. Winebrake, James J. & Sakva, Denys, 2006. "An evaluation of errors in US energy forecasts: 1982-2003," Energy Policy, Elsevier, vol. 34(18), pages 3475-3483, December.
    44. Bashmakov, Igor, 2007. "Three laws of energy transitions," Energy Policy, Elsevier, vol. 35(7), pages 3583-3594, July.
    45. Armstrong, J. Scott & Collopy, Fred, 1992. "Error measures for generalizing about forecasting methods: Empirical comparisons," International Journal of Forecasting, Elsevier, vol. 8(1), pages 69-80, June.
    46. Chris Tofallis, 2015. "A better measure of relative prediction accuracy for model selection and model estimation," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 66(3), pages 524-524, March.
    47. Felix Creutzig & Peter Agoston & Jan Christoph Goldschmidt & Gunnar Luderer & Gregory Nemet & Robert C. Pietzcker, 2017. "The underestimated potential of solar energy to mitigate climate change," Nature Energy, Nature, vol. 2(9), pages 1-9, September.
    48. Jing Meng & Rupert Way & Elena Verdolini & Laura Diaz Anadon, 2021. "Comparing expert elicitation and model-based probabilistic technology cost forecasts for the energy transition," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 118(27), pages 1917165118-, July.
    49. Berntsen, Philip B. & Trutnevyte, Evelina, 2017. "Ensuring diversity of national energy scenarios: Bottom-up energy system model with Modeling to Generate Alternatives," Energy, Elsevier, vol. 126(C), pages 886-898.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Fan, Xiaoyu & Ji, Wei & Li, Junxian & Gao, Zhaozhao & Chen, Liubiao & Wang, Junjie, 2024. "Advancing liquid air energy storage with moving packed bed: Development and analysis from components to system level," Applied Energy, Elsevier, vol. 355(C).
    2. Sasse, Jan-Philipp & Trutnevyte, Evelina, 2023. "Cost-effective options and regional interdependencies of reaching a low-carbon European electricity system in 2035," Energy, Elsevier, vol. 282(C).
    3. Andrea Costantino, 2023. "Development, Validation, and Application of Building Energy Simulation Models for Livestock Houses: A Systematic Review," Agriculture, MDPI, vol. 13(12), pages 1-28, December.
    4. Tomasz Ząbkowski & Krzysztof Gajowniczek & Grzegorz Matejko & Jacek Brożyna & Grzegorz Mentel & Małgorzata Charytanowicz & Jolanta Jarnicka & Anna Olwert & Weronika Radziszewska & Jörg Verstraete, 2023. "Cluster-Based Approach to Estimate Demand in the Polish Power System Using Commercial Customers’ Data," Energies, MDPI, vol. 16(24), pages 1-21, December.
    5. Wen, Xin & Heinisch, Verena & Müller, Jonas & Sasse, Jan-Philipp & Trutnevyte, Evelina, 2023. "Comparison of statistical and optimization models for projecting future PV installations at a sub-national scale," Energy, Elsevier, vol. 285(C).
    6. Wen, Xin & Jaxa-Rozen, Marc & Trutnevyte, Evelina, 2023. "Hindcasting to inform the development of bottom-up electricity system models: The cases of endogenous demand and technology learning," Applied Energy, Elsevier, vol. 340(C).
    7. Gorman, Nicholas & MacGill, Iain & Bruce, Anna, 2024. "Re-dispatch simplification analysis: Confirmation holism and assessing the impact of simplifications on energy system model performance," Applied Energy, Elsevier, vol. 365(C).
    8. Jan-Philipp Sasse & Evelina Trutnevyte, 2023. "A low-carbon electricity sector in Europe risks sustaining regional inequalities in benefits and vulnerabilities," Nature Communications, Nature, vol. 14(1), pages 1-15, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wen, Xin & Jaxa-Rozen, Marc & Trutnevyte, Evelina, 2023. "Hindcasting to inform the development of bottom-up electricity system models: The cases of endogenous demand and technology learning," Applied Energy, Elsevier, vol. 340(C).
    2. Wachtmeister, Henrik & Henke, Petter & Höök, Mikael, 2018. "Oil projections in retrospect: Revisions, accuracy and current uncertainty," Applied Energy, Elsevier, vol. 220(C), pages 138-153.
    3. Wen, Xin & Heinisch, Verena & Müller, Jonas & Sasse, Jan-Philipp & Trutnevyte, Evelina, 2023. "Comparison of statistical and optimization models for projecting future PV installations at a sub-national scale," Energy, Elsevier, vol. 285(C).
    4. Berntsen, Philip B. & Trutnevyte, Evelina, 2017. "Ensuring diversity of national energy scenarios: Bottom-up energy system model with Modeling to Generate Alternatives," Energy, Elsevier, vol. 126(C), pages 886-898.
    5. Trutnevyte, Evelina, 2016. "Does cost optimization approximate the real-world energy transition?," Energy, Elsevier, vol. 106(C), pages 182-193.
    6. DeCarolis, Joseph & Daly, Hannah & Dodds, Paul & Keppo, Ilkka & Li, Francis & McDowall, Will & Pye, Steve & Strachan, Neil & Trutnevyte, Evelina & Usher, Will & Winning, Matthew & Yeh, Sonia & Zeyring, 2017. "Formalizing best practice for energy system optimization modelling," Applied Energy, Elsevier, vol. 194(C), pages 184-198.
    7. Moret, Stefano & Codina Gironès, Víctor & Bierlaire, Michel & Maréchal, François, 2017. "Characterization of input uncertainties in strategic energy planning models," Applied Energy, Elsevier, vol. 202(C), pages 597-617.
    8. Wang, Fangzhi & Liao, Hua, 2022. "Unexpected economic growth and oil price shocks," Energy Economics, Elsevier, vol. 116(C).
    9. Xexakis, Georgios & Hansmann, Ralph & Volken, Sandra P. & Trutnevyte, Evelina, 2020. "Models on the wrong track: Model-based electricity supply scenarios in Switzerland are not aligned with the perspectives of energy experts and the public," Renewable and Sustainable Energy Reviews, Elsevier, vol. 134(C).
    10. Kim, Sungil & Kim, Heeyoung, 2016. "A new metric of absolute percentage error for intermittent demand forecasts," International Journal of Forecasting, Elsevier, vol. 32(3), pages 669-679.
    11. Makridakis, Spyros & Spiliotis, Evangelos & Assimakopoulos, Vassilios, 2020. "The M4 Competition: 100,000 time series and 61 forecasting methods," International Journal of Forecasting, Elsevier, vol. 36(1), pages 54-74.
    12. Price, James & Keppo, Ilkka, 2017. "Modelling to generate alternatives: A technique to explore uncertainty in energy-environment-economy models," Applied Energy, Elsevier, vol. 195(C), pages 356-369.
    13. Colin Singleton & Peter Grindrod, 2021. "Forecasting for Battery Storage: Choosing the Error Metric," Energies, MDPI, vol. 14(19), pages 1-11, October.
    14. Charlie Wilson & Céline Guivarch & Elmar Kriegler & Bas Ruijven & Detlef P. Vuuren & Volker Krey & Valeria Jana Schwanitz & Erica L. Thompson, 2021. "Evaluating process-based integrated assessment models of climate change mitigation," Climatic Change, Springer, vol. 166(1), pages 1-22, May.
    15. Che-Yu Hung & Chien-Chih Wang & Shi-Woei Lin & Bernard C. Jiang, 2022. "An Empirical Comparison of the Sales Forecasting Performance for Plastic Tray Manufacturing Using Missing Data," Sustainability, MDPI, vol. 14(4), pages 1-21, February.
    16. Larissa Koupriouchina & Jean-Pierre van der Rest & Zvi Schwartz, 2023. "Judgmental Adjustments of Algorithmic Hotel Occupancy Forecasts: Does User Override Frequency Impact Accuracy at Different Time Horizons?," Tourism Economics, , vol. 29(8), pages 2143-2164, December.
    17. al Irsyad, Muhammad Indra & Halog, Anthony & Nepal, Rabindra, 2019. "Renewable energy projections for climate change mitigation: An analysis of uncertainty and errors," Renewable Energy, Elsevier, vol. 130(C), pages 536-546.
    18. Philippe St-Aubin & Bruno Agard, 2022. "Precision and Reliability of Forecasts Performance Metrics," Forecasting, MDPI, vol. 4(4), pages 1-22, October.
    19. Blanco, Herib & Gómez Vilchez, Jonatan J. & Nijs, Wouter & Thiel, Christian & Faaij, André, 2019. "Soft-linking of a behavioral model for transport with energy system cost optimization applied to hydrogen in EU," Renewable and Sustainable Energy Reviews, Elsevier, vol. 115(C).
    20. Crow, Daniel J.G. & Giarola, Sara & Hawkes, Adam D., 2018. "A dynamic model of global natural gas supply," Applied Energy, Elsevier, vol. 218(C), pages 452-469.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:325:y:2022:i:c:s0306261922011667. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.