IDEAS home Printed from https://ideas.repec.org/a/eee/ecomod/v487y2024ics0304380023002922.html
   My bibliography  Save this article

Linking error measures to model questions

Author

Listed:
  • Jacobs, Bas
  • Tobi, Hilde
  • Hengeveld, Geerten M.

Abstract

Models for forecasting various ecosystem properties have great potential that comes with a need for model validation. Before we can perform such validation, we need to define what it means for the model to perform well, which depends on the question being asked. Often, it seems easy to ignore the model question and take a standard well-known error measure for comparing the model to the available data. The question is whether this practice is adequate. Here, we defined different types of model-data mismatches that may be more or less relevant to different types of questions. We show that error measures differ in their sensitivity to the type of mismatch and robustness to sparse and noisy data. The results imply that a careful selection of error measures, using a clearly defined ecological question as a starting point, is vital to proper model evaluation. While we present our results as generally applicable to the validation of any type of forecasting model, we also illustrate them using cyanobacterial bloom modelling as a detailed example of a case where different questions could be asked of the same model.

Suggested Citation

  • Jacobs, Bas & Tobi, Hilde & Hengeveld, Geerten M., 2024. "Linking error measures to model questions," Ecological Modelling, Elsevier, vol. 487(C).
    • Handle: RePEc:eee:ecomod:v:487:y:2024:i:c:s0304380023002922
    DOI: 10.1016/j.ecolmodel.2023.110562
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0304380023002922
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ecolmodel.2023.110562?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Diamantis Koutsandreas & Evangelos Spiliotis & Fotios Petropoulos & Vassilios Assimakopoulos, 2022. "On the selection of forecasting accuracy measures," Journal of the Operational Research Society, Taylor & Francis Journals, vol. 73(5), pages 937-954, May.
    2. Hyndman, Rob J. & Koehler, Anne B., 2006. "Another look at measures of forecast accuracy," International Journal of Forecasting, Elsevier, vol. 22(4), pages 679-688.
    3. Chao Chen & Jamie Twycross & Jonathan M Garibaldi, 2017. "A new accuracy measure based on bounded relative error for time series forecasting," PLOS ONE, Public Library of Science, vol. 12(3), pages 1-23, March.
    4. Saloranta, Tuomo M. & Andersen, Tom, 2007. "MyLake—A multi-year lake simulation model code suitable for uncertainty and sensitivity analysis simulations," Ecological Modelling, Elsevier, vol. 207(1), pages 45-60.
    5. James W. Taylor & Keming Yu, 2016. "Using auto-regressive logit models to forecast the exceedance probability for financial risk management," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(4), pages 1069-1092, October.
    6. Janssen, Annette B.G. & Teurlincx, Sven & Beusen, Arthur H.W. & Huijbregts, Mark A.J. & Rost, Jasmijn & Schipper, Aafke M. & Seelen, Laura M.S. & Mooij, Wolf M. & Janse, Jan H., 2019. "PCLake+: A process-based ecological model to assess the trophic state of stratified and non-stratified freshwater lakes worldwide," Ecological Modelling, Elsevier, vol. 396(C), pages 23-32.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hewamalage, Hansika & Bergmeir, Christoph & Bandara, Kasun, 2021. "Recurrent Neural Networks for Time Series Forecasting: Current status and future directions," International Journal of Forecasting, Elsevier, vol. 37(1), pages 388-427.
    2. Urko Aguirre-Larracoechea & Cruz E. Borges, 2021. "Imputation for Repeated Bounded Outcome Data: Statistical and Machine-Learning Approaches," Mathematics, MDPI, vol. 9(17), pages 1-27, August.
    3. Makridakis, Spyros & Spiliotis, Evangelos & Assimakopoulos, Vassilios, 2020. "The M4 Competition: 100,000 time series and 61 forecasting methods," International Journal of Forecasting, Elsevier, vol. 36(1), pages 54-74.
    4. Brummelhuis, Raymond & Luo, Zhongmin, 2019. "Bank Net Interest Margin Forecasting and Capital Adequacy Stress Testing by Machine Learning Techniques," MPRA Paper 94779, University Library of Munich, Germany.
    5. Spiliotis, Evangelos & Nikolopoulos, Konstantinos & Assimakopoulos, Vassilios, 2019. "Tales from tails: On the empirical distributions of forecasting errors and their implication to risk," International Journal of Forecasting, Elsevier, vol. 35(2), pages 687-698.
    6. Jenny Cifuentes & Geovanny Marulanda & Antonio Bello & Javier Reneses, 2020. "Air Temperature Forecasting Using Machine Learning Techniques: A Review," Energies, MDPI, vol. 13(16), pages 1-28, August.
    7. Claudia Furlan & Cinzia Mortarino & Mohammad Salim Zahangir, 2021. "Interaction among three substitute products: an extended innovation diffusion model," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 30(1), pages 269-293, March.
    8. Wen-Jie Liu & Yu-Ting Bai & Xue-Bo Jin & Ting-Li Su & Jian-Lei Kong, 2022. "Adaptive Broad Echo State Network for Nonstationary Time Series Forecasting," Mathematics, MDPI, vol. 10(17), pages 1-21, September.
    9. Cheng-Hong Yang & Po-Yin Chang, 2020. "Forecasting the Demand for Container Throughput Using a Mixed-Precision Neural Architecture Based on CNN–LSTM," Mathematics, MDPI, vol. 8(10), pages 1-17, October.
    10. repec:prg:jnlcfu:v:2022:y:2022:i:1:id:572 is not listed on IDEAS
    11. Chang, Andrew C. & Hanson, Tyler J., 2016. "The accuracy of forecasts prepared for the Federal Open Market Committee," Journal of Economics and Business, Elsevier, vol. 83(C), pages 23-43.
    12. Ling Tang & Chengyuan Zhang & Tingfei Li & Ling Li, 2021. "A novel BEMD-based method for forecasting tourist volume with search engine data," Tourism Economics, , vol. 27(5), pages 1015-1038, August.
    13. Michael Vössing & Niklas Kühl & Matteo Lind & Gerhard Satzger, 2022. "Designing Transparency for Effective Human-AI Collaboration," Information Systems Frontiers, Springer, vol. 24(3), pages 877-895, June.
    14. Frank, Johannes, 2023. "Forecasting realized volatility in turbulent times using temporal fusion transformers," FAU Discussion Papers in Economics 03/2023, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics.
    15. Kourentzes, Nikolaos & Petropoulos, Fotios & Trapero, Juan R., 2014. "Improving forecasting by estimating time series structural components across multiple frequencies," International Journal of Forecasting, Elsevier, vol. 30(2), pages 291-302.
    16. Jeon, Yunho & Seong, Sihyeon, 2022. "Robust recurrent network model for intermittent time-series forecasting," International Journal of Forecasting, Elsevier, vol. 38(4), pages 1415-1425.
    17. Snyder, Ralph D. & Ord, J. Keith & Koehler, Anne B. & McLaren, Keith R. & Beaumont, Adrian N., 2017. "Forecasting compositional time series: A state space approach," International Journal of Forecasting, Elsevier, vol. 33(2), pages 502-512.
    18. Paulo Júlio & Pedro M. Esperança, 2012. "Evaluating the forecast quality of GDP components: An application to G7," GEE Papers 0047, Gabinete de Estratégia e Estudos, Ministério da Economia, revised Apr 2012.
    19. Rivera, Nilza & Guzmán, Juan Ignacio & Jara, José Joaquín & Lagos, Gustavo, 2021. "Evaluation of econometric models of secondary refined copper supply," Resources Policy, Elsevier, vol. 73(C).
    20. Cameron Roach & Rob Hyndman & Souhaib Ben Taieb, 2021. "Non‐linear mixed‐effects models for time series forecasting of smart meter demand," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 40(6), pages 1118-1130, September.
    21. Massimo Guidolin & Manuela Pedio, 2019. "Forecasting and Trading Monetary Policy Effects on the Riskless Yield Curve with Regime Switching Nelson†Siegel Models," Working Papers 639, IGIER (Innocenzo Gasparini Institute for Economic Research), Bocconi University.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ecomod:v:487:y:2024:i:c:s0304380023002922. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/ecological-modelling .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.