IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0146467.html
   My bibliography  Save this article

Ecosystem Model Skill Assessment. Yes We Can!

Author

Listed:
  • Erik Olsen
  • Gavin Fay
  • Sarah Gaichas
  • Robert Gamble
  • Sean Lucey
  • Jason S Link

Abstract

Need to Assess the Skill of Ecosystem Models: Accelerated changes to global ecosystems call for holistic and integrated analyses of past, present and future states under various pressures to adequately understand current and projected future system states. Ecosystem models can inform management of human activities in a complex and changing environment, but are these models reliable? Ensuring that models are reliable for addressing management questions requires evaluating their skill in representing real-world processes and dynamics. Skill has been evaluated for just a limited set of some biophysical models. A range of skill assessment methods have been reviewed but skill assessment of full marine ecosystem models has not yet been attempted. Northeast US Atlantis Marine Ecosystem Model: We assessed the skill of the Northeast U.S. (NEUS) Atlantis marine ecosystem model by comparing 10-year model forecasts with observed data. Model forecast performance was compared to that obtained from a 40-year hindcast. Multiple metrics (average absolute error, root mean squared error, modeling efficiency, and Spearman rank correlation), and a suite of time-series (species biomass, fisheries landings, and ecosystem indicators) were used to adequately measure model skill. Overall, the NEUS model performed above average and thus better than expected for the key species that had been the focus of the model tuning. Model forecast skill was comparable to the hindcast skill, showing that model performance does not degenerate in a 10-year forecast mode, an important characteristic for an end-to-end ecosystem model to be useful for strategic management purposes. Skill Assessment Is Both Possible and Advisable: We identify best-practice approaches for end-to-end ecosystem model skill assessment that would improve both operational use of other ecosystem models and future model development. We show that it is possible to not only assess the skill of a complicated marine ecosystem model, but that it is necessary do so to instill confidence in model results and encourage their use for strategic management. Our methods are applicable to any type of predictive model, and should be considered for use in fields outside ecology (e.g. economics, climate change, and risk assessment).

Suggested Citation

  • Erik Olsen & Gavin Fay & Sarah Gaichas & Robert Gamble & Sean Lucey & Jason S Link, 2016. "Ecosystem Model Skill Assessment. Yes We Can!," PLOS ONE, Public Library of Science, vol. 11(1), pages 1-24, January.
  • Handle: RePEc:plo:pone00:0146467
    DOI: 10.1371/journal.pone.0146467
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0146467
    Download Restriction: no

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0146467&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pone.0146467?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Graham Elliott & Allan Timmermann, 2016. "Economic Forecasting," Economics Books, Princeton University Press, edition 1, number 10740.
    2. Hendry, David F. & Clements, Michael P., 2003. "Economic forecasting: some lessons from recent research," Economic Modelling, Elsevier, vol. 20(2), pages 301-329, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Barbara Bauer & Jan Horbowy & Mika Rahikainen & Nataliia Kulatska & Bärbel Müller-Karulis & Maciej T Tomczak & Valerio Bartolino, 2019. "Model uncertainty and simulated multispecies fisheries management advice in the Baltic Sea," PLOS ONE, Public Library of Science, vol. 14(1), pages 1-22, January.
    2. Planque, Benjamin & Aarflot, Johanna M. & Buttay, Lucie & Carroll, JoLynn & Fransner, Filippa & Hansen, Cecilie & Husson, Bérengère & Langangen, Øystein & Lindstrøm, Ulf & Pedersen, Torstein & Primice, 2022. "A standard protocol for describing the evaluation of ecological models," Ecological Modelling, Elsevier, vol. 471(C).
    3. Lopez de Gamiz-Zearra, A. & Hansen, C. & Corrales, X. & Andonegi, E., 2024. "Increasing the reliability of the Bay of Biscay Atlantis model: A sensitivity analysis to parameters perturbations using a Morris screening approach," Ecological Modelling, Elsevier, vol. 488(C).
    4. Perryman, Holly A. & Kaplan, Isaac C. & Blanchard, Julia L. & Fay, Gavin & Gaichas, Sarah K. & McGregor, Vidette L. & Morzaria-Luna, Hem Nalini & Porobic, Javier & Townsend, Howard & Fulton, Elizabeth, 2023. "Atlantis Ecosystem Model Summit 2022: Report from a workshop," Ecological Modelling, Elsevier, vol. 483(C).
    5. Bossier, Sieme & Nielsen, J. Rasmus & Almroth-Rosell, Elin & Höglund, Anders & Bastardie, Francois & Neuenfeldt, Stefan & Wåhlström, Iréne & Christensen, Asbjørn, 2021. "Integrated ecosystem impacts of climate change and eutrophication on main Baltic fishery resources," Ecological Modelling, Elsevier, vol. 453(C).
    6. Püts, Miriam & Taylor, Marc & Núñez-Riboni, Ismael & Steenbeek, Jeroen & Stäbler, Moritz & Möllmann, Christian & Kempf, Alexander, 2020. "Insights on integrating habitat preferences in process-oriented ecological models – a case study of the southern North Sea," Ecological Modelling, Elsevier, vol. 431(C).
    7. Laura S Storch & Sarah M Glaser & Hao Ye & Andrew A Rosenberg, 2017. "Stock assessment and end-to-end ecosystem models alter dynamics of fisheries data," PLOS ONE, Public Library of Science, vol. 12(2), pages 1-11, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Muhammad Nadim Hanif & Muhammad Jahanzeb Malik, 2015. "Evaluating the Performance of Inflation Forecasting Models of Pakistan," SBP Research Bulletin, State Bank of Pakistan, Research Department, vol. 11, pages 43-78.
    2. Norman Swanson & Oleg Korenok, 2006. "The Incremental Predictive Information Associated with Using Theoretical New Keynesian DSGE Models Versus Simple Linear Alternatives," Departmental Working Papers 200615, Rutgers University, Department of Economics.
    3. Shahzad Ahmad & Farooq Pasha, 2015. "A Pragmatic Model for Monetary Policy Analysis I: The Case of Pakistan," SBP Research Bulletin, State Bank of Pakistan, Research Department, vol. 11, pages 1-42.
    4. Christophe Chorro & Florian Ielpo & Benoît Sévi, 2017. "The contribution of jumps to forecasting the density of returns," Post-Print halshs-01442618, HAL.
    5. Carlos Medel, 2017. "Forecasting Chilean inflation with the hybrid new keynesian Phillips curve: globalisation, combination, and accuracy," Journal Economía Chilena (The Chilean Economy), Central Bank of Chile, vol. 20(3), pages 004-050, December.
    6. Hayashi, Masayoshi, 2014. "Forecasting welfare caseloads: The case of the Japanese public assistance program," Socio-Economic Planning Sciences, Elsevier, vol. 48(2), pages 105-114.
    7. Julia Eichholz & Thorsten Knauer & Sandra Winkelmann, 2023. "Digital Maturity of Forecasting and its Impact in Times of Crisis," Schmalenbach Journal of Business Research, Springer, vol. 75(4), pages 443-481, December.
    8. Gloria González‐Rivera & C. Vladimir Rodríguez‐Caballero & Esther Ruiz, 2024. "Expecting the unexpected: Stressed scenarios for economic growth," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 39(5), pages 926-942, August.
    9. Paroissien, Emmanuel, 2020. "Forecasting bulk prices of Bordeaux wines using leading indicators," International Journal of Forecasting, Elsevier, vol. 36(2), pages 292-309.
    10. Rossi, Barbara, 2006. "Are Exchange Rates Really Random Walks? Some Evidence Robust To Parameter Instability," Macroeconomic Dynamics, Cambridge University Press, vol. 10(1), pages 20-38, February.
    11. Alex Tagliabracci, 2020. "Asymmetry in the conditional distribution of euro-area inflation," Temi di discussione (Economic working papers) 1270, Bank of Italy, Economic Research and International Relations Area.
    12. Dennis Kant & Andreas Pick & Jasper de Winter, 2022. "Nowcasting GDP using machine learning methods," Working Papers 754, DNB.
    13. Carlos A. Medel, 2015. "Probabilidad Clásica de Sobreajuste con Criterios de Información: Estimaciones con Series Macroeconómicas Chilenas," Revista de Analisis Economico – Economic Analysis Review, Universidad Alberto Hurtado/School of Economics and Business, vol. 30(1), pages 57-72, Abril.
    14. Dimitrios Papastamos & Fotis Mouzakis & Simon Stevenson, 2014. "Rationality and Momentum in Real Estate Investment Forecasts," Real Estate & Planning Working Papers rep-wp2014-07, Henley Business School, University of Reading.
    15. Chen, Heng & Li, Xu & Pei, Guangyu & Xin, Qian, 2024. "Heterogeneous overreaction in expectation formation: Evidence and theory," Journal of Economic Theory, Elsevier, vol. 218(C).
    16. Kaya, Huseyin, 2013. "Forecasting the yield curve and the role of macroeconomic information in Turkey," Economic Modelling, Elsevier, vol. 33(C), pages 1-7.
    17. Pat McAllister & Graeme Newell & George Matysiak, 2005. "An Evaluation Of The Performance Of UK Real Estate Forecasters," Real Estate & Planning Working Papers rep-wp2005-23, Henley Business School, University of Reading.
    18. Nowotarski, Jakub & Weron, Rafał, 2018. "Recent advances in electricity price forecasting: A review of probabilistic forecasting," Renewable and Sustainable Energy Reviews, Elsevier, vol. 81(P1), pages 1548-1568.
    19. Jeff Schott & Jugal Kalita, 2011. "Neuro‐fuzzy time‐series analysis of large‐volume data," Intelligent Systems in Accounting, Finance and Management, John Wiley & Sons, Ltd., vol. 18(1), pages 39-57, January.
    20. Eraslan, Sercan & Schröder, Maximilian, 2019. "Nowcasting GDP with a large factor model space," Discussion Papers 41/2019, Deutsche Bundesbank.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0146467. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.