IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0138212.html
   My bibliography  Save this article

Beyond Corroboration: Strengthening Model Validation by Looking for Unexpected Patterns

Author

Listed:
  • Guillaume Chérel
  • Clémentine Cottineau
  • Romain Reuillon

Abstract

Models of emergent phenomena are designed to provide an explanation to global-scale phenomena from local-scale processes. Model validation is commonly done by verifying that the model is able to reproduce the patterns to be explained. We argue that robust validation must not only be based on corroboration, but also on attempting to falsify the model, i.e. making sure that the model behaves soundly for any reasonable input and parameter values. We propose an open-ended evolutionary method based on Novelty Search to look for the diverse patterns a model can produce. The Pattern Space Exploration method was tested on a model of collective motion and compared to three common a priori sampling experiment designs. The method successfully discovered all known qualitatively different kinds of collective motion, and performed much better than the a priori sampling methods. The method was then applied to a case study of city system dynamics to explore the model’s predicted values of city hierarchisation and population growth. This case study showed that the method can provide insights on potential predictive scenarios as well as falsifiers of the model when the simulated dynamics are highly unrealistic.

Suggested Citation

  • Guillaume Chérel & Clémentine Cottineau & Romain Reuillon, 2015. "Beyond Corroboration: Strengthening Model Validation by Looking for Unexpected Patterns," PLOS ONE, Public Library of Science, vol. 10(9), pages 1-28, September.
    • Handle: RePEc:plo:pone00:0138212
    DOI: 10.1371/journal.pone.0138212
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0138212
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0138212&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0138212?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Paul Windrum & Giorgio Fagiolo & Alessio Moneta, 2007. "Empirical Validation of Agent-Based Models: Alternatives and Prospects," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 10(2), pages 1-8.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Aymeric Vié & Alfredo J. Morales, 2021. "How Connected is Too Connected? Impact of Network Topology on Systemic Risk and Collapse of Complex Economic Systems," Computational Economics, Springer;Society for Computational Economics, vol. 57(4), pages 1327-1351, April.
    2. Aymeric Vi'e & Alfredo J. Morales, 2019. "How connected is too connected? Impact of network topology on systemic risk and collapse of complex economic systems," Papers 1912.09814, arXiv.org.
    3. Jahel, Camille & Bourgeois, Robin & Bourgoin, Jérémy & Daré, William's & De Lattre-Gasquet, Marie & Delay, Etienne & Dumas, Patrice & Le Page, Christophe & Piraux, Marc & Prudhomme, Rémi, 2023. "The future of social-ecological systems at the crossroads of quantitative and qualitative methods," Technological Forecasting and Social Change, Elsevier, vol. 193(C).
    4. Mert Edali & Gönenç Yücel, 2020. "Analysis of an individual‐based influenza epidemic model using random forest metamodels and adaptive sequential sampling," Systems Research and Behavioral Science, Wiley Blackwell, vol. 37(6), pages 936-958, November.
    5. Juste Raimbault & Julien Perret, 2019. "Generating urban morphologies at large scales," Post-Print halshs-02265415, HAL.
    6. Juste Raimbault, 2018. "Calibration of a density-based model of urban morphogenesis," PLOS ONE, Public Library of Science, vol. 13(9), pages 1-18, September.
    7. Verstegen, Judith A. & Goch, Katarzyna, 2022. "Pattern-oriented calibration and validation of urban growth models: Case studies of Dublin, Milan and Warsaw," Land Use Policy, Elsevier, vol. 112(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lamperti, Francesco & Bosetti, Valentina & Roventini, Andrea & Tavoni, Massimo & Treibich, Tania, 2021. "Three green financial policies to address climate risks," Journal of Financial Stability, Elsevier, vol. 54(C).
    2. Francesco Lamperti & Giovanni Dosi & Mauro Napoletano & Andrea Roventini & Alessandro Sapio, 2018. "And then he wasn't a she : Climate change and green transitions in an agent-based integrated assessment model," Working Papers hal-03443464, HAL.
    3. Kukacka, Jiri & Jang, Tae-Seok & Sacht, Stephen, 2018. "On the estimation of behavioral macroeconomic models via simulated maximum likelihood," Economics Working Papers 2018-11, Christian-Albrechts-University of Kiel, Department of Economics.
    4. Juan Manuel Larrosa, 2016. "Agentes computacionales y análisis económico," Revista de Economía Institucional, Universidad Externado de Colombia - Facultad de Economía, vol. 18(34), pages 87-113, January-J.
    5. Sarah Wolf & Steffen Fürst & Antoine Mandel & Wiebke Lass & Daniel Lincke & Federico Pablo-Marti & Carlo Jaeger, 2013. "A multi-agent model of several economic regions," PSE - Labex "OSE-Ouvrir la Science Economique" halshs-00825217, HAL.
    6. LeBaron Blake & Winker Peter, 2008. "Introduction to the Special Issue on Agent-Based Models for Economic Policy Advice," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 228(2-3), pages 141-148, April.
    7. Arnaud Z. Dragicevic, 2019. "Market Coordination Under Non-Equilibrium Dynamics," Networks and Spatial Economics, Springer, vol. 19(3), pages 697-715, September.
    8. Corrado Monti & Marco Pangallo & Gianmarco De Francisci Morales & Francesco Bonchi, 2022. "On learning agent-based models from data," Papers 2205.05052, arXiv.org, revised Nov 2022.
    9. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    10. Sandrine Jacob Leal & Mauro Napoletano & Andrea Roventini & Giorgio Fagiolo, 2016. "Rock around the clock: An agent-based model of low- and high-frequency trading," Journal of Evolutionary Economics, Springer, vol. 26(1), pages 49-76, March.
    11. Giorgio Fagiolo & Mattia Guerini & Francesco Lamperti & Alessio Moneta & Andrea Roventini, 2017. "Validation of Agent-Based Models in Economics and Finance," LEM Papers Series 2017/23, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    12. Gianluca Capone & Franco Malerba & Richard R. Nelson & Luigi Orsenigo & Sidney G. Winter, 2019. "History friendly models: retrospective and future perspectives," Eurasian Business Review, Springer;Eurasia Business and Economics Society, vol. 9(1), pages 1-23, March.
    13. Gomes, Sharlene L. & Hermans, Leon M. & Thissen, Wil A.H., 2018. "Extending community operational research to address institutional aspects of societal problems: Experiences from peri-urban Bangladesh," European Journal of Operational Research, Elsevier, vol. 268(3), pages 904-917.
    14. Dosi, Giovanni & Palagi, Elisa & Roventini, Andrea & Russo, Emanuele, 2023. "Do patents really foster innovation in the pharmaceutical sector? Results from an evolutionary, agent-based model," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 564-589.
    15. Flaminio Squazzoni, 2010. "The impact of agent-based models in the social sciences after 15 years of incursions," History of Economic Ideas, Fabrizio Serra Editore, Pisa - Roma, vol. 18(2), pages 197-234.
    16. John DiNardo & David S. Lee, 2010. "Program Evaluation and Research Designs," Working Papers 1228, Princeton University, Department of Economics, Industrial Relations Section..
    17. Mattia Guerini & Francesco Lamperti & Mauro Napoletano & Andrea Roventini & Tania Treibich, 2022. "Unconventional monetary policies in an agent-based model with mark-to-market standards," Review of Evolutionary Political Economy, Springer, vol. 3(1), pages 73-107, April.
    18. Fatouh, Mahmoud & Markose, Sheri & Giansante, Simone, 2021. "The impact of quantitative easing on UK bank lending: Why banks do not lend to businesses?," Journal of Economic Behavior & Organization, Elsevier, vol. 183(C), pages 928-953.
    19. Hossein Sabzian & Mohammad Ali Shafia & Ali Maleki & Seyeed Mostapha Seyeed Hashemi & Ali Baghaei & Hossein Gharib, 2019. "Theories and Practice of Agent based Modeling: Some practical Implications for Economic Planners," Papers 1901.08932, arXiv.org.
    20. Colasante, Annarita, 2016. "Evolution of Cooperation in Public Good Game," MPRA Paper 72577, University Library of Munich, Germany.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0138212. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.