IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0138212.html

Beyond Corroboration: Strengthening Model Validation by Looking for Unexpected Patterns

Author

Listed:
  • Guillaume Chérel
  • Clémentine Cottineau
  • Romain Reuillon

Abstract

Models of emergent phenomena are designed to provide an explanation to global-scale phenomena from local-scale processes. Model validation is commonly done by verifying that the model is able to reproduce the patterns to be explained. We argue that robust validation must not only be based on corroboration, but also on attempting to falsify the model, i.e. making sure that the model behaves soundly for any reasonable input and parameter values. We propose an open-ended evolutionary method based on Novelty Search to look for the diverse patterns a model can produce. The Pattern Space Exploration method was tested on a model of collective motion and compared to three common a priori sampling experiment designs. The method successfully discovered all known qualitatively different kinds of collective motion, and performed much better than the a priori sampling methods. The method was then applied to a case study of city system dynamics to explore the model’s predicted values of city hierarchisation and population growth. This case study showed that the method can provide insights on potential predictive scenarios as well as falsifiers of the model when the simulated dynamics are highly unrealistic.

Suggested Citation

  • Guillaume Chérel & Clémentine Cottineau & Romain Reuillon, 2015. "Beyond Corroboration: Strengthening Model Validation by Looking for Unexpected Patterns," PLOS ONE, Public Library of Science, vol. 10(9), pages 1-28, September.
    • Handle: RePEc:plo:pone00:0138212
    DOI: 10.1371/journal.pone.0138212
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0138212
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0138212&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0138212?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Paul Windrum & Giorgio Fagiolo & Alessio Moneta, 2007. "Empirical Validation of Agent-Based Models: Alternatives and Prospects," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 10(2), pages 1-8.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Aymeric Vié & Alfredo J. Morales, 2021. "How Connected is Too Connected? Impact of Network Topology on Systemic Risk and Collapse of Complex Economic Systems," Computational Economics, Springer;Society for Computational Economics, vol. 57(4), pages 1327-1351, April.
    2. Assogba, Gildas G.C. & Speelman, Erika N. & Berre, David & Adam, Myriam & Descheemaeker, Katrien, 2025. "Exploring opportunities to improve crop-livestock integration and production in mixed farms with a serious game: The case of semi-arid Burkina Faso," Agricultural Systems, Elsevier, vol. 227(C).
    3. Juste Raimbault, 2019. "Second-order control of complex systems with correlated synthetic data," Post-Print halshs-02376968, HAL.
    4. Aymeric Vi'e & Alfredo J. Morales, 2019. "How connected is too connected? Impact of network topology on systemic risk and collapse of complex economic systems," Papers 1912.09814, arXiv.org.
    5. Jahel, Camille & Bourgeois, Robin & Bourgoin, Jérémy & Daré, William's & De Lattre-Gasquet, Marie & Delay, Etienne & Dumas, Patrice & Le Page, Christophe & Piraux, Marc & Prudhomme, Rémi, 2023. "The future of social-ecological systems at the crossroads of quantitative and qualitative methods," Technological Forecasting and Social Change, Elsevier, vol. 193(C).
    6. Mert Edali & Gönenç Yücel, 2020. "Analysis of an individual‐based influenza epidemic model using random forest metamodels and adaptive sequential sampling," Systems Research and Behavioral Science, Wiley Blackwell, vol. 37(6), pages 936-958, November.
    7. Juste Raimbault & Julien Perret, 2019. "Generating urban morphologies at large scales," Post-Print halshs-02265415, HAL.
    8. Juste Raimbault, 2018. "Calibration of a density-based model of urban morphogenesis," PLOS ONE, Public Library of Science, vol. 13(9), pages 1-18, September.
    9. Verstegen, Judith A. & Goch, Katarzyna, 2022. "Pattern-oriented calibration and validation of urban growth models: Case studies of Dublin, Milan and Warsaw," Land Use Policy, Elsevier, vol. 112(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:osf:agrixi:xutyz_v1 is not listed on IDEAS
    2. LeBaron Blake & Winker Peter, 2008. "Introduction to the Special Issue on Agent-Based Models for Economic Policy Advice," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 228(2-3), pages 141-148, April.
    3. Arnaud Z. Dragicevic, 2019. "Market Coordination Under Non-Equilibrium Dynamics," Networks and Spatial Economics, Springer, vol. 19(3), pages 697-715, September.
    4. Federico Gabriele & Aldo Glielmo & Marco Taboga, 2025. "Heterogeneous RBCs via Deep Multi-Agent Reinforcement Learning," Papers 2510.12272, arXiv.org, revised Feb 2026.
    5. Corrado Monti & Marco Pangallo & Gianmarco De Francisci Morales & Francesco Bonchi, 2022. "On learning agent-based models from data," Papers 2205.05052, arXiv.org, revised Nov 2022.
    6. Giorgio Fagiolo & Mattia Guerini & Francesco Lamperti & Alessio Moneta & Andrea Roventini, 2017. "Validation of Agent-Based Models in Economics and Finance," LEM Papers Series 2017/23, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    7. Gomes, Sharlene L. & Hermans, Leon M. & Thissen, Wil A.H., 2018. "Extending community operational research to address institutional aspects of societal problems: Experiences from peri-urban Bangladesh," European Journal of Operational Research, Elsevier, vol. 268(3), pages 904-917.
    8. Flaminio Squazzoni, 2010. "The impact of agent-based models in the social sciences after 15 years of incursions," History of Economic Ideas, Fabrizio Serra Editore, Pisa - Roma, vol. 18(2), pages 197-234.
    9. John DiNardo & David S. Lee, 2010. "Program Evaluation and Research Designs," Working Papers 1228, Princeton University, Department of Economics, Industrial Relations Section..
    10. Hossein Sabzian & Mohammad Ali Shafia & Ali Maleki & Seyeed Mostapha Seyeed Hashemi & Ali Baghaei & Hossein Gharib, 2019. "Theories and Practice of Agent based Modeling: Some practical Implications for Economic Planners," Papers 1901.08932, arXiv.org.
    11. Colasante, Annarita, 2016. "Evolution of Cooperation in Public Good Game," MPRA Paper 72577, University Library of Munich, Germany.
    12. Fontana, Magda, 2010. "Can neoclassical economics handle complexity? The fallacy of the oil spot dynamic," Journal of Economic Behavior & Organization, Elsevier, vol. 76(3), pages 584-596, December.
    13. Sylvain Barde & Sander van der Hoog, 2017. "An empirical validation protocol for large-scale agent-based models," Studies in Economics 1712, School of Economics, University of Kent.
    14. Colasante, Annarita, 2017. "Selection of the distributional rule as an alternative tool to foster cooperation in a Public Good Game," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 468(C), pages 482-492.
    15. Sandrine Jacob Leal & Mauro Napoletano & Andrea Roventini & Giorgio Fagiolo, 2016. "Rock around the clock: An agent-based model of low- and high-frequency trading," Journal of Evolutionary Economics, Springer, vol. 26(1), pages 49-76, March.
    16. Bert, Federico E. & Rovere, Santiago L. & Macal, Charles M. & North, Michael J. & Podestá, Guillermo P., 2014. "Lessons from a comprehensive validation of an agent based-model: The experience of the Pampas Model of Argentinean agricultural systems," Ecological Modelling, Elsevier, vol. 273(C), pages 284-298.
    17. Wang, Liming & Sun, Xuchu & Zhu, Hongliang & Li, Tangrong, 2025. "Exploring the dynamic impact of transaction taxes on market quality in HFT and non-HFT environments: An agent-based modeling approach," The North American Journal of Economics and Finance, Elsevier, vol. 76(C).
    18. Faber, Albert & Valente, Marco & Janssen, Peter, 2010. "Exploring domestic micro-cogeneration in the Netherlands: An agent-based demand model for technology diffusion," Energy Policy, Elsevier, vol. 38(6), pages 2763-2775, June.
    19. Francesco Lamperti, 2015. "An Information Theoretic Criterion for Empirical Validation of Time Series Models," LEM Papers Series 2015/02, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    20. Albert Faber & Koen Frenken, 2008. "Models in evolutionary economics and environmental policy: Towards an evolutionary environmental economics," Innovation Studies Utrecht (ISU) working paper series 08-15, Utrecht University, Department of Innovation Studies, revised Apr 2008.
    21. Muqtafi Akhmad & Shuang Chang & Hiroshi Deguchi, 2021. "Closed-mindedness and insulation in groupthink: their effects and the devil’s advocacy as a preventive measure," Journal of Computational Social Science, Springer, vol. 4(2), pages 455-478, November.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0138212. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.