IDEAS home Printed from https://ideas.repec.org/p/spo/wpmain/infohdl2441-4pa18fd9lf9h59m4vfavfcf61e.html
   My bibliography  Save this paper

An empirical validation protocol for large-scale agent-based models

Author

Listed:
  • Sylvain Barde

    (Sciences Po Paris - Institut d'études politiques de Paris (IEP Paris))

  • Sander Van Der Hoog

    (Universität Bielefeld)

Abstract

Despite recent advances in bringing agent-based models (ABMs) to the data, the estimation or calibration of model parameters remains a challenge, especially when it comes to large-scale agentbased macroeconomic models. Most methods, such as the method of simulated moments (MSM), require in-the-loop simulation of new data, which may not be feasible for such computationally heavy simulation models. The purpose of this paper is to provide a proof-of-concept of a generic empirical validation methodology for such large-scale simulation models. We introduce an alternative ‘large-scale’ empirical validation approach, and apply it to the Eurace@Unibi macroeconomic simulation model (Dawid et al., 2016). This model was selected because it displays strong emergent behaviour and is able to generate a wide variety of nonlinear economic dynamics, including endogenous business- and financial cycles. In addition, it is a computationally heavy simulation model, so it fits our targeted use-case. The validation protocol consists of three stages. At the first stage we use Nearly-Orthogonal Latin Hypercube sampling (NOLH) in order to generate a set of 513 parameter combinations with good space-filling properties. At the second stage we use the recently developed Markov Information Criterion (MIC) to score the simulated data against empirical data. Finally, at the third stage we use stochastic kriging to construct a surrogate model of the MIC response surface, resulting in an interpolation of the response surface as a function of the parameters. The parameter combinations providing the best fit to the data are then identified as the local minima of the interpolated MIC response surface. The Model Confidence Set (MCS) procedure of Hansen et al. (2011) is used to restrict the set of model calibrations to those models that cannot be rejected to have equal predictive ability, at a given confidence level. Validation of the surrogate model is carried out by re-running the second stage of the analysis on the so identified optima and cross-checking that the realised MIC scores equal the MIC scores predicted by the surrogate model. The results we obtain so far look promising as a first proof-of-concept for the empirical validation methodology since we are able to validate the model using empirical data series for 30 OECD countries and the euro area. The internal validation procedure of the surrogate model also suggests that the combination of NOLH sampling, MIC measurement and stochastic kriging yields reliable predictions of the MIC scores for samples not included in the original NOLH sample set. In our opinion, this is a strong indication that the method we propose could provide a viable statistical machine learning technique for the empirical validation of (large-scale) ABMs

Suggested Citation

  • Sylvain Barde & Sander Van Der Hoog, 2017. "An empirical validation protocol for large-scale agent-based models," Sciences Po publications 17/12, Sciences Po.
  • Handle: RePEc:spo:wpmain:info:hdl:2441/4pa18fd9lf9h59m4vfavfcf61e
    as

    Download full text from publisher

    File URL: https://spire.sciencespo.fr/hdl:/2441/4pa18fd9lf9h59m4vfavfcf61e/resources/2017-anempirical-validationprotocol-sbarde.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Barde, Sylvain, 2016. "Direct comparison of agent-based models of herding in financial markets," Journal of Economic Dynamics and Control, Elsevier, vol. 73(C), pages 329-353.
    2. Kleijnen, Jack P.C., 2017. "Regression and Kriging metamodels with their experimental designs in simulation: A review," European Journal of Operational Research, Elsevier, vol. 256(1), pages 1-16.
    3. G. Fagiolo & C. Birchenhall & P. Windrum, 2007. "Empirical Validation in Agent-based Models: Introduction to the Special Issue," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 189-194, October.
    4. Jasmina Arifovic & James Bullard & Olena Kostyshyna, 2013. "Social Learning and Monetary Policy Rules," Economic Journal, Royal Economic Society, vol. 123(567), pages 38-76, March.
    5. Sylvain Barde & Sander van der Hoog, 2017. "An empirical validation protocol for large-scale agent-based models," Working Papers hal-03458672, HAL.
    6. Assenza, Tiziana & Delli Gatti, Domenico, 2013. "E Pluribus Unum: Macroeconomic modelling for multi-agent economies," Journal of Economic Dynamics and Control, Elsevier, vol. 37(8), pages 1659-1682.
    7. Paul Windrum & Giorgio Fagiolo & Alessio Moneta, 2007. "Empirical Validation of Agent-Based Models: Alternatives and Prospects," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 10(2), pages 1-8.
    8. Alexandru Mandes & Peter Winker, 2017. "Complexity and model comparison in agent based modeling of financial markets," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 12(3), pages 469-506, October.
    9. G. Dosi & M. C. Pereira & M. E. Virgillito, 2018. "On the robustness of the fat-tailed distribution of firm growth rates: a global sensitivity analysis," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 13(1), pages 173-193, April.
    10. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    11. McFadden, Daniel, 1980. "Econometric Models for Probabilistic Choice among Products," The Journal of Business, University of Chicago Press, vol. 53(3), pages 13-29, July.
    12. Peter R. Hansen & Asger Lunde & James M. Nason, 2011. "The Model Confidence Set," Econometrica, Econometric Society, vol. 79(2), pages 453-497, March.
    13. Dosi, Giovanni & Fagiolo, Giorgio & Roventini, Andrea, 2010. "Schumpeter meeting Keynes: A policy-friendly model of endogenous growth and business cycles," Journal of Economic Dynamics and Control, Elsevier, vol. 34(9), pages 1748-1767, September.
    14. Isabelle Salle & Murat Yıldızoğlu, 2014. "Efficient Sampling and Meta-Modeling for Computational Economic Models," Computational Economics, Springer;Society for Computational Economics, vol. 44(4), pages 507-536, December.
    15. Sylvain Barde, 2017. "A Practical, Accurate, Information Criterion for Nth Order Markov Processes," Computational Economics, Springer;Society for Computational Economics, vol. 50(2), pages 281-324, August.
    16. Dosi, Giovanni & Fagiolo, Giorgio & Napoletano, Mauro & Roventini, Andrea & Treibich, Tania, 2015. "Fiscal and monetary policies in complex evolving economies," Journal of Economic Dynamics and Control, Elsevier, vol. 52(C), pages 166-189.
    17. Anna, Petrenko, 2016. "Мaркування готової продукції як складова частина інформаційного забезпечення маркетингової діяльності підприємств овочепродуктового підкомплексу," Agricultural and Resource Economics: International Scientific E-Journal, Agricultural and Resource Economics: International Scientific E-Journal, vol. 2(1), March.
    18. Dosi, Giovanni & Fagiolo, Giorgio & Napoletano, Mauro & Roventini, Andrea & Treibich, Tania, 2015. "Fiscal and monetary policies in complex evolving economies," Journal of Economic Dynamics and Control, Elsevier, vol. 52(C), pages 166-189.
    19. Antoine Mandel & Carlo Jaeger & Steffen Fürst & Wiebke Lass & Daniel Lincke & Frank Meissner & Federico Pablo-Marti & Sarah Wolf, 2010. "Agent-based dynamics in disaggregated growth models," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-00542442, HAL.
    20. Guerini, Mattia & Moneta, Alessio, 2017. "A method for agent-based models validation," Journal of Economic Dynamics and Control, Elsevier, vol. 82(C), pages 125-141.
    21. Matteo G. Richiardi, 2017. "The Future of Agent-Based Modeling," Eastern Economic Journal, Palgrave Macmillan;Eastern Economic Association, vol. 43(2), pages 271-287, March.
    22. Wieland, Volker & Cwik, Tobias & Müller, Gernot J. & Schmidt, Sebastian & Wolters, Maik, 2012. "A new comparative approach to macroeconomic modeling and policy analysis," Journal of Economic Behavior & Organization, Elsevier, vol. 83(3), pages 523-541.
    23. Sander van der Hoog, 2017. "Deep Learning in (and of) Agent-Based Models: A Prospectus," Papers 1706.06302, arXiv.org.
    24. Gilli, M. & Winker, P., 2003. "A global optimization heuristic for estimating agent based models," Computational Statistics & Data Analysis, Elsevier, vol. 42(3), pages 299-312, March.
    25. Luca Riccetti & Alberto Russo & Mauro Gallegati, 2015. "An agent based decentralized matching macroeconomic model," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 10(2), pages 305-332, October.
    26. Peter Winker and Manfred Gilli, 2001. "Indirect Estimation of the Parameters of Agent Based Models of Financial Markets," Computing in Economics and Finance 2001 59, Society for Computational Economics.
    27. Leonardo Bargigli & Luca Riccetti & Alberto Russo & Mauro Gallegati, 2020. "Network calibration and metamodeling of a financial accelerator agent based model," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 15(2), pages 413-440, April.
    28. Antoine Mandel & Amir Sani, 2016. "Learning Time-Varying Forecast Combinations," Documents de travail du Centre d'Economie de la Sorbonne 16036, Université Panthéon-Sorbonne (Paris 1), Centre d'Economie de la Sorbonne.
    29. Grazzini, Jakob & Richiardi, Matteo, 2015. "Estimation of ergodic agent-based models by simulated minimum distance," Journal of Economic Dynamics and Control, Elsevier, vol. 51(C), pages 148-165.
    30. De Grauwe, Paul & Macchiarelli, Corrado, 2015. "Animal spirits and credit cycles," Journal of Economic Dynamics and Control, Elsevier, vol. 59(C), pages 95-117.
    31. Lucas, Robert Jr, 1976. "Econometric policy evaluation: A critique," Carnegie-Rochester Conference Series on Public Policy, Elsevier, vol. 1(1), pages 19-46, January.
    32. Sylvain Barde, 2015. "A Practical, Universal, Information Criterion over Nth Order Markov Processes," Studies in Economics 1504, School of Economics, University of Kent.
    33. repec:hal:journl:halshs-01317974 is not listed on IDEAS
    34. Isabelle Salle & Murat Yıldızoğlu, 2014. "Efficient Sampling and Meta-Modeling for Computational Economic Models," Computational Economics, Springer;Society for Computational Economics, vol. 44(4), pages 507-536, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    2. Torsten Heinrich & Juan Sabuco & J. Doyne Farmer, 2019. "A simulation of the insurance industry: The problem of risk model homogeneity," Papers 1907.05954, arXiv.org, revised Nov 2019.
    3. Kukacka, Jiri & Jang, Tae-Seok & Sacht, Stephen, 2018. "On the estimation of behavioral macroeconomic models via simulated maximum likelihood," Economics Working Papers 2018-11, Christian-Albrechts-University of Kiel, Department of Economics.
    4. Delli Gatti, Domenico & Grazzini, Jakob, 2020. "Rising to the challenge: Bayesian estimation and forecasting techniques for macroeconomic Agent Based Models," Journal of Economic Behavior & Organization, Elsevier, vol. 178(C), pages 875-902.
    5. Severin Reissl, 2021. "Heterogeneous expectations, forecasting behaviour and policy experiments in a hybrid Agent-based Stock-flow-consistent model," Journal of Evolutionary Economics, Springer, vol. 31(1), pages 251-299, January.
    6. Sander Hoog, 2019. "Surrogate Modelling in (and of) Agent-Based Models: A Prospectus," Computational Economics, Springer;Society for Computational Economics, vol. 53(3), pages 1245-1263, March.
    7. Herbert Dawid & Philipp Harting & Sander Hoog & Michael Neugart, 2019. "Macroeconomics with heterogeneous agent models: fostering transparency, reproducibility and replication," Journal of Evolutionary Economics, Springer, vol. 29(1), pages 467-538, March.
    8. Platt, Donovan, 2020. "A comparison of economic agent-based model calibration methods," Journal of Economic Dynamics and Control, Elsevier, vol. 113(C).
    9. Sylvain Barde & Sander van der Hoog, 2017. "An empirical validation protocol for large-scale agent-based models," Working Papers hal-03458672, HAL.
    10. Leonardo Bargigli & Luca Riccetti & Alberto Russo & Mauro Gallegati, 2020. "Network calibration and metamodeling of a financial accelerator agent based model," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 15(2), pages 413-440, April.
    11. Giorgio Fagiolo & Mattia Guerini & Francesco Lamperti & Alessio Moneta & Andrea Roventini, 2017. "Validation of Agent-Based Models in Economics and Finance," LEM Papers Series 2017/23, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    12. Guus ten Broeke & George van Voorn & Arend Ligtenberg & Jaap Molenaar, 2021. "The Use of Surrogate Models to Analyse Agent-Based Models," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 24(2), pages 1-3.
    13. Alperen Bektas & Valentino Piana & René Schumann, 2021. "A meso-level empirical validation approach for agent-based computational economic models drawing on micro-data: a use case with a mobility mode-choice model," SN Business & Economics, Springer, vol. 1(6), pages 1-25, June.
    14. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    15. Herbert Dawid & Philipp Harting & Sander van der Hoog, 2019. "Manager remuneration, share buybacks, and firm performance," Industrial and Corporate Change, Oxford University Press, vol. 28(3), pages 681-706.
    16. Kukacka, Jiri & Sacht, Stephen, 2021. "Estimation of Heuristic Switching in Behavioral Macroeconomic Models," Economics Working Papers 2021-01, Christian-Albrechts-University of Kiel, Department of Economics.
    17. Donovan Platt, 2019. "A Comparison of Economic Agent-Based Model Calibration Methods," Papers 1902.05938, arXiv.org.
    18. Siyan Chen & Saul Desiderio, 2022. "A Regression-Based Calibration Method for Agent-Based Models," Computational Economics, Springer;Society for Computational Economics, vol. 59(2), pages 687-700, February.
    19. Severin Reissl, 0. "Heterogeneous expectations, forecasting behaviour and policy experiments in a hybrid Agent-based Stock-flow-consistent model," Journal of Evolutionary Economics, Springer, vol. 0, pages 1-49.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    2. Giorgio Fagiolo & Mattia Guerini & Francesco Lamperti & Alessio Moneta & Andrea Roventini, 2017. "Validation of Agent-Based Models in Economics and Finance," LEM Papers Series 2017/23, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    3. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    4. Guerini, Mattia & Moneta, Alessio, 2017. "A method for agent-based models validation," Journal of Economic Dynamics and Control, Elsevier, vol. 82(C), pages 125-141.
    5. Giovanni Dosi & Andrea Roventini, 2019. "More is different ... and complex! the case for agent-based macroeconomics," Journal of Evolutionary Economics, Springer, vol. 29(1), pages 1-37, March.
    6. Herbert Dawid & Philipp Harting & Sander Hoog & Michael Neugart, 2019. "Macroeconomics with heterogeneous agent models: fostering transparency, reproducibility and replication," Journal of Evolutionary Economics, Springer, vol. 29(1), pages 467-538, March.
    7. Giorgio Fagiolo & Andrea Roventini, 2017. "Macroeconomic Policy in DSGE and Agent-Based Models Redux: New Developments and Challenges Ahead," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 20(1), pages 1-1.
    8. Francesco Lamperti, 2018. "Empirical validation of simulated models through the GSL-div: an illustrative application," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 13(1), pages 143-171, April.
    9. Francesco Lamperti, 2016. "Empirical Validation of Simulated Models through the GSL-div: an Illustrative Application," LEM Papers Series 2016/18, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    10. Lamperti, Francesco, 2018. "An information theoretic criterion for empirical validation of simulation models," Econometrics and Statistics, Elsevier, vol. 5(C), pages 83-106.
    11. Barde, Sylvain, 2020. "Macroeconomic simulation comparison with a multivariate extension of the Markov information criterion," Journal of Economic Dynamics and Control, Elsevier, vol. 111(C).
    12. Lamperti, F. & Dosi, G. & Napoletano, M. & Roventini, A. & Sapio, A., 2018. "Faraway, So Close: Coupled Climate and Economic Dynamics in an Agent-based Integrated Assessment Model," Ecological Economics, Elsevier, vol. 150(C), pages 315-339.
    13. Donovan Platt, 2019. "A Comparison of Economic Agent-Based Model Calibration Methods," Papers 1902.05938, arXiv.org.
    14. Guerini, Mattia & Napoletano, Mauro & Roventini, Andrea, 2018. "No man is an Island: The impact of heterogeneity and local interactions on macroeconomic dynamics," Economic Modelling, Elsevier, vol. 68(C), pages 82-95.
    15. Francesco Lamperti, 2015. "An Information Theoretic Criterion for Empirical Validation of Time Series Models," LEM Papers Series 2015/02, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    16. Platt, Donovan, 2020. "A comparison of economic agent-based model calibration methods," Journal of Economic Dynamics and Control, Elsevier, vol. 113(C).
    17. Seri, Raffaello & Martinoli, Mario & Secchi, Davide & Centorrino, Samuele, 2021. "Model calibration and validation via confidence sets," Econometrics and Statistics, Elsevier, vol. 20(C), pages 62-86.
    18. Giovanni Dosi & Marcelo Pereira & Andrea Roventini & Maria Enrica Virgillito, 2016. "The Effects of Labour Market Reforms upon Unemployment and Income Inequalities: an Agent Based Model," Working Papers hal-03459264, HAL.
    19. Donovan Platt, 2022. "Bayesian Estimation of Economic Simulation Models Using Neural Networks," Computational Economics, Springer;Society for Computational Economics, vol. 59(2), pages 599-650, February.
    20. Dosi, G. & Pereira, M.C. & Roventini, A. & Virgillito, M.E., 2019. "What if supply-side policies are not enough? The perverse interaction of flexibility and austerity," Journal of Economic Behavior & Organization, Elsevier, vol. 162(C), pages 360-388.

    More about this item

    Keywords

    Statistical machine learning; Surrogate modelling; Empirical validation;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spo:wpmain:info:hdl:2441/4pa18fd9lf9h59m4vfavfcf61e. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/ecspofr.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Spire @ Sciences Po Library (email available below). General contact details of provider: https://edirc.repec.org/data/ecspofr.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.