IDEAS home Printed from https://ideas.repec.org/p/ukc/ukcedp/1712.html
   My bibliography  Save this paper

An empirical validation protocol for large-scale agent-based models

Author

Listed:
  • Sylvain Barde

    ()

  • Sander van der Hoog

    ()

Abstract

Despite recent advances in bringing agent-based models (ABMs) to the data, the estimation or calibration of model parameters remains a challenge, especially when it comes to large-scale agent-based macroeconomic models. Most methods, such as the method of simulated moments (MSM), require in-the-loop simulation of new data, which may not be feasible for such computationally heavy simulation models. The purpose of this paper is to provide a proof-of-concept of a generic empirical validation methodology for such large-scale simulation models. We introduce an alternative 'large-scale' empirical validation approach, and apply it to the Eurace@Unibi macroeconomic simulation model (Dawid et al., 2016). This model was selected because it displays strong emergent behaviour and is able to generate a wide variety of nonlinear economic dynamics, including endogenous business- and financial cycles. In addition, it is a computationally heavy simulation model, so it ts our targeted use-case. The validation protocol consists of three stages. At the first stage we use Nearly-Orthogonal Latin Hypercube sampling (NOLH) in order to generate a set of 513 parameter combinations with good space-filling properties. At the second stage we use the recently developed Markov Information Criterion (MIC) to score the simulated data against empirical data. Finally, at the third stage we use stochastic kriging to construct a surrogate model of the MIC response surface, resulting in an interpolation of the response surface as a function of the parameters. The parameter combinations providing the best fit to the data are then identified as the local minima of the interpolated MIC response surface. The Model Confidence Set (MCS) procedure of Hansen et al. (2011) is used to restrict the set of model calibrations to those models that cannot be rejected to have equal predictive ability, at a given confidence level. Validation of the surrogate model is carried out by re-running the second stage of the analysis on the so identified optima and cross-checking that the realised MIC scores equal the MIC scores predicted by the surrogate model. The results we obtain so far look promising as a first proof-of-concept for the empirical validation methodology since we are able to validate the model using empirical data series for 30 OECD countries and the euro area. The internal validation procedure of the surrogate model also suggests that the combination of NOLH sampling, MIC measurement and stochastic kriging yields reliable predictions of the MIC scores for samples not included in the original NOLH sample set. In our opinion, this is a strong indication that the method we propose could provide a viable statistical machine learning technique for the empirical validation of (large-scale) ABMs.

Suggested Citation

  • Sylvain Barde & Sander van der Hoog, 2017. "An empirical validation protocol for large-scale agent-based models," Studies in Economics 1712, School of Economics, University of Kent.
  • Handle: RePEc:ukc:ukcedp:1712
    as

    Download full text from publisher

    File URL: https://www.kent.ac.uk/economics/repec/1712.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. G. Fagiolo & C. Birchenhall & P. Windrum, 2007. "Empirical Validation in Agent-based Models: Introduction to the Special Issue," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 189-194, October.
    2. Jasmina Arifovic & James Bullard & Olena Kostyshyna, 2013. "Social Learning and Monetary Policy Rules," Economic Journal, Royal Economic Society, vol. 123(567), pages 38-76, March.
    3. Barde, Sylvain, 2016. "Direct comparison of agent-based models of herding in financial markets," Journal of Economic Dynamics and Control, Elsevier, vol. 73(C), pages 329-353.
    4. Sylvain Barde & Sander Van Der Hoog, 2017. "An empirical validation protocol for large-scale agent-based models," Sciences Po publications 17/12, Sciences Po.
    5. Assenza, Tiziana & Delli Gatti, Domenico, 2013. "E Pluribus Unum: Macroeconomic modelling for multi-agent economies," Journal of Economic Dynamics and Control, Elsevier, vol. 37(8), pages 1659-1682.
    6. Paul Windrum & Giorgio Fagiolo & Alessio Moneta, 2007. "Empirical Validation of Agent-Based Models: Alternatives and Prospects," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 10(2), pages 1-8.
    7. repec:spr:jeicoo:v:13:y:2018:i:1:d:10.1007_s11403-017-0193-4 is not listed on IDEAS
    8. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    9. McFadden, Daniel, 1980. "Econometric Models for Probabilistic Choice among Products," The Journal of Business, University of Chicago Press, vol. 53(3), pages 13-29, July.
    10. Peter R. Hansen & Asger Lunde & James M. Nason, 2011. "The Model Confidence Set," Econometrica, Econometric Society, vol. 79(2), pages 453-497, March.
    11. Dosi, Giovanni & Fagiolo, Giorgio & Roventini, Andrea, 2010. "Schumpeter meeting Keynes: A policy-friendly model of endogenous growth and business cycles," Journal of Economic Dynamics and Control, Elsevier, vol. 34(9), pages 1748-1767, September.
    12. Isabelle Salle & Murat Yıldızoğlu, 2014. "Efficient Sampling and Meta-Modeling for Computational Economic Models," Computational Economics, Springer;Society for Computational Economics, vol. 44(4), pages 507-536, December.
    13. Dosi, Giovanni & Fagiolo, Giorgio & Napoletano, Mauro & Roventini, Andrea & Treibich, Tania, 2015. "Fiscal and monetary policies in complex evolving economies," Journal of Economic Dynamics and Control, Elsevier, vol. 52(C), pages 166-189.
    14. Kleijnen, Jack P.C., 2017. "Regression and Kriging metamodels with their experimental designs in simulation: A review," European Journal of Operational Research, Elsevier, vol. 256(1), pages 1-16.
    15. Antoine Mandel & Carlo Jaeger & Steffen Fürst & Wiebke Lass & Daniel Lincke & Frank Meissner & Federico Pablo-Marti & Sarah Wolf, 2010. "Agent-based dynamics in disaggregated growth models," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-00542442, HAL.
    16. Guerini, Mattia & Moneta, Alessio, 2017. "A method for agent-based models validation," Journal of Economic Dynamics and Control, Elsevier, vol. 82(C), pages 125-141.
    17. Wieland, Volker & Cwik, Tobias & Müller, Gernot J. & Schmidt, Sebastian & Wolters, Maik, 2012. "A new comparative approach to macroeconomic modeling and policy analysis," Journal of Economic Behavior & Organization, Elsevier, vol. 83(3), pages 523-541.
    18. Sander van der Hoog, 2017. "Deep Learning in (and of) Agent-Based Models: A Prospectus," Papers 1706.06302, arXiv.org.
    19. Gilli, M. & Winker, P., 2003. "A global optimization heuristic for estimating agent based models," Computational Statistics & Data Analysis, Elsevier, vol. 42(3), pages 299-312, March.
    20. Luca Riccetti & Alberto Russo & Mauro Gallegati, 2015. "An agent based decentralized matching macroeconomic model," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 10(2), pages 305-332, October.
    21. Sylvain Barde, 2017. "A Practical, Accurate, Information Criterion for Nth Order Markov Processes," Computational Economics, Springer;Society for Computational Economics, vol. 50(2), pages 281-324, August.
    22. Leonardo Bargigli & Luca Riccetti & Alberto Russo & Mauro Gallegati, 2016. "Network Calibration and Metamodeling of a Financial Accelerator Agent Based Model," Working Papers - Economics wp2016_01.rdf, Universita' degli Studi di Firenze, Dipartimento di Scienze per l'Economia e l'Impresa.
    23. Antoine Mandel & Amir Sani, 2016. "Learning Time-Varying Forecast Combinations," Documents de travail du Centre d'Economie de la Sorbonne 16036, Université Panthéon-Sorbonne (Paris 1), Centre d'Economie de la Sorbonne.
    24. Grazzini, Jakob & Richiardi, Matteo, 2015. "Estimation of ergodic agent-based models by simulated minimum distance," Journal of Economic Dynamics and Control, Elsevier, vol. 51(C), pages 148-165.
    25. G. Dosi & M. C. Pereira & M. E. Virgillito, 2018. "On the robustness of the fat-tailed distribution of firm growth rates: a global sensitivity analysis," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 13(1), pages 173-193, April.
    26. De Grauwe, Paul & Macchiarelli, Corrado, 2015. "Animal spirits and credit cycles," Journal of Economic Dynamics and Control, Elsevier, vol. 59(C), pages 95-117.
    27. Sylvain Barde, 2015. "A Practical, Universal, Information Criterion over Nth Order Markov Processes," Studies in Economics 1504, School of Economics, University of Kent.
    28. Antoine Mandel & Carlo Jaeger & Steffen Fürst & Wiebke Lass & Daniel Lincke & Frank Meissner & Federico Pablo-Marti & Sarah Wolf, 2010. "Agent-based dynamics in disaggregated growth models," Post-Print halshs-00542442, HAL.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    2. Sylvain Barde & Sander van der Hoog, 2017. "An empirical validation protocol for large-scale agent-based models," Studies in Economics 1712, School of Economics, University of Kent.
    3. Kukacka, Jiri & Jang, Tae-Seok & Sacht, Stephen, 2018. "On the estimation of behavioral macroeconomic models via simulated maximum likelihood," Economics Working Papers 2018-11, Christian-Albrechts-University of Kiel, Department of Economics.
    4. Torsten Heinrich & Juan Sabuco & J. Doyne Farmer, 2019. "A simulation of the insurance industry: The problem of risk model homogeneity," Papers 1907.05954, arXiv.org.
    5. Heinrich, Torsten & Sabuco, Juan & Farmer, J. Doyne, 2019. "A simulation of the insurance industry: The problem of risk model homogeneity," MPRA Paper 95096, University Library of Munich, Germany.
    6. repec:spr:joevec:v:29:y:2019:i:1:d:10.1007_s00191-018-0594-0 is not listed on IDEAS
    7. Giorgio Fagiolo & Mattia Guerini & Francesco Lamperti & Alessio Moneta & Andrea Roventini, 2017. "Validation of Agent-Based Models in Economics and Finance," LEM Papers Series 2017/23, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.

    More about this item

    Keywords

    Statistical machine learning; surrogate modelling; empirical validation;

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ukc:ukcedp:1712. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Tracey Girling). General contact details of provider: https://www.kent.ac.uk/economics/ .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.