IDEAS home Printed from https://ideas.repec.org/a/eee/dyncon/v113y2020ics0165188920300294.html
   My bibliography  Save this article

A comparison of economic agent-based model calibration methods

Author

Listed:
  • Platt, Donovan

Abstract

Despite significant expansion in recent years, the literature on quantitative and data-driven approaches to economic agent-based model validation and calibration consists primarily of studies that have focused on the introduction of new calibration methods that are neither benchmarked against existing alternatives nor rigorously tested in terms of the quality of the estimates they produce. In response, we compare a number of prominent agent-based model calibration methods, both established and novel, through a series of computational experiments in an attempt to determine the respective strengths and weaknesses of each approach. Overall, we find that a simple, likelihood-based approach to Bayesian estimation consistently outperforms several members of the more popular class of simulated minimum distance methods and results in reasonable parameter estimates in many contexts, with a degradation in performance observed only when considering a large-scale model and attempting to fit a substantial number of its parameters.

Suggested Citation

  • Platt, Donovan, 2020. "A comparison of economic agent-based model calibration methods," Journal of Economic Dynamics and Control, Elsevier, vol. 113(C).
  • Handle: RePEc:eee:dyncon:v:113:y:2020:i:c:s0165188920300294
    DOI: 10.1016/j.jedc.2020.103859
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0165188920300294
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.jedc.2020.103859?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Barde, Sylvain, 2016. "Direct comparison of agent-based models of herding in financial markets," Journal of Economic Dynamics and Control, Elsevier, vol. 73(C), pages 329-353.
    2. S. Alfarano & T. Lux & F. Wagner, 2007. "Empirical validation of stochastic models of interacting agents," The European Physical Journal B: Condensed Matter and Complex Systems, Springer;EDP Sciences, vol. 55(2), pages 183-187, January.
    3. repec:hal:spmain:info:hdl:2441/4pa18fd9lf9h59m4vfavfcf61e is not listed on IDEAS
    4. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    5. Sandrine Jacob Leal & Mauro Napoletano & Andrea Roventini & Giorgio Fagiolo, 2016. "Rock around the clock: An agent-based model of low- and high-frequency trading," Journal of Evolutionary Economics, Springer, vol. 26(1), pages 49-76, March.
    6. Recchioni, Maria Cristina & Tedeschi, Gabriele & Gallegati, Mauro, 2015. "A calibration procedure for analyzing stock price dynamics in an agent-based framework," Journal of Economic Dynamics and Control, Elsevier, vol. 60(C), pages 1-25.
    7. Sylvain Barde & Sander van Der Hoog, 2017. "An empirical validation protocol for large-scale agent-based models," Working Papers hal-03458672, HAL.
    8. Giorgio Fagiolo & Andrea Roventini, 2017. "Macroeconomic Policy in DSGE and Agent-Based Models Redux: New Developments and Challenges Ahead," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 20(1), pages 1-1.
    9. Annalisa Fabretti, 2013. "On the problem of calibrating an agent based model for financial markets," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 8(2), pages 277-293, October.
    10. repec:hal:spmain:info:hdl:2441/7kr9gv74ut9ngo58gia97t83i7 is not listed on IDEAS
    11. Lars Peter Hansen & James J. Heckman, 1996. "The Empirical Foundations of Calibration," Journal of Economic Perspectives, American Economic Association, vol. 10(1), pages 87-104, Winter.
    12. John Geanakoplos & Robert Axtell & J. Doyne Farmer & Peter Howitt & Benjamin Conlee & Jonathan Goldstein & Matthew Hendrey & Nathan M. Palmer & Chun-Yi Yang, 2012. "Getting at Systemic Risk via an Agent-Based Model of the Housing Market," American Economic Review, American Economic Association, vol. 102(3), pages 53-58, May.
    13. McFadden, Daniel, 1989. "A Method of Simulated Moments for Estimation of Discrete Response Models without Numerical Integration," Econometrica, Econometric Society, vol. 57(5), pages 995-1026, September.
    14. Giorgio Fagiolo & Mattia Guerini & Francesco Lamperti & Alessio Moneta & Andrea Roventini, 2017. "Validation of Agent-Based Models in Economics and Finance," LEM Papers Series 2017/23, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    15. Grazzini, Jakob & Richiardi, Matteo G. & Tsionas, Mike, 2017. "Bayesian estimation of agent-based models," Journal of Economic Dynamics and Control, Elsevier, vol. 77(C), pages 26-47.
    16. Isabelle Salle & Murat Yıldızoğlu, 2014. "Efficient Sampling and Meta-Modeling for Computational Economic Models," Computational Economics, Springer;Society for Computational Economics, vol. 44(4), pages 507-536, December.
    17. Sylvain Barde, 2017. "A Practical, Accurate, Information Criterion for Nth Order Markov Processes," Computational Economics, Springer;Society for Computational Economics, vol. 50(2), pages 281-324, August.
    18. Kukacka, Jiri & Barunik, Jozef, 2017. "Estimation of financial agent-based models with simulated maximum likelihood," Journal of Economic Dynamics and Control, Elsevier, vol. 85(C), pages 21-45.
    19. Rommel Regis & Christine Shoemaker, 2005. "Constrained Global Optimization of Expensive Black Box Functions Using Radial Basis Functions," Journal of Global Optimization, Springer, vol. 31(1), pages 153-171, January.
    20. J. Doyne Farmer & John Geanakoplos, 2008. "The Virtues and Vices of Equilibrium and the Future of Financial Economics," Levine's Working Paper Archive 122247000000002067, David K. Levine.
    21. repec:hal:spmain:info:hdl:2441/13thfd12aa8rmplfudlgvgahff is not listed on IDEAS
    22. J. Doyne Farmer & Duncan Foley, 2009. "The economy needs agent-based modelling," Nature, Nature, vol. 460(7256), pages 685-686, August.
    23. LeBaron, Blake, 2006. "Agent-based Computational Finance," Handbook of Computational Economics, in: Leigh Tesfatsion & Kenneth L. Judd (ed.), Handbook of Computational Economics, edition 1, volume 2, chapter 24, pages 1187-1233, Elsevier.
    24. Lux, Thomas, 2018. "Estimation of agent-based models using sequential Monte Carlo methods," Journal of Economic Dynamics and Control, Elsevier, vol. 91(C), pages 391-408.
    25. Franke, Reiner, 2009. "Applying the method of simulated moments to estimate a small agent-based asset pricing model," Journal of Empirical Finance, Elsevier, vol. 16(5), pages 804-815, December.
    26. repec:hal:spmain:info:hdl:2441/5fafm6me7k8omq5jbo61urqq27 is not listed on IDEAS
    27. Guerini, Mattia & Moneta, Alessio, 2017. "A method for agent-based models validation," Journal of Economic Dynamics and Control, Elsevier, vol. 82(C), pages 125-141.
    28. Simone Alfarano & Thomas Lux & Friedrich Wagner, 2005. "Estimation of Agent-Based Models: The Case of an Asymmetric Herding Model," Computational Economics, Springer;Society for Computational Economics, vol. 26(1), pages 19-49, August.
    29. Farmer, J. Doyne & Joshi, Shareen, 2002. "The price dynamics of common trading strategies," Journal of Economic Behavior & Organization, Elsevier, vol. 49(2), pages 149-171, October.
    30. Brock, William A. & Hommes, Cars H., 1998. "Heterogeneous beliefs and routes to chaos in a simple asset pricing model," Journal of Economic Dynamics and Control, Elsevier, vol. 22(8-9), pages 1235-1274, August.
    31. Cincotti, Silvano & Raberto, Marco & Teglio, Andrea, 2010. "Credit money and macroeconomic instability in the agent-based model and simulator Eurace," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 4, pages 1-32.
    32. Gilli, M. & Winker, P., 2003. "A global optimization heuristic for estimating agent based models," Computational Statistics & Data Analysis, Elsevier, vol. 42(3), pages 299-312, March.
    33. Platt, Donovan & Gebbie, Tim, 2018. "Can agent-based models probe market microstructure?," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 503(C), pages 1092-1106.
    34. Alfarano, Simone & Lux, Thomas & Wagner, Friedrich, 2006. "Estimation of a simple agent-based model of financial markets: An application to Australian stock and foreign exchange data," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 370(1), pages 38-42.
    35. Carlo Bianchi & Pasquale Cirillo & Mauro Gallegati & Pietro Vagliasindi, 2007. "Validating and Calibrating Agent-Based Models: A Case Study," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 245-264, October.
    36. Grazzini, Jakob & Richiardi, Matteo, 2015. "Estimation of ergodic agent-based models by simulated minimum distance," Journal of Economic Dynamics and Control, Elsevier, vol. 51(C), pages 148-165.
    37. Francesco Lamperti, 2018. "Empirical validation of simulated models through the GSL-div: an illustrative application," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 13(1), pages 143-171, April.
    38. Franke, Reiner & Westerhoff, Frank, 2012. "Structural stochastic volatility in asset pricing dynamics: Estimation and model contest," Journal of Economic Dynamics and Control, Elsevier, vol. 36(8), pages 1193-1211.
    39. Baptista, Rafa & Farmer, J. Doyne & Hinterschweiger, Marc & Low, Katie & Tang, Daniel & Uluc, Arzu, 2016. "Macroprudential policy in an agent-based model of the UK housing market," Bank of England working papers 619, Bank of England.
    40. Jakob Grazzini, 2012. "Analysis of the Emergent Properties: Stationarity and Ergodicity," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 15(2), pages 1-7.
    41. Lamperti, Francesco, 2018. "An information theoretic criterion for empirical validation of simulation models," Econometrics and Statistics, Elsevier, vol. 5(C), pages 83-106.
    42. Isabelle Salle & Murat Yıldızoğlu, 2014. "Efficient Sampling and Meta-Modeling for Computational Economic Models," Computational Economics, Springer;Society for Computational Economics, vol. 44(4), pages 507-536, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Chao, Xiangrui & Ran, Qin & Chen, Jia & Li, Tie & Qian, Qian & Ergu, Daji, 2022. "Regulatory technology (Reg-Tech) in financial stability supervision: Taxonomy, key methods, applications and future directions," International Review of Financial Analysis, Elsevier, vol. 80(C).
    2. Pichler, Anton & Pangallo, Marco & del Rio-Chanona, R. Maria & Lafond, François & Farmer, J. Doyne, 2022. "Forecasting the propagation of pandemic shocks with a dynamic input-output model," Journal of Economic Dynamics and Control, Elsevier, vol. 144(C).
    3. Ivan Jericevich & Patrick Chang & Tim Gebbie, 2021. "Simulation and estimation of an agent-based market-model with a matching engine," Papers 2108.07806, arXiv.org, revised Aug 2021.
    4. Kukacka, Jiri & Kristoufek, Ladislav, 2021. "Does parameterization affect the complexity of agent-based models?," Journal of Economic Behavior & Organization, Elsevier, vol. 192(C), pages 324-356.
    5. Michael Heinrich Baumann & Michaela Baumann & Lars Grüne & Bernhard Herz, 2023. "Market Structure and Instability Artifacts in Heterogeneous Agent Models: Lessons from Implicit Discretizations of Stiff Equations," Computational Economics, Springer;Society for Computational Economics, vol. 62(3), pages 855-890, October.
    6. Ivan Jericevich & Murray McKechnie & Tim Gebbie, 2021. "Calibrating an adaptive Farmer-Joshi agent-based model for financial markets," Papers 2104.09863, arXiv.org.
    7. Farmer, J. Doyne & Dyer, Joel & Cannon, Patrick & Schmon, Sebastian, 2022. "Black-box Bayesian inference for economic agent-based models," INET Oxford Working Papers 2022-05, Institute for New Economic Thinking at the Oxford Martin School, University of Oxford.
    8. Schmitt, Noemi & Westerhoff, Frank, 2021. "Trend followers, contrarians and fundamentalists: Explaining the dynamics of financial markets," Journal of Economic Behavior & Organization, Elsevier, vol. 192(C), pages 117-136.
    9. Siyan Chen & Saul Desiderio, 2022. "Calibration of Agent-Based Models by Means of Meta-Modeling and Nonparametric Regression," Computational Economics, Springer;Society for Computational Economics, vol. 60(4), pages 1457-1478, December.
    10. Zila, Eric & Kukacka, Jiri, 2023. "Moment set selection for the SMM using simple machine learning," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 366-391.
    11. Papadopoulos, Georgios, 2020. "Probing the mechanism: lending rate setting in a data-driven agent-based model," MPRA Paper 102749, University Library of Munich, Germany.
    12. Donovan Platt, 2022. "Bayesian Estimation of Economic Simulation Models Using Neural Networks," Computational Economics, Springer;Society for Computational Economics, vol. 59(2), pages 599-650, February.
    13. Ghaderi, Mohammad, 2022. "Public health interventions in the face of pandemics: Network structure, social distancing, and heterogeneity," European Journal of Operational Research, Elsevier, vol. 298(3), pages 1016-1031.
    14. Emanuele Ciola & Edoardo Gaffeo & Mauro Gallegati, 2021. "Search for Profits and Business Fluctuations: How Banks' Behaviour Explain Cycles?," Working Papers 450, Universita' Politecnica delle Marche (I), Dipartimento di Scienze Economiche e Sociali.
    15. Noemi Schmitt & Ivonne Schwartz & Frank Westerhoff, 2022. "Heterogeneous speculators and stock market dynamics: a simple agent-based computational model," The European Journal of Finance, Taylor & Francis Journals, vol. 28(13-15), pages 1263-1282, October.
    16. Zhang, Jinyu & Zhang, Qiaosen & Li, Yong & Wang, Qianchao, 2023. "Sequential Bayesian inference for agent-based models with application to the Chinese business cycle," Economic Modelling, Elsevier, vol. 126(C).
    17. Shiono, Takashi, 2021. "Estimation of agent-based models using Bayesian deep learning approach of BayesFlow," Journal of Economic Dynamics and Control, Elsevier, vol. 125(C).
    18. Silvano Cincotti & Marco Raberto & Andrea Teglio, 2022. "Why do we need agent-based macroeconomics?," Review of Evolutionary Political Economy, Springer, vol. 3(1), pages 5-29, April.
    19. Anna Varga-Csajkás & Tamás Sebestyén & Attila Varga, 2023. "Dynamics of collaboration among high-growth firms: results from an agent-based policy simulation," The Annals of Regional Science, Springer;Western Regional Science Association, vol. 70(2), pages 353-377, April.
    20. Andrea Coletta & Joseph Jerome & Rahul Savani & Svitlana Vyetrenko, 2023. "Conditional Generators for Limit Order Book Environments: Explainability, Challenges, and Robustness," Papers 2306.12806, arXiv.org.
    21. Ivan Jericevich & Dharmesh Sing & Tim Gebbie, 2021. "CoinTossX: An open-source low-latency high-throughput matching engine," Papers 2102.10925, arXiv.org.
    22. Elizabeth Jane Casabianca & Alessia Lo Turco & Daniela Maggioni, 2021. "Migration And The Structure Of Manufacturing Production. A View From Italian Provinces," Working Papers 448, Universita' Politecnica delle Marche (I), Dipartimento di Scienze Economiche e Sociali.
    23. Tubbenhauer, Tobias & Fieberg, Christian & Poddig, Thorsten, 2021. "Multi-agent-based VaR forecasting," Journal of Economic Dynamics and Control, Elsevier, vol. 131(C).
    24. Ciola, Emanuele & Gaffeo, Edoardo & Gallegati, Mauro, 2022. "Search for profits and business fluctuations: How does banks’ behaviour explain cycles?," Journal of Economic Dynamics and Control, Elsevier, vol. 135(C).
    25. Richiardi, Matteo & Bronka, Patryk & van de Ven, Justin, 2023. "Back to the future: Agent-based modelling and dynamic microsimulation," Centre for Microsimulation and Policy Analysis Working Paper Series CEMPA8/23, Centre for Microsimulation and Policy Analysis at the Institute for Social and Economic Research.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Donovan Platt, 2019. "A Comparison of Economic Agent-Based Model Calibration Methods," Papers 1902.05938, arXiv.org.
    2. Donovan Platt, 2022. "Bayesian Estimation of Economic Simulation Models Using Neural Networks," Computational Economics, Springer;Society for Computational Economics, vol. 59(2), pages 599-650, February.
    3. Giorgio Fagiolo & Mattia Guerini & Francesco Lamperti & Alessio Moneta & Andrea Roventini, 2017. "Validation of Agent-Based Models in Economics and Finance," LEM Papers Series 2017/23, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    4. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    5. repec:hal:spmain:info:hdl:2441/20hflp7eqn97boh50no50tv67n is not listed on IDEAS
    6. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    7. repec:hal:spmain:info:hdl:2441/13thfd12aa8rmplfudlgvgahff is not listed on IDEAS
    8. Zila, Eric & Kukacka, Jiri, 2023. "Moment set selection for the SMM using simple machine learning," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 366-391.
    9. Barde, Sylvain, 2020. "Macroeconomic simulation comparison with a multivariate extension of the Markov information criterion," Journal of Economic Dynamics and Control, Elsevier, vol. 111(C).
    10. Tubbenhauer, Tobias & Fieberg, Christian & Poddig, Thorsten, 2021. "Multi-agent-based VaR forecasting," Journal of Economic Dynamics and Control, Elsevier, vol. 131(C).
    11. Francesco Lamperti, 2018. "Empirical validation of simulated models through the GSL-div: an illustrative application," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 13(1), pages 143-171, April.
    12. Siyan Chen & Saul Desiderio, 2022. "Calibration of Agent-Based Models by Means of Meta-Modeling and Nonparametric Regression," Computational Economics, Springer;Society for Computational Economics, vol. 60(4), pages 1457-1478, December.
    13. Lamperti, Francesco, 2018. "An information theoretic criterion for empirical validation of simulation models," Econometrics and Statistics, Elsevier, vol. 5(C), pages 83-106.
    14. Grazzini, Jakob & Richiardi, Matteo G. & Tsionas, Mike, 2017. "Bayesian estimation of agent-based models," Journal of Economic Dynamics and Control, Elsevier, vol. 77(C), pages 26-47.
    15. Platt, Donovan & Gebbie, Tim, 2018. "Can agent-based models probe market microstructure?," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 503(C), pages 1092-1106.
    16. Donovan Platt & Tim Gebbie, 2016. "The Problem of Calibrating an Agent-Based Model of High-Frequency Trading," Papers 1606.01495, arXiv.org, revised Mar 2017.
    17. Seri, Raffaello & Martinoli, Mario & Secchi, Davide & Centorrino, Samuele, 2021. "Model calibration and validation via confidence sets," Econometrics and Statistics, Elsevier, vol. 20(C), pages 62-86.
    18. Mario Martinoli & Alessio Moneta & Gianluca Pallante, 2022. "Calibration and Validation of Macroeconomic Simulation Models by Statistical Causal Search," LEM Papers Series 2022/33, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    19. Guerini, Mattia & Moneta, Alessio, 2017. "A method for agent-based models validation," Journal of Economic Dynamics and Control, Elsevier, vol. 82(C), pages 125-141.
    20. Kukacka, Jiri & Kristoufek, Ladislav, 2021. "Does parameterization affect the complexity of agent-based models?," Journal of Economic Behavior & Organization, Elsevier, vol. 192(C), pages 324-356.
    21. Kukacka, Jiri & Sacht, Stephen, 2023. "Estimation of heuristic switching in behavioral macroeconomic models," Journal of Economic Dynamics and Control, Elsevier, vol. 146(C).
    22. Delli Gatti, Domenico & Grazzini, Jakob, 2020. "Rising to the challenge: Bayesian estimation and forecasting techniques for macroeconomic Agent Based Models," Journal of Economic Behavior & Organization, Elsevier, vol. 178(C), pages 875-902.

    More about this item

    Keywords

    Agent-based modelling; Calibration; Simulated minimum distance; Bayesian estimation;
    All these keywords.

    JEL classification:

    • C13 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Estimation: General
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:dyncon:v:113:y:2020:i:c:s0165188920300294. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jedc .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.