IDEAS home Printed from https://ideas.repec.org/a/eee/ecosta/v5y2018icp83-106.html
   My bibliography  Save this article

An information theoretic criterion for empirical validation of simulation models

Author

Listed:
  • Lamperti, Francesco

Abstract

Simulated models suffer intrinsically from validation and comparison problems. The choice of a suitable indicator quantifying the distance between the model and the data is pivotal to model selection. An information theoretic criterion, called GSL-div, is introduced to measure how closely models’ synthetic output replicates the properties of observable time series without the need to resort to the likelihood function or to impose stationarity requirements. The indicator is sufficiently general to be applied to any model able to simulate or predict time series data, from simple univariate models to more complex objects including Agent-Based Models. When a set of models is given, a simple function of the L-divergence is used to select the candidate producing distributions of patterns that are closest to those observed in the data. The proposed approach is illustrated through three examples of increasing complexity where the GSL-div is used to discriminate among a variety of competing models. Results are compared to those obtained employing alternative measures of model’s fit. The GSL-div is found to perform, in the vast majority of cases, better than the alternatives.

Suggested Citation

  • Lamperti, Francesco, 2018. "An information theoretic criterion for empirical validation of simulation models," Econometrics and Statistics, Elsevier, vol. 5(C), pages 83-106.
  • Handle: RePEc:eee:ecosta:v:5:y:2018:i:c:p:83-106
    DOI: 10.1016/j.ecosta.2017.01.006
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S2452306217300084
    Download Restriction: Full text for ScienceDirect subscribers only. Contains open access articles

    File URL: https://libkey.io/10.1016/j.ecosta.2017.01.006?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. S. Alfarano & T. Lux & F. Wagner, 2007. "Empirical validation of stochastic models of interacting agents," The European Physical Journal B: Condensed Matter and Complex Systems, Springer;EDP Sciences, vol. 55(2), pages 183-187, January.
    2. Gilli, M. & Winker, P., 2003. "A global optimization heuristic for estimating agent based models," Computational Statistics & Data Analysis, Elsevier, vol. 42(3), pages 299-312, March.
    3. M. Hashem Pesaran & Andreas Pick, 2008. "Forecasting Random Walks Under Drift Instability," CESifo Working Paper Series 2293, CESifo.
    4. Jiri Kukacka & Jozef Barunik, 2016. "Simulated ML Estimation of Financial Agent-Based Models," Working Papers IES 2016/07, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Mar 2016.
    5. Canova, Fabio & Sala, Luca, 2009. "Back to square one: Identification issues in DSGE models," Journal of Monetary Economics, Elsevier, vol. 56(4), pages 431-449, May.
    6. G. Fagiolo & C. Birchenhall & P. Windrum, 2007. "Empirical Validation in Agent-based Models: Introduction to the Special Issue," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 189-194, October.
    7. Barde, Sylvain, 2016. "Direct comparison of agent-based models of herding in financial markets," Journal of Economic Dynamics and Control, Elsevier, vol. 73(C), pages 329-353.
    8. Sylvain Barde, 2017. "A Practical, Accurate, Information Criterion for Nth Order Markov Processes," Computational Economics, Springer;Society for Computational Economics, vol. 50(2), pages 281-324, August.
    9. Giorgio Fagiolo & Andrea Roventini, 2017. "Macroeconomic Policy in DSGE and Agent-Based Models Redux: New Developments and Challenges Ahead," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 20(1), pages 1-1.
    10. Recchioni, Maria Cristina & Tedeschi, Gabriele & Gallegati, Mauro, 2015. "A calibration procedure for analyzing stock price dynamics in an agent-based framework," Journal of Economic Dynamics and Control, Elsevier, vol. 60(C), pages 1-25.
    11. Annalisa Fabretti, 2013. "On the problem of calibrating an agent based model for financial markets," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 8(2), pages 277-293, October.
    12. A. Pyka & G. Fagiolo, 2007. "Agent-based Modelling: A Methodology for Neo-Schumpetarian Economics," Chapters, in: Horst Hanusch & Andreas Pyka (ed.), Elgar Companion to Neo-Schumpeterian Economics, chapter 29, Edward Elgar Publishing.
    13. Gualdi, Stanislao & Tarzia, Marco & Zamponi, Francesco & Bouchaud, Jean-Philippe, 2015. "Tipping points in macroeconomic agent-based models," Journal of Economic Dynamics and Control, Elsevier, vol. 50(C), pages 29-61.
    14. Paul Windrum & Giorgio Fagiolo & Alessio Moneta, 2007. "Empirical Validation of Agent-Based Models: Alternatives and Prospects," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 10(2), pages 1-8.
    15. Boswijk, H. Peter & Hommes, Cars H. & Manzan, Sebastiano, 2007. "Behavioral heterogeneity in stock prices," Journal of Economic Dynamics and Control, Elsevier, vol. 31(6), pages 1938-1970, June.
    16. Canova, Fabio, 1998. "Detrending and business cycle facts: A user's guide," Journal of Monetary Economics, Elsevier, vol. 41(3), pages 533-540, May.
    17. Matteo Richiardi & Roberto Leombruni & Nicole J. Saam & Michele Sonnessa, 2006. "A Common Protocol for Agent-Based Social Simulation," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 9(1), pages 1-15.
    18. Canova, Fabio, 1998. "Detrending and business cycle facts," Journal of Monetary Economics, Elsevier, vol. 41(3), pages 475-512, May.
    19. Giorgio Fagiolo & Andrea Roventini, 2012. "Macroeconomic Policy in DSGE and Agent-Based Models," Revue de l'OFCE, Presses de Sciences-Po, vol. 0(5), pages 67-116.
    20. Alfarano, Simone & Lux, Thomas & Wagner, Friedrich, 2006. "Estimation of a simple agent-based model of financial markets: An application to Australian stock and foreign exchange data," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 370(1), pages 38-42.
    21. Carlo Bianchi & Pasquale Cirillo & Mauro Gallegati & Pietro Vagliasindi, 2007. "Validating and Calibrating Agent-Based Models: A Case Study," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 245-264, October.
    22. Peter Winker & Manfred Gilli & Vahidin Jeleskovic, 2007. "An objective function for simulation based inference on exchange rate data," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 2(2), pages 125-145, December.
    23. Grazzini, Jakob & Richiardi, Matteo, 2015. "Estimation of ergodic agent-based models by simulated minimum distance," Journal of Economic Dynamics and Control, Elsevier, vol. 51(C), pages 148-165.
    24. Francesco Lamperti, 2016. "Empirical Validation of Simulated Models through the GSL-div: an Illustrative Application," LEM Papers Series 2016/18, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    25. Robert Marks, 2007. "Validating Simulation Models: A General Framework and Four Applied Examples," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 265-290, October.
    26. Grazzini, Jakob & Richiardi, Matteo G. & Tsionas, Mike, 2017. "Bayesian estimation of agent-based models," Journal of Economic Dynamics and Control, Elsevier, vol. 77(C), pages 26-47.
    27. Dosi, Giovanni & Fagiolo, Giorgio & Napoletano, Mauro & Roventini, Andrea & Treibich, Tania, 2015. "Fiscal and monetary policies in complex evolving economies," Journal of Economic Dynamics and Control, Elsevier, vol. 52(C), pages 166-189.
    28. Franke, Reiner & Westerhoff, Frank, 2012. "Structural stochastic volatility in asset pricing dynamics: Estimation and model contest," Journal of Economic Dynamics and Control, Elsevier, vol. 36(8), pages 1193-1211.
    29. Peter R. Hansen & Asger Lunde & James M. Nason, 2011. "The Model Confidence Set," Econometrica, Econometric Society, vol. 79(2), pages 453-497, March.
    30. Giannone, Domenico & Reichlin, Lucrezia & Sala, Luca, 2006. "VARs, common factors and the empirical validation of equilibrium business cycle models," Journal of Econometrics, Elsevier, vol. 132(1), pages 257-279, May.
    31. Guerini, Mattia & Moneta, Alessio, 2017. "A method for agent-based models validation," Journal of Economic Dynamics and Control, Elsevier, vol. 82(C), pages 125-141.
    32. Paulo, Rui & García-Donato, Gonzalo & Palomo, Jesús, 2012. "Calibration of computer models with multivariate output," Computational Statistics & Data Analysis, Elsevier, vol. 56(12), pages 3959-3974.
    33. Tesfatsion, Leigh & Judd, Kenneth L., 2006. "Handbook of Computational Economics, Vol. 2: Agent-Based Computational Economics," Staff General Research Papers Archive 10368, Iowa State University, Department of Economics.
    34. Dosi, Giovanni & Fagiolo, Giorgio & Napoletano, Mauro & Roventini, Andrea & Treibich, Tania, 2015. "Fiscal and monetary policies in complex evolving economies," Journal of Economic Dynamics and Control, Elsevier, vol. 52(C), pages 166-189.
    35. Amilon, Henrik, 2008. "Estimation of an adaptive stock market model with heterogeneous agents," Journal of Empirical Finance, Elsevier, vol. 15(2), pages 342-362, March.
    36. Assenza, Tiziana & Delli Gatti, Domenico & Grazzini, Jakob, 2015. "Emergent dynamics of a macroeconomic agent based model with capital and credit," Journal of Economic Dynamics and Control, Elsevier, vol. 50(C), pages 5-28.
    37. Caiani, Alessandro & Godin, Antoine & Caverzasi, Eugenio & Gallegati, Mauro & Kinsella, Stephen & Stiglitz, Joseph E., 2016. "Agent based-stock flow consistent macroeconomics: Towards a benchmark model," Journal of Economic Dynamics and Control, Elsevier, vol. 69(C), pages 375-408.
    38. Jakob Grazzini, 2012. "Analysis of the Emergent Properties: Stationarity and Ergodicity," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 15(2), pages 1-7.
    39. Reiner Franke & Frank Westerhoff, 2016. "Why a simple herding model may generate the stylized facts of daily returns: explanation and estimation," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 11(1), pages 1-34, April.
    40. Simone Alfarano & Thomas Lux & Friedrich Wagner, 2005. "Estimation of Agent-Based Models: The Case of an Asymmetric Herding Model," Computational Economics, Springer;Society for Computational Economics, vol. 26(1), pages 19-49, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Giovanni Dosi & Elisa Palagi & Andrea Roventini & Emanuele Russo, 2021. "Do patents really foster innovation in the pharmaceutical sector? Results from an evolutionary, agent-based model," LEM Papers Series 2021/37, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    2. Brouillat, Eric & Saint Jean, Maïder, 2020. "Mind the gap: Investigating the impact of implementation gaps on cleaner technology transition," Technological Forecasting and Social Change, Elsevier, vol. 158(C).
    3. Giovanni Dosi & Francesco Lamperti & Mariana Mazzucato & Mauro Napoletano & Andrea Roventini, 2021. "Mission-Oriented Policies and the “Entrepreneurial State” at Work: An Agent-Based Exploration," Sciences Po publications 14/2021, Sciences Po.
    4. Kukacka, Jiri & Sacht, Stephen, 2021. "Estimation of Heuristic Switching in Behavioral Macroeconomic Models," Economics Working Papers 2021-01, Christian-Albrechts-University of Kiel, Department of Economics.
    5. Nils Bertschinger & Iurii Mozzhorin, 2021. "Bayesian estimation and likelihood-based comparison of agent-based volatility models," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 16(1), pages 173-210, January.
    6. Lamperti, F. & Dosi, G. & Napoletano, M. & Roventini, A. & Sapio, A., 2018. "Faraway, So Close: Coupled Climate and Economic Dynamics in an Agent-based Integrated Assessment Model," Ecological Economics, Elsevier, vol. 150(C), pages 315-339.
    7. Lamperti, F. & Dosi, G. & Napoletano, M. & Roventini, A. & Sapio, A., 2020. "Climate change and green transitions in an agent-based integrated assessment model," Technological Forecasting and Social Change, Elsevier, vol. 153(C).
    8. Platt, Donovan, 2020. "A comparison of economic agent-based model calibration methods," Journal of Economic Dynamics and Control, Elsevier, vol. 113(C).
    9. Barde, Sylvain, 2020. "Macroeconomic simulation comparison with a multivariate extension of the Markov information criterion," Journal of Economic Dynamics and Control, Elsevier, vol. 111(C).
    10. Eric Brouillat & Maïder Saint-Jean, 2019. "Dura lex sed lex: why implementation gaps in environmental policy matter?," Cahiers du GREThA (2007-2019) 2019-04, Groupe de Recherche en Economie Théorique et Appliquée (GREThA).
    11. Andrea Vandin & Daniele Giachini & Francesco Lamperti & Francesca Chiaromonte, 2020. "Automated and Distributed Statistical Analysis of Economic Agent-Based Models," LEM Papers Series 2020/31, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    12. Andrea Vandin & Daniele Giachini & Francesco Lamperti & Francesca Chiaromonte, 2021. "Automated and Distributed Statistical Analysis of Economic Agent-Based Models," Papers 2102.05405, arXiv.org.
    13. Lamperti, Francesco & Bosetti, Valentina & Roventini, Andrea & Tavoni, Massimo & Treibich, Tania, 2021. "Three green financial policies to address climate risks," Journal of Financial Stability, Elsevier, vol. 54(C).
    14. Nils Bertschinger & Iurii Mozzhorin, 0. "Bayesian estimation and likelihood-based comparison of agent-based volatility models," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 0, pages 1-38.
    15. G. Rigatos, 2021. "Statistical Validation of Multi-Agent Financial Models Using the H-Infinity Kalman Filter," Computational Economics, Springer;Society for Computational Economics, vol. 58(3), pages 777-798, October.
    16. Giorgio Fagiolo & Mattia Guerini & Francesco Lamperti & Alessio Moneta & Andrea Roventini, 2017. "Validation of Agent-Based Models in Economics and Finance," LEM Papers Series 2017/23, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    17. Krichene, H. & Geiger, T. & Frieler, K. & Willner, S.N. & Sauer, I. & Otto, C., 2021. "Long-term impacts of tropical cyclones and fluvial floods on economic growth – Empirical evidence on transmission channels at different levels of development," World Development, Elsevier, vol. 144(C).
    18. Kukacka, Jiri & Jang, Tae-Seok & Sacht, Stephen, 2018. "On the estimation of behavioral macroeconomic models via simulated maximum likelihood," Economics Working Papers 2018-11, Christian-Albrechts-University of Kiel, Department of Economics.
    19. Mauro Napoletano & Eric Guerci & Nobuyuki Hanaki, 2018. "Recent advances in financial networks and agent-based model validation," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 13(1), pages 1-7, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    2. Giorgio Fagiolo & Mattia Guerini & Francesco Lamperti & Alessio Moneta & Andrea Roventini, 2017. "Validation of Agent-Based Models in Economics and Finance," LEM Papers Series 2017/23, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    3. Francesco Lamperti, 2018. "Empirical validation of simulated models through the GSL-div: an illustrative application," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 13(1), pages 143-171, April.
    4. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    5. Francesco Lamperti, 2016. "Empirical Validation of Simulated Models through the GSL-div: an Illustrative Application," LEM Papers Series 2016/18, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    6. Francesco Lamperti, 2015. "An Information Theoretic Criterion for Empirical Validation of Time Series Models," LEM Papers Series 2015/02, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    7. Donovan Platt, 2019. "A Comparison of Economic Agent-Based Model Calibration Methods," Papers 1902.05938, arXiv.org.
    8. Guerini, Mattia & Moneta, Alessio, 2017. "A method for agent-based models validation," Journal of Economic Dynamics and Control, Elsevier, vol. 82(C), pages 125-141.
    9. Platt, Donovan, 2020. "A comparison of economic agent-based model calibration methods," Journal of Economic Dynamics and Control, Elsevier, vol. 113(C).
    10. Giorgio Fagiolo & Andrea Roventini, 2017. "Macroeconomic Policy in DSGE and Agent-Based Models Redux: New Developments and Challenges Ahead," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 20(1), pages 1-1.
    11. Barde, Sylvain, 2020. "Macroeconomic simulation comparison with a multivariate extension of the Markov information criterion," Journal of Economic Dynamics and Control, Elsevier, vol. 111(C).
    12. Grazzini, Jakob & Richiardi, Matteo G. & Tsionas, Mike, 2017. "Bayesian estimation of agent-based models," Journal of Economic Dynamics and Control, Elsevier, vol. 77(C), pages 26-47.
    13. Donovan Platt & Tim Gebbie, 2016. "The Problem of Calibrating an Agent-Based Model of High-Frequency Trading," Papers 1606.01495, arXiv.org, revised Mar 2017.
    14. Giovanni Dosi & Andrea Roventini, 2019. "More is different ... and complex! the case for agent-based macroeconomics," Journal of Evolutionary Economics, Springer, vol. 29(1), pages 1-37, March.
    15. Lamperti, F. & Dosi, G. & Napoletano, M. & Roventini, A. & Sapio, A., 2018. "Faraway, So Close: Coupled Climate and Economic Dynamics in an Agent-based Integrated Assessment Model," Ecological Economics, Elsevier, vol. 150(C), pages 315-339.
    16. Grazzini, Jakob & Richiardi, Matteo, 2015. "Estimation of ergodic agent-based models by simulated minimum distance," Journal of Economic Dynamics and Control, Elsevier, vol. 51(C), pages 148-165.
    17. Grazzini, Jakob & Richiardi, Matteo, 2013. "Consistent Estimation of Agent-Based Models by Simulated Minimum Distance," Department of Economics and Statistics Cognetti de Martiis. Working Papers 201335, University of Turin.
    18. Kukacka, Jiri & Barunik, Jozef, 2017. "Estimation of financial agent-based models with simulated maximum likelihood," Journal of Economic Dynamics and Control, Elsevier, vol. 85(C), pages 21-45.
    19. Andrea Vandin & Daniele Giachini & Francesco Lamperti & Francesca Chiaromonte, 2021. "Automated and Distributed Statistical Analysis of Economic Agent-Based Models," Papers 2102.05405, arXiv.org.
    20. Andrea Vandin & Daniele Giachini & Francesco Lamperti & Francesca Chiaromonte, 2020. "Automated and Distributed Statistical Analysis of Economic Agent-Based Models," LEM Papers Series 2020/31, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ecosta:v:5:y:2018:i:c:p:83-106. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://www.journals.elsevier.com/econometrics-and-statistics .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/econometrics-and-statistics .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.