IDEAS home Printed from https://ideas.repec.org/a/eee/ecosta/v5y2018icp83-106.html
   My bibliography  Save this article

An information theoretic criterion for empirical validation of simulation models

Author

Listed:
  • Lamperti, Francesco

Abstract

Simulated models suffer intrinsically from validation and comparison problems. The choice of a suitable indicator quantifying the distance between the model and the data is pivotal to model selection. An information theoretic criterion, called GSL-div, is introduced to measure how closely models’ synthetic output replicates the properties of observable time series without the need to resort to the likelihood function or to impose stationarity requirements. The indicator is sufficiently general to be applied to any model able to simulate or predict time series data, from simple univariate models to more complex objects including Agent-Based Models. When a set of models is given, a simple function of the L-divergence is used to select the candidate producing distributions of patterns that are closest to those observed in the data. The proposed approach is illustrated through three examples of increasing complexity where the GSL-div is used to discriminate among a variety of competing models. Results are compared to those obtained employing alternative measures of model’s fit. The GSL-div is found to perform, in the vast majority of cases, better than the alternatives.

Suggested Citation

  • Lamperti, Francesco, 2018. "An information theoretic criterion for empirical validation of simulation models," Econometrics and Statistics, Elsevier, vol. 5(C), pages 83-106.
  • Handle: RePEc:eee:ecosta:v:5:y:2018:i:c:p:83-106
    DOI: 10.1016/j.ecosta.2017.01.006
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S2452306217300084
    Download Restriction: Full text for ScienceDirect subscribers only. Contains open access articles

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. S. Alfarano & T. Lux & F. Wagner, 2007. "Empirical validation of stochastic models of interacting agents," The European Physical Journal B: Condensed Matter and Complex Systems, Springer;EDP Sciences, vol. 55(2), pages 183-187, January.
    2. Gilli, M. & Winker, P., 2003. "A global optimization heuristic for estimating agent based models," Computational Statistics & Data Analysis, Elsevier, vol. 42(3), pages 299-312, March.
    3. M. Hashem Pesaran & Andreas Pick, 2008. "Forecasting Random Walks Under Drift Instability," CESifo Working Paper Series 2293, CESifo Group Munich.
    4. Jiri Kukacka & Jozef Barunik, 2016. "Simulated ML Estimation of Financial Agent-Based Models," Working Papers IES 2016/07, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Mar 2016.
    5. Canova, Fabio & Sala, Luca, 2009. "Back to square one: Identification issues in DSGE models," Journal of Monetary Economics, Elsevier, vol. 56(4), pages 431-449, May.
    6. G. Fagiolo & C. Birchenhall & P. Windrum, 2007. "Empirical Validation in Agent-based Models: Introduction to the Special Issue," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 189-194, October.
    7. Barde, Sylvain, 2016. "Direct comparison of agent-based models of herding in financial markets," Journal of Economic Dynamics and Control, Elsevier, vol. 73(C), pages 329-353.
    8. Giorgio Fagiolo & Andrea Roventini, 2017. "Macroeconomic Policy in DSGE and Agent-Based Models Redux: New Developments and Challenges Ahead," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 20(1), pages 1-1.
    9. Recchioni, Maria Cristina & Tedeschi, Gabriele & Gallegati, Mauro, 2015. "A calibration procedure for analyzing stock price dynamics in an agent-based framework," Journal of Economic Dynamics and Control, Elsevier, vol. 60(C), pages 1-25.
    10. Annalisa Fabretti, 2013. "On the problem of calibrating an agent based model for financial markets," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 8(2), pages 277-293, October.
    11. A. Pyka & G. Fagiolo, 2007. "Agent-based Modelling: A Methodology for Neo-Schumpetarian Economics," Chapters,in: Elgar Companion to Neo-Schumpeterian Economics, chapter 29 Edward Elgar Publishing.
    12. Gualdi, Stanislao & Tarzia, Marco & Zamponi, Francesco & Bouchaud, Jean-Philippe, 2015. "Tipping points in macroeconomic agent-based models," Journal of Economic Dynamics and Control, Elsevier, vol. 50(C), pages 29-61.
    13. Paul Windrum & Giorgio Fagiolo & Alessio Moneta, 2007. "Empirical Validation of Agent-Based Models: Alternatives and Prospects," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 10(2), pages 1-8.
    14. Boswijk, H. Peter & Hommes, Cars H. & Manzan, Sebastiano, 2007. "Behavioral heterogeneity in stock prices," Journal of Economic Dynamics and Control, Elsevier, vol. 31(6), pages 1938-1970, June.
    15. Canova, Fabio, 1998. "Detrending and business cycle facts: A user's guide," Journal of Monetary Economics, Elsevier, vol. 41(3), pages 533-540, May.
    16. Matteo Richiardi & Roberto Leombruni & Nicole J. Saam & Michele Sonnessa, 2006. "A Common Protocol for Agent-Based Social Simulation," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 9(1), pages 1-15.
    17. Canova, Fabio, 1998. "Detrending and business cycle facts," Journal of Monetary Economics, Elsevier, vol. 41(3), pages 475-512, May.
    18. Giorgio Fagiolo & Andrea Roventini, 2012. "Macroeconomic Policy in DSGE and Agent-Based Models," Revue de l'OFCE, Presses de Sciences-Po, vol. 0(5), pages 67-116.
    19. Alfarano, Simone & Lux, Thomas & Wagner, Friedrich, 2006. "Estimation of a simple agent-based model of financial markets: An application to Australian stock and foreign exchange data," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 370(1), pages 38-42.
    20. Carlo Bianchi & Pasquale Cirillo & Mauro Gallegati & Pietro Vagliasindi, 2007. "Validating and Calibrating Agent-Based Models: A Case Study," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 245-264, October.
    21. Peter Winker & Manfred Gilli & Vahidin Jeleskovic, 2007. "An objective function for simulation based inference on exchange rate data," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 2(2), pages 125-145, December.
    22. Grazzini, Jakob & Richiardi, Matteo, 2015. "Estimation of ergodic agent-based models by simulated minimum distance," Journal of Economic Dynamics and Control, Elsevier, vol. 51(C), pages 148-165.
    23. Francesco Lamperti, 2016. "Empirical Validation of Simulated Models through the GSL-div: an Illustrative Application," LEM Papers Series 2016/18, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    24. Robert Marks, 2007. "Validating Simulation Models: A General Framework and Four Applied Examples," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 265-290, October.
    25. Grazzini, Jakob & Richiardi, Matteo G. & Tsionas, Mike, 2017. "Bayesian estimation of agent-based models," Journal of Economic Dynamics and Control, Elsevier, vol. 77(C), pages 26-47.
    26. Dosi, Giovanni & Fagiolo, Giorgio & Napoletano, Mauro & Roventini, Andrea & Treibich, Tania, 2015. "Fiscal and monetary policies in complex evolving economies," Journal of Economic Dynamics and Control, Elsevier, vol. 52(C), pages 166-189.
    27. Franke, Reiner & Westerhoff, Frank, 2012. "Structural stochastic volatility in asset pricing dynamics: Estimation and model contest," Journal of Economic Dynamics and Control, Elsevier, vol. 36(8), pages 1193-1211.
    28. Peter R. Hansen & Asger Lunde & James M. Nason, 2011. "The Model Confidence Set," Econometrica, Econometric Society, vol. 79(2), pages 453-497, March.
    29. Giannone, Domenico & Reichlin, Lucrezia & Sala, Luca, 2006. "VARs, common factors and the empirical validation of equilibrium business cycle models," Journal of Econometrics, Elsevier, vol. 132(1), pages 257-279, May.
    30. Guerini, Mattia & Moneta, Alessio, 2017. "A method for agent-based models validation," Journal of Economic Dynamics and Control, Elsevier, vol. 82(C), pages 125-141.
    31. repec:eee:csdana:v:56:y:2012:i:12:p:3959-3974 is not listed on IDEAS
    32. Tesfatsion, Leigh & Judd, Kenneth L., 2006. "Handbook of Computational Economics, Vol. 2: Agent-Based Computational Economics," Staff General Research Papers Archive 10368, Iowa State University, Department of Economics.
    33. Amilon, Henrik, 2008. "Estimation of an adaptive stock market model with heterogeneous agents," Journal of Empirical Finance, Elsevier, vol. 15(2), pages 342-362, March.
    34. Assenza, Tiziana & Delli Gatti, Domenico & Grazzini, Jakob, 2015. "Emergent dynamics of a macroeconomic agent based model with capital and credit," Journal of Economic Dynamics and Control, Elsevier, vol. 50(C), pages 5-28.
    35. Caiani, Alessandro & Godin, Antoine & Caverzasi, Eugenio & Gallegati, Mauro & Kinsella, Stephen & Stiglitz, Joseph E., 2016. "Agent based-stock flow consistent macroeconomics: Towards a benchmark model," Journal of Economic Dynamics and Control, Elsevier, vol. 69(C), pages 375-408.
    36. Jakob Grazzini, 2012. "Analysis of the Emergent Properties: Stationarity and Ergodicity," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 15(2), pages 1-7.
    37. Reiner Franke & Frank Westerhoff, 2016. "Why a simple herding model may generate the stylized facts of daily returns: explanation and estimation," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 11(1), pages 1-34, April.
    38. Simone Alfarano & Thomas Lux & Friedrich Wagner, 2005. "Estimation of Agent-Based Models: The Case of an Asymmetric Herding Model," Computational Economics, Springer;Society for Computational Economics, vol. 26(1), pages 19-49, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lamperti, F. & Dosi, G. & Napoletano, M. & Roventini, A. & Sapio, A., 2018. "Faraway, So Close: Coupled Climate and Economic Dynamics in an Agent-based Integrated Assessment Model," Ecological Economics, Elsevier, vol. 150(C), pages 315-339.
    2. Eric Brouillat & Maïder Saint-Jean, 2019. "Dura lex sed lex: why implementation gaps in environmental policy matter?," Cahiers du GREThA 2019-04, Groupe de Recherche en Economie Théorique et Appliquée (GREThA).
    3. Kukacka, Jiri & Jang, Tae-Seok & Sacht, Stephen, 2018. "On the estimation of behavioral macroeconomic models via simulated maximum likelihood," Economics Working Papers 2018-11, Christian-Albrechts-University of Kiel, Department of Economics.
    4. Mauro Napoletano & Eric Guerci & Nobuyuki Hanaki, 2018. "Recent advances in financial networks and agent-based model validation," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 13(1), pages 1-7, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ecosta:v:5:y:2018:i:c:p:83-106. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Dana Niculescu). General contact details of provider: https://www.journals.elsevier.com/econometrics-and-statistics .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.