IDEAS home Printed from https://ideas.repec.org/p/ssa/lemwps/2015-02.html
   My bibliography  Save this paper

An Information Theoretic Criterion for Empirical Validation of Time Series Models

Author

Listed:
  • Francesco Lamperti

Abstract

Simulated models suffer intrinsically from validation and comparison problems. The choice of a suitable indicator quantifying the distance between the model and the data is pivotal to model selection. However, how to validate and discriminate between alternative models is still an open problem calling for further investigation, especially in light of the increasing use of simulations in social sciences. In this paper, we present an information theoretic criterion to measure how close models' synthetic output replicates the properties of observable time series without the need to resort to any likelihood function or to impose stationarity requirements. The indicator is sufficiently general to be applied to any kind of model able to simulate or predict time series data, from simple univariate models such as Auto Regressive Moving Average (ARMA) and Markov processes to more complex objects including agent-based or dynamic stochastic general equilibrium models. More specifically, we use a simple function of the L-divergence computed at different block lengths in order to select the model that is better able to reproduce the distributions of time changes in the data. To evaluate the L-divergence, probabilities are estimated across frequencies including a correction for the systematic bias. Finally, using a known data generating process, we show how this indicator can be used to validate and discriminate between different models providing a precise measure of the distance between each of them and the data.

Suggested Citation

  • Francesco Lamperti, 2015. "An Information Theoretic Criterion for Empirical Validation of Time Series Models," LEM Papers Series 2015/02, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
  • Handle: RePEc:ssa:lemwps:2015/02
    as

    Download full text from publisher

    File URL: http://www.lem.sssup.it/WPLem/files/2015-02.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. S. Alfarano & T. Lux & F. Wagner, 2007. "Empirical validation of stochastic models of interacting agents," The European Physical Journal B: Condensed Matter and Complex Systems, Springer;EDP Sciences, vol. 55(2), pages 183-187, January.
    2. repec:hal:wpspec:info:hdl:2441/f6h8764enu2lskk9p4oq9ig8k is not listed on IDEAS
    3. Lee, Bong-Soo & Ingram, Beth Fisher, 1991. "Simulation estimation of time-series models," Journal of Econometrics, Elsevier, vol. 47(2-3), pages 197-205, February.
    4. G. Fagiolo & C. Birchenhall & P. Windrum, 2007. "Empirical Validation in Agent-based Models: Introduction to the Special Issue," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 189-194, October.
    5. repec:hal:spmain:info:hdl:2441/3qv4spsglp8tmorvev1h0duo4p is not listed on IDEAS
    6. Sandrine Jacob Leal & Mauro Napoletano & Andrea Roventini & Giorgio Fagiolo, 2016. "Rock around the clock: An agent-based model of low- and high-frequency trading," Journal of Evolutionary Economics, Springer, vol. 26(1), pages 49-76, March.
    7. repec:hal:spmain:info:hdl:2441/f6h8764enu2lskk9p4oq9ig8k is not listed on IDEAS
    8. Matteo Richiardi & Roberto Leombruni & Nicole J. Saam & Michele Sonnessa, 2006. "A Common Protocol for Agent-Based Social Simulation," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 9(1), pages 1-15.
    9. Giorgio Fagiolo & Andrea Roventini, 2012. "Macroeconomic Policy in DSGE and Agent-Based Models," Revue de l'OFCE, Presses de Sciences-Po, vol. 0(5), pages 67-116.
    10. Dosi, Giovanni & Fagiolo, Giorgio & Roventini, Andrea, 2010. "Schumpeter meeting Keynes: A policy-friendly model of endogenous growth and business cycles," Journal of Economic Dynamics and Control, Elsevier, vol. 34(9), pages 1748-1767, September.
    11. Giannone, Domenico & Reichlin, Lucrezia & Sala, Luca, 2006. "VARs, common factors and the empirical validation of equilibrium business cycle models," Journal of Econometrics, Elsevier, vol. 132(1), pages 257-279, May.
    12. Dosi, Giovanni & Fagiolo, Giorgio & Napoletano, Mauro & Roventini, Andrea & Treibich, Tania, 2015. "Fiscal and monetary policies in complex evolving economies," Journal of Economic Dynamics and Control, Elsevier, vol. 52(C), pages 166-189.
    13. Tesfatsion, Leigh & Judd, Kenneth L., 2006. "Handbook of Computational Economics, Vol. 2: Agent-Based Computational Economics," Staff General Research Papers Archive 10368, Iowa State University, Department of Economics.
    14. Duffie, Darrell & Singleton, Kenneth J, 1993. "Simulated Moments Estimation of Markov Models of Asset Prices," Econometrica, Econometric Society, vol. 61(4), pages 929-952, July.
    15. Sandrine Jacob Leal & Mauro Napoletano & Andrea Roventini & Giorgio Fagiolo, 2016. "Rock around the clock: An agent-based model of low- and high-frequency trading," Journal of Evolutionary Economics, Springer, vol. 26(1), pages 49-76, March.
    16. Gourieroux, C & Monfort, A & Renault, E, 1993. "Indirect Inference," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 8(S), pages 85-118, Suppl. De.
    17. James H. Stock & Mark W. Watson, 2003. "Has the Business Cycle Changed and Why?," NBER Chapters, in: NBER Macroeconomics Annual 2002, Volume 17, pages 159-230, National Bureau of Economic Research, Inc.
    18. Alfarano, Simone & Lux, Thomas & Wagner, Friedrich, 2006. "Estimation of a simple agent-based model of financial markets: An application to Australian stock and foreign exchange data," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 370(1), pages 38-42.
    19. Leigh Tesfatsion & Kenneth L. Judd (ed.), 2006. "Handbook of Computational Economics," Handbook of Computational Economics, Elsevier, edition 1, volume 2, number 2.
    20. Robert Marks, 2007. "Validating Simulation Models: A General Framework and Four Applied Examples," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 265-290, October.
    21. Canova, Fabio & Sala, Luca, 2009. "Back to square one: Identification issues in DSGE models," Journal of Monetary Economics, Elsevier, vol. 56(4), pages 431-449, May.
    22. Sandrine Jacob Leal & Mauro Napoletano & Andrea Roventini & Giorgio Fagiolo, 2016. "Rock around the clock: An agent-based model of low- and high-frequency trading," Journal of Evolutionary Economics, Springer, vol. 26(1), pages 49-76, March.
    23. Dosi, Giovanni & Fagiolo, Giorgio & Napoletano, Mauro & Roventini, Andrea, 2013. "Income distribution, credit and fiscal policies in an agent-based Keynesian model," Journal of Economic Dynamics and Control, Elsevier, vol. 37(8), pages 1598-1625.
    24. Kristin J. Forbes & Roberto Rigobon, 2002. "No Contagion, Only Interdependence: Measuring Stock Market Comovements," Journal of Finance, American Finance Association, vol. 57(5), pages 2223-2261, October.
    25. Grazzini, Jakob & Richiardi, Matteo, 2015. "Estimation of ergodic agent-based models by simulated minimum distance," Journal of Economic Dynamics and Control, Elsevier, vol. 51(C), pages 148-165.
    26. Dosi, Giovanni & Fagiolo, Giorgio & Napoletano, Mauro & Roventini, Andrea & Treibich, Tania, 2015. "Fiscal and monetary policies in complex evolving economies," Journal of Economic Dynamics and Control, Elsevier, vol. 52(C), pages 166-189.
    27. Sylvain Barde, 2015. "A Practical, Universal, Information Criterion over Nth Order Markov Processes," Studies in Economics 1504, School of Economics, University of Kent.
    28. Paul Windrum & Giorgio Fagiolo & Alessio Moneta, 2007. "Empirical Validation of Agent-Based Models: Alternatives and Prospects," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 10(2), pages 1-8.
    29. Gianbiagio Curato & Fabrizio Lillo, 2013. "Modeling the coupled return-spread high frequency dynamics of large tick assets," Papers 1310.4539, arXiv.org.
    30. Dosi, Giovanni & Fagiolo, Giorgio & Napoletano, Mauro & Roventini, Andrea & Treibich, Tania, 2015. "Fiscal and monetary policies in complex evolving economies," Journal of Economic Dynamics and Control, Elsevier, vol. 52(C), pages 166-189.
    31. Horst Hanusch & Andreas Pyka (ed.), 2007. "Elgar Companion to Neo-Schumpeterian Economics," Books, Edward Elgar Publishing, number 2973, June.
    32. E. Samanidou & E. Zschischang & D. Stauffer & T. Lux, 2007. "Agent-based Models of Financial Markets," Papers physics/0701140, arXiv.org.
    33. Dosi, Giovanni & Fagiolo, Giorgio & Napoletano, Mauro & Roventini, Andrea & Treibich, Tania, 2015. "Fiscal and monetary policies in complex evolving economies," Journal of Economic Dynamics and Control, Elsevier, vol. 52(C), pages 166-189.
    34. Simone Alfarano & Thomas Lux & Friedrich Wagner, 2005. "Estimation of Agent-Based Models: The Case of an Asymmetric Herding Model," Computational Economics, Springer;Society for Computational Economics, vol. 26(1), pages 19-49, August.
    35. Gilli, M. & Winker, P., 2003. "A global optimization heuristic for estimating agent based models," Computational Statistics & Data Analysis, Elsevier, vol. 42(3), pages 299-312, March.
    36. Pasquale Cirillo & Mauro Gallegati, 2012. "The Empirical Validation of an Agent-based Model," Eastern Economic Journal, Palgrave Macmillan;Eastern Economic Association, vol. 38(4), pages 525-547.
    37. Giovanni Dosi & Giorgio Fagiolo & Mauro Napoletano & Andrea Roventini, 2012. "Income Distribution, Credit and Fiscal Policies in an Agent-Based Keynesian Model," Working Papers hal-04141136, HAL.
    38. Bianchi, Carlo & Cirillo, Pasquale & Gallegati, Mauro & Vagliasindi, Pietro A., 2008. "Validation in agent-based models: An investigation on the CATS model," Journal of Economic Behavior & Organization, Elsevier, vol. 67(3-4), pages 947-964, September.
    39. Carlo Bianchi & Pasquale Cirillo & Mauro Gallegati & Pietro Vagliasindi, 2007. "Validating and Calibrating Agent-Based Models: A Case Study," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 245-264, October.
    40. Dosi, Giovanni & Fagiolo, Giorgio & Napoletano, Mauro & Roventini, Andrea & Treibich, Tania, 2015. "Fiscal and monetary policies in complex evolving economies," Journal of Economic Dynamics and Control, Elsevier, vol. 52(C), pages 166-189.
    41. Jakob Grazzini, 2012. "Analysis of the Emergent Properties: Stationarity and Ergodicity," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 15(2), pages 1-7.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Giorgio Fagiolo & Mattia Guerini & Francesco Lamperti & Alessio Moneta & Andrea Roventini, 2017. "Validation of Agent-Based Models in Economics and Finance," LEM Papers Series 2017/23, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    2. Francesco Lamperti, 2018. "Empirical validation of simulated models through the GSL-div: an illustrative application," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 13(1), pages 143-171, April.
    3. Francesco Lamperti, 2016. "Empirical Validation of Simulated Models through the GSL-div: an Illustrative Application," LEM Papers Series 2016/18, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    4. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    5. repec:hal:spmain:info:hdl:2441/20hflp7eqn97boh50no50tv67n is not listed on IDEAS
    6. Lamperti, Francesco, 2018. "An information theoretic criterion for empirical validation of simulation models," Econometrics and Statistics, Elsevier, vol. 5(C), pages 83-106.
    7. Guerini, Mattia & Moneta, Alessio, 2017. "A method for agent-based models validation," Journal of Economic Dynamics and Control, Elsevier, vol. 82(C), pages 125-141.
    8. repec:hal:spmain:info:hdl:2441/13thfd12aa8rmplfudlgvgahff is not listed on IDEAS
    9. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    10. Grazzini, Jakob & Richiardi, Matteo, 2015. "Estimation of ergodic agent-based models by simulated minimum distance," Journal of Economic Dynamics and Control, Elsevier, vol. 51(C), pages 148-165.
    11. Francesco Lamperti & Giovanni Dosi & Mauro Napoletano & Andrea Roventini & Alessandro Sapio, 2018. "And then he wasn't a she : Climate change and green transitions in an agent-based integrated assessment model," Working Papers hal-03443464, HAL.
    12. Lamperti, F. & Dosi, G. & Napoletano, M. & Roventini, A. & Sapio, A., 2018. "Faraway, So Close: Coupled Climate and Economic Dynamics in an Agent-based Integrated Assessment Model," Ecological Economics, Elsevier, vol. 150(C), pages 315-339.
    13. repec:hal:spmain:info:hdl:2441/5vt1fet9fq9o5pkgj2qh2vn1cm is not listed on IDEAS
    14. Lamperti, F. & Dosi, G. & Napoletano, M. & Roventini, A. & Sapio, A., 2020. "Climate change and green transitions in an agent-based integrated assessment model," Technological Forecasting and Social Change, Elsevier, vol. 153(C).
    15. repec:hal:spmain:info:hdl:2441/4hs7liq1f49gh9chdf7r17gam6 is not listed on IDEAS
    16. Teglio, Andrea & Mazzocchetti, Andrea & Ponta, Linda & Raberto, Marco & Cincotti, Silvano, 2019. "Budgetary rigour with stimulus in lean times: Policy advices from an agent-based model," Journal of Economic Behavior & Organization, Elsevier, vol. 157(C), pages 59-83.
    17. Delli Gatti,Domenico & Fagiolo,Giorgio & Gallegati,Mauro & Richiardi,Matteo & Russo,Alberto (ed.), 2018. "Agent-Based Models in Economics," Cambridge Books, Cambridge University Press, number 9781108400046, February.
    18. Barde, Sylvain, 2020. "Macroeconomic simulation comparison with a multivariate extension of the Markov information criterion," Journal of Economic Dynamics and Control, Elsevier, vol. 111(C).
    19. Giovanni Dosi & Andrea Roventini, 2017. "Agent-Based Macroeconomics and Classical Political Economy: Some Italian Roots," Italian Economic Journal: A Continuation of Rivista Italiana degli Economisti and Giornale degli Economisti, Springer;Società Italiana degli Economisti (Italian Economic Association), vol. 3(3), pages 261-283, November.
    20. Mario Martinoli & Alessio Moneta & Gianluca Pallante, 2022. "Calibration and Validation of Macroeconomic Simulation Models by Statistical Causal Search," LEM Papers Series 2022/33, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    21. Donovan Platt, 2019. "A Comparison of Economic Agent-Based Model Calibration Methods," Papers 1902.05938, arXiv.org.
    22. Lamperti, Francesco & Bosetti, Valentina & Roventini, Andrea & Tavoni, Massimo & Treibich, Tania, 2021. "Three green financial policies to address climate risks," Journal of Financial Stability, Elsevier, vol. 54(C).
    23. Tommaso Ciarli & André Lorentz & Marco Valente & Maria Savona, 2019. "Structural changes and growth regimes," Journal of Evolutionary Economics, Springer, vol. 29(1), pages 119-176, March.
    24. Guerini, Mattia & Napoletano, Mauro & Roventini, Andrea, 2018. "No man is an Island: The impact of heterogeneity and local interactions on macroeconomic dynamics," Economic Modelling, Elsevier, vol. 68(C), pages 82-95.

    More about this item

    Keywords

    Simulations; Empirical Validation; Time Series; Agent Based Models;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ssa:lemwps:2015/02. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/labssit.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (email available below). General contact details of provider: https://edirc.repec.org/data/labssit.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.