IDEAS home Printed from https://ideas.repec.org/p/ssa/lemwps/2015-02.html
   My bibliography  Save this paper

An Information Theoretic Criterion for Empirical Validation of Time Series Models

Author

Listed:
  • Francesco Lamperti

Abstract

Simulated models suffer intrinsically from validation and comparison problems. The choice of a suitable indicator quantifying the distance between the model and the data is pivotal to model selection. However, how to validate and discriminate between alternative models is still an open problem calling for further investigation, especially in light of the increasing use of simulations in social sciences. In this paper, we present an information theoretic criterion to measure how close models' synthetic output replicates the properties of observable time series without the need to resort to any likelihood function or to impose stationarity requirements. The indicator is sufficiently general to be applied to any kind of model able to simulate or predict time series data, from simple univariate models such as Auto Regressive Moving Average (ARMA) and Markov processes to more complex objects including agent-based or dynamic stochastic general equilibrium models. More specifically, we use a simple function of the L-divergence computed at different block lengths in order to select the model that is better able to reproduce the distributions of time changes in the data. To evaluate the L-divergence, probabilities are estimated across frequencies including a correction for the systematic bias. Finally, using a known data generating process, we show how this indicator can be used to validate and discriminate between different models providing a precise measure of the distance between each of them and the data.

Suggested Citation

  • Francesco Lamperti, 2015. "An Information Theoretic Criterion for Empirical Validation of Time Series Models," LEM Papers Series 2015/02, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
  • Handle: RePEc:ssa:lemwps:2015/02
    as

    Download full text from publisher

    File URL: http://www.lem.sssup.it/WPLem/files/2015-02.pdf
    Download Restriction: no

    References listed on IDEAS

    as
    1. S. Alfarano & T. Lux & F. Wagner, 2007. "Empirical validation of stochastic models of interacting agents," The European Physical Journal B: Condensed Matter and Complex Systems, Springer;EDP Sciences, vol. 55(2), pages 183-187, January.
    2. Lee, Bong-Soo & Ingram, Beth Fisher, 1991. "Simulation estimation of time-series models," Journal of Econometrics, Elsevier, vol. 47(2-3), pages 197-205, February.
    3. G. Fagiolo & C. Birchenhall & P. Windrum, 2007. "Empirical Validation in Agent-based Models: Introduction to the Special Issue," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 189-194, October.
    4. Sandrine Jacob Leal & Mauro Napoletano & Andrea Roventini & Giorgio Fagiolo, 2016. "Rock around the clock: An agent-based model of low- and high-frequency trading," Journal of Evolutionary Economics, Springer, vol. 26(1), pages 49-76, March.
    5. Duffie, Darrell & Singleton, Kenneth J, 1993. "Simulated Moments Estimation of Markov Models of Asset Prices," Econometrica, Econometric Society, vol. 61(4), pages 929-952, July.
    6. Paul Windrum & Giorgio Fagiolo & Alessio Moneta, 2007. "Empirical Validation of Agent-Based Models: Alternatives and Prospects," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 10(2), pages 1-8.
    7. Canova, Fabio & Sala, Luca, 2009. "Back to square one: Identification issues in DSGE models," Journal of Monetary Economics, Elsevier, vol. 56(4), pages 431-449, May.
    8. Matteo Richiardi & Roberto Leombruni & Nicole J. Saam & Michele Sonnessa, 2006. "A Common Protocol for Agent-Based Social Simulation," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 9(1), pages 1-15.
    9. Giorgio Fagiolo & Andrea Roventini, 2012. "Macroeconomic Policy in DSGE and Agent-Based Models," Revue de l'OFCE, Presses de Sciences-Po, vol. 0(5), pages 67-116.
    10. Dosi, Giovanni & Fagiolo, Giorgio & Napoletano, Mauro & Roventini, Andrea, 2013. "Income distribution, credit and fiscal policies in an agent-based Keynesian model," Journal of Economic Dynamics and Control, Elsevier, vol. 37(8), pages 1598-1625.
    11. Dosi, Giovanni & Fagiolo, Giorgio & Roventini, Andrea, 2010. "Schumpeter meeting Keynes: A policy-friendly model of endogenous growth and business cycles," Journal of Economic Dynamics and Control, Elsevier, vol. 34(9), pages 1748-1767, September.
    12. Giannone, Domenico & Reichlin, Lucrezia & Sala, Luca, 2006. "VARs, common factors and the empirical validation of equilibrium business cycle models," Journal of Econometrics, Elsevier, vol. 132(1), pages 257-279, May.
    13. Dosi, Giovanni & Fagiolo, Giorgio & Napoletano, Mauro & Roventini, Andrea & Treibich, Tania, 2015. "Fiscal and monetary policies in complex evolving economies," Journal of Economic Dynamics and Control, Elsevier, vol. 52(C), pages 166-189.
    14. Gianbiagio Curato & Fabrizio Lillo, 2013. "Modeling the coupled return-spread high frequency dynamics of large tick assets," Papers 1310.4539, arXiv.org.
    15. Tesfatsion, Leigh & Judd, Kenneth L., 2006. "Handbook of Computational Economics, Vol. 2: Agent-Based Computational Economics," Staff General Research Papers Archive 10368, Iowa State University, Department of Economics.
    16. Simone Alfarano & Thomas Lux & Friedrich Wagner, 2005. "Estimation of Agent-Based Models: The Case of an Asymmetric Herding Model," Computational Economics, Springer;Society for Computational Economics, vol. 26(1), pages 19-49, August.
    17. Gilli, M. & Winker, P., 2003. "A global optimization heuristic for estimating agent based models," Computational Statistics & Data Analysis, Elsevier, vol. 42(3), pages 299-312, March.
    18. Pasquale Cirillo & Mauro Gallegati, 2012. "The Empirical Validation of an Agent-based Model," Eastern Economic Journal, Palgrave Macmillan;Eastern Economic Association, vol. 38(4), pages 525-547.
    19. Gourieroux, C & Monfort, A & Renault, E, 1993. "Indirect Inference," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 8(S), pages 85-118, Suppl. De.
    20. Kristin J. Forbes & Roberto Rigobon, 2002. "No Contagion, Only Interdependence: Measuring Stock Market Comovements," Journal of Finance, American Finance Association, vol. 57(5), pages 2223-2261, October.
    21. Alfarano, Simone & Lux, Thomas & Wagner, Friedrich, 2006. "Estimation of a simple agent-based model of financial markets: An application to Australian stock and foreign exchange data," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 370(1), pages 38-42.
    22. Bianchi, Carlo & Cirillo, Pasquale & Gallegati, Mauro & Vagliasindi, Pietro A., 2008. "Validation in agent-based models: An investigation on the CATS model," Journal of Economic Behavior & Organization, Elsevier, vol. 67(3-4), pages 947-964, September.
    23. Carlo Bianchi & Pasquale Cirillo & Mauro Gallegati & Pietro Vagliasindi, 2007. "Validating and Calibrating Agent-Based Models: A Case Study," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 245-264, October.
    24. Grazzini, Jakob & Richiardi, Matteo, 2015. "Estimation of ergodic agent-based models by simulated minimum distance," Journal of Economic Dynamics and Control, Elsevier, vol. 51(C), pages 148-165.
    25. Robert Marks, 2007. "Validating Simulation Models: A General Framework and Four Applied Examples," Computational Economics, Springer;Society for Computational Economics, vol. 30(3), pages 265-290, October.
    26. Jakob Grazzini, 2012. "Analysis of the Emergent Properties: Stationarity and Ergodicity," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 15(2), pages 1-7.
    27. Sylvain Barde, 2015. "A Practical, Universal, Information Criterion over Nth Order Markov Processes," Studies in Economics 1504, School of Economics, University of Kent.
    28. Sandrine Jacob Leal & Mauro Napoletano & Andrea Roventini & Giorgio Fagiolo, 2016. "Rock around the clock: An agent-based model of low- and high-frequency trading," Journal of Evolutionary Economics, Springer, vol. 26(1), pages 49-76, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Giorgio Fagiolo & Andrea Roventini, 2017. "Macroeconomic Policy in DSGE and Agent-Based Models Redux: New Developments and Challenges Ahead," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 20(1), pages 1-1.
    2. Lamperti, F. & Dosi, G. & Napoletano, M. & Roventini, A. & Sapio, A., 2018. "Faraway, So Close: Coupled Climate and Economic Dynamics in an Agent-based Integrated Assessment Model," Ecological Economics, Elsevier, vol. 150(C), pages 315-339.
    3. repec:eee:dyncon:v:82:y:2017:i:c:p:125-141 is not listed on IDEAS
    4. Sylvain Barde, 2015. "A Practical, Universal, Information Criterion over Nth Order Markov Processes," Studies in Economics 1504, School of Economics, University of Kent.
    5. Lamperti, Francesco & Roventini, Andrea & Sani, Amir, 2018. "Agent-based model calibration using machine learning surrogates," Journal of Economic Dynamics and Control, Elsevier, vol. 90(C), pages 366-389.
    6. Grazzini, Jakob & Richiardi, Matteo G. & Tsionas, Mike, 2017. "Bayesian estimation of agent-based models," Journal of Economic Dynamics and Control, Elsevier, vol. 77(C), pages 26-47.
    7. Francesco Lamperti, 2016. "Empirical Validation of Simulated Models through the GSL-div: an Illustrative Application," LEM Papers Series 2016/18, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    8. Guerini, Mattia & Moneta, Alessio, 2017. "A method for agent-based models validation," Journal of Economic Dynamics and Control, Elsevier, vol. 82(C), pages 125-141.
    9. Giovanni Dosi & Andrea Roventini & Emanuele Russo, 2017. "Endogenous growth and global divergence in a multi-country agent-based model," LEM Papers Series 2017/32, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    10. Barde, Sylvain, 2016. "Direct comparison of agent-based models of herding in financial markets," Journal of Economic Dynamics and Control, Elsevier, vol. 73(C), pages 329-353.
    11. repec:spr:jeicoo:v:13:y:2018:i:1:d:10.1007_s11403-018-0221-z is not listed on IDEAS
    12. Sylvain Barde, 2017. "A Practical, Accurate, Information Criterion for Nth Order Markov Processes," Computational Economics, Springer;Society for Computational Economics, vol. 50(2), pages 281-324, August.
    13. Francesco Lamperti & Giovanni Dosi & Mauro Napoletano & Andrea Roventini & Alessandro Sapio, 2018. "And Then He Wasn't a She: Climate Change and Green Transitions in an Agent-Based Integrated Assessment Model," LEM Papers Series 2018/14, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    14. Giorgio Fagiolo & Mattia Guerini & Francesco Lamperti & Alessio Moneta & Andrea Roventini, 2017. "Validation of Agent-Based Models in Economics and Finance," LEM Papers Series 2017/23, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    15. Sylvain Barde, 2015. "Direct calibration and comparison of agent-based herding models of financial markets," Studies in Economics 1507, School of Economics, University of Kent.
    16. Sander van der Hoog, 2017. "Deep Learning in (and of) Agent-Based Models: A Prospectus," Papers 1706.06302, arXiv.org.

    More about this item

    Keywords

    Simulations; Empirical Validation; Time Series; Agent Based Models;

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ssa:lemwps:2015/02. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (). General contact details of provider: http://edirc.repec.org/data/labssit.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.