IDEAS home Printed from https://ideas.repec.org/p/kof/wpskof/15-397.html
   My bibliography  Save this paper

Dissecting Models' Forecasting Performance

Author

Listed:
  • Boriss Siliverstovs

Abstract

In this paper we suggest an approach to comparison of models' forecasting performance in unstable environments. Our approach is based on combination of the Cumulated Sum of Squared Forecast Error Differential (CSSFED) suggested earlier in Welch and Goyal (2008) and the Bayesian change point analysis based on Barry and Hartigan (1993). The latter methodology provides the formal statistical analysis of the CSSFED time series which turned out to be a powerful graphical tool for tracking how the relative forecasting performance of competing models evolves over time. We illustrate the suggested approach by using forecasts of the GDP growth rate in Switzerland.

Suggested Citation

  • Boriss Siliverstovs, 2015. "Dissecting Models' Forecasting Performance," KOF Working papers 15-397, KOF Swiss Economic Institute, ETH Zurich.
    • Handle: RePEc:kof:wpskof:15-397
    DOI: 10.3929/ethz-a-010692101
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.3929/ethz-a-010692101
    Download Restriction: no
    File URL: https://libkey.io/10.3929/ethz-a-010692101?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Ivo Welch & Amit Goyal, 2008. "A Comprehensive Look at The Empirical Performance of Equity Premium Prediction," The Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1455-1508, July.
    2. Giannone, Domenico & Reichlin, Lucrezia & Small, David, 2008. "Nowcasting: The real-time informational content of macroeconomic data," Journal of Monetary Economics, Elsevier, vol. 55(4), pages 665-676, May.
    3. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    4. G. Rünstler & K. Barhoumi & S. Benk & R. Cristadoro & A. Den Reijer & A. Jakaitiene & P. Jelonek & A. Rua & K. Ruth & C. Van Nieuwenhuyze, 2009. "Short-term forecasting of GDP using large datasets: a pseudo real-time forecast evaluation exercise," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 28(7), pages 595-611.
    5. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.
    6. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    7. Barbara Rossi, 2013. "Exchange Rate Predictability," Journal of Economic Literature, American Economic Association, vol. 51(4), pages 1063-1119, December.
    8. Kapetanios, George & Labhard, Vincent & Price, Simon, 2008. "Forecast combination and the Bank of England's suite of statistical forecasting models," Economic Modelling, Elsevier, vol. 25(4), pages 772-792, July.
    9. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    10. Liu, Philip & Matheson, Troy & Romeu, Rafael, 2012. "Real-time forecasts of economic activity for Latin American economies," Economic Modelling, Elsevier, vol. 29(4), pages 1090-1098.
    11. Buncic, Daniel & Moretto, Carlo, 2015. "Forecasting copper prices with dynamic averaging and selection models," The North American Journal of Economics and Finance, Elsevier, vol. 33(C), pages 1-38.
    12. Boriss Siliverstovs, 2016. "The franc shock and Swiss GDP: how long does it take to start feeling the pain?," Applied Economics, Taylor & Francis Journals, vol. 48(36), pages 3432-3441, August.
    13. Clark, Todd E. & West, Kenneth D., 2007. "Approximately normal tests for equal predictive accuracy in nested models," Journal of Econometrics, Elsevier, vol. 138(1), pages 291-311, May.
    14. Knut Are Aastveit & Claudia Foroni & Francesco Ravazzolo, 2017. "Density Forecasts With Midas Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(4), pages 783-801, June.
    15. Reichlin, Lucrezia & Giannone, Domenico & Small, David, 2005. "Nowcasting GDP and Inflation: The Real Time Informational Content of Macroeconomic Data Releases," CEPR Discussion Papers 5178, C.E.P.R. Discussion Papers.
    16. Schinasi, Garry J. & Swamy, P. A. V. B., 1989. "The out-of-sample forecasting performance of exchange rate models when coefficients are allowed to change," Journal of International Money and Finance, Elsevier, vol. 8(3), pages 375-390, September.
    17. Kenneth S. Rogoff & Vania Stavrakeva, 2008. "The Continuing Puzzle of Short Horizon Exchange Rate Forecasting," NBER Working Papers 14071, National Bureau of Economic Research, Inc.
    18. Siliverstovs Boriss & Kholodilin Konstantin A., 2012. "Assessing the Real-Time Informational Content of Macroeconomic Data Releases for Now-/Forecasting GDP: Evidence for Switzerland," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 232(4), pages 429-444, August.
    19. Paye, Bradley S. & Timmermann, Allan, 2006. "Instability of return prediction models," Journal of Empirical Finance, Elsevier, vol. 13(3), pages 274-315, June.
    20. Raffaella Giacomini & Barbara Rossi, 2006. "How Stable is the Forecasting Performance of the Yield Curve for Output Growth?," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 68(s1), pages 783-795, December.
    21. Dias, Francisco & Pinheiro, Maximiano & Rua, António, 2015. "Forecasting Portuguese GDP with factor models: Pre- and post-crisis evidence," Economic Modelling, Elsevier, vol. 44(C), pages 266-272.
    22. Raffaella Giacomini & Barbara Rossi, 2010. "Forecast comparisons in unstable environments," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 595-620.
    23. Schrimpf, Andreas & Wang, Qingwei, 2010. "A reappraisal of the leading indicator properties of the yield curve under structural instability," International Journal of Forecasting, Elsevier, vol. 26(4), pages 836-857, October.
    24. Alessandro Girardi & Christian Gayer & Andreas Reuter, 2016. "The Role of Survey Data in Nowcasting Euro Area GDP Growth," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 35(5), pages 400-418, August.
    25. James H. Stock & Mark W. Watson, 2007. "Why Has U.S. Inflation Become Harder to Forecast?," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 39(s1), pages 3-33, February.
    26. David E. Rapach & Jack K. Strauss & Guofu Zhou, 2013. "International Stock Return Predictability: What Is the Role of the United States?," Journal of Finance, American Finance Association, vol. 68(4), pages 1633-1662, August.
    27. Erdman, Chandra & Emerson, John W., 2007. "bcp: An R Package for Performing a Bayesian Analysis of Change Point Problems," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 23(i03).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Boriss Siliverstovs & Daniel S. Wochner, 2021. "State‐dependent evaluation of predictive ability," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 40(3), pages 547-574, April.
    2. Boriss Siliverstovs, 2021. "New York FED Staff Nowcasts and Reality: What Can We Learn about the Future, the Present, and the Past?," Econometrics, MDPI, vol. 9(1), pages 1-25, March.
    3. Boriss Siliverstovs, 2021. "Gauging the Effect of Influential Observations on Measures of Relative Forecast Accuracy in a Post-COVID-19 Era: Application to Nowcasting Euro Area GDP Growth," Working Papers 2021/01, Latvijas Banka.
    4. Yingying Xu & Donald Lien, 2022. "Forecasting volatilities of oil and gas assets: A comparison of GAS, GARCH, and EGARCH models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 41(2), pages 259-278, March.
    5. Daniel Wochner, 2020. "Dynamic Factor Trees and Forests – A Theory-led Machine Learning Framework for Non-Linear and State-Dependent Short-Term U.S. GDP Growth Predictions," KOF Working papers 20-472, KOF Swiss Economic Institute, ETH Zurich.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    2. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    3. Barbara Rossi, 2019. "Forecasting in the presence of instabilities: How do we know whether models predict well and how to improve them," Economics Working Papers 1711, Department of Economics and Business, Universitat Pompeu Fabra, revised Jul 2021.
    4. Mogliani, Matteo & Darné, Olivier & Pluyaud, Bertrand, 2017. "The new MIBA model: Real-time nowcasting of French GDP using the Banque de France's monthly business survey," Economic Modelling, Elsevier, vol. 64(C), pages 26-39.
    5. Odendahl, Florens & Rossi, Barbara & Sekhposyan, Tatevik, 2023. "Evaluating forecast performance with state dependence," Journal of Econometrics, Elsevier, vol. 237(2).
    6. Daniel Borup & Jonas N. Eriksen & Mads M. Kjær & Martin Thyrsgaard, 2020. "Predicting bond return predictability," CREATES Research Papers 2020-09, Department of Economics and Business Economics, Aarhus University.
    7. Rossi, Barbara & Odendahl, Florens & Sekhposyan, Tatevik, 2020. "Comparing Forecast Performance with State Dependence," CEPR Discussion Papers 15217, C.E.P.R. Discussion Papers.
    8. Daniel Borup & Martin Thyrsgaard, 2017. "Statistical tests for equal predictive ability across multiple forecasting methods," CREATES Research Papers 2017-19, Department of Economics and Business Economics, Aarhus University.
    9. Jamali, Ibrahim & Yamani, Ehab, 2019. "Out-of-sample exchange rate predictability in emerging markets: Fundamentals versus technical analysis," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 61(C), pages 241-263.
    10. Matei Demetrescu & Christoph Hanck & Robinson Kruse‐Becher, 2022. "Robust inference under time‐varying volatility: A real‐time evaluation of professional forecasters," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 1010-1030, August.
    11. Ahmed, Shamim & Liu, Xiaoquan & Valente, Giorgio, 2016. "Can currency-based risk factors help forecast exchange rates?," International Journal of Forecasting, Elsevier, vol. 32(1), pages 75-97.
    12. Byron Botha & Geordie Reid & Tim Olds & Daan Steenkamp & Rossouw van Jaarsveld, 2021. "Nowcasting South African GDP using a suite of statistical models," Working Papers 11001, South African Reserve Bank.
    13. Faust, Jon & Wright, Jonathan H., 2013. "Forecasting Inflation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 2-56, Elsevier.
    14. Barbara Rossi & Atsushi Inoue, 2012. "Out-of-Sample Forecast Tests Robust to the Choice of Window Size," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(3), pages 432-453, April.
    15. Amat, Christophe & Michalski, Tomasz & Stoltz, Gilles, 2018. "Fundamentals and exchange rate forecastability with simple machine learning methods," Journal of International Money and Finance, Elsevier, vol. 88(C), pages 1-24.
    16. Christian Glocker & Serguei Kaniovski, 2022. "Macroeconometric forecasting using a cluster of dynamic factor models," Empirical Economics, Springer, vol. 63(1), pages 43-91, July.
    17. Caruso, Alberto, 2018. "Nowcasting with the help of foreign indicators: The case of Mexico," Economic Modelling, Elsevier, vol. 69(C), pages 160-168.
    18. Blasques, F. & Koopman, S.J. & Mallee, M. & Zhang, Z., 2016. "Weighted maximum likelihood for dynamic factor analysis and forecasting with mixed frequency data," Journal of Econometrics, Elsevier, vol. 193(2), pages 405-417.
    19. David Coble & Pablo Pincheira, 2021. "Forecasting building permits with Google Trends," Empirical Economics, Springer, vol. 61(6), pages 3315-3345, December.
    20. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.

    More about this item

    Keywords

    Forecasting; Forecast Evaluation; Change Point Detection; Bayesian Estimation;
    All these keywords.

    JEL classification:

    • C22 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:kof:wpskof:15-397. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/koethch.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.