IDEAS home Printed from https://ideas.repec.org/p/zbw/bubdps/072018.html
   My bibliography  Save this paper

How far can we forecast? Statistical tests of the predictive content

Author

Listed:
  • Breitung, Jörg
  • Knüppel, Malte

Abstract

Forecasts are useless whenever the forecast error variance fails to be smaller than the unconditional variance of the target variable. This paper develops tests for the null hypothesis that forecasts become uninformative beyond some limiting forecast horizon h. Following Diebold and Mariano (DM, 1995) we propose a test based on the comparison of the mean-squared error of the forecast and the sample variance. We show that the resulting test does not possess a limiting normal distribution and suggest two simple modifications of the DM-type test with different limiting null distributions. Furthermore, a forecast encompassing test is developed that tends to better control the size of the test. In our empirical analysis, we apply our tests to macroeconomic forecasts from the survey of Consensus Economics. Our results suggest that forecasts of macroeconomic key variables are barely informative beyond 2-4 quarters ahead.

Suggested Citation

  • Breitung, Jörg & Knüppel, Malte, 2018. "How far can we forecast? Statistical tests of the predictive content," Discussion Papers 07/2018, Deutsche Bundesbank.
    • Handle: RePEc:zbw:bubdps:072018
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/178500/1/1020157062.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    2. Andrew Patton & Allan Timmermann, 2012. "Forecast Rationality Tests Based on Multi-Horizon Bounds," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(1), pages 1-17.
    3. John W. Galbraith & Greg Tkacz, 2007. "Forecast content and content horizons for some important macroeconomic time series," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 40(3), pages 935-953, August.
    4. Francis X. Diebold & Lutz Kilian, 2001. "Measuring predictability: theory and macroeconomic applications," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 16(6), pages 657-669.
    5. Clark, Todd E. & West, Kenneth D., 2007. "Approximately normal tests for equal predictive accuracy in nested models," Journal of Econometrics, Elsevier, vol. 138(1), pages 291-311, May.
    6. Ang, Andrew & Bekaert, Geert & Wei, Min, 2007. "Do macro variables, asset markets, or surveys forecast inflation better?," Journal of Monetary Economics, Elsevier, vol. 54(4), pages 1163-1212, May.
    7. Jörg Breitung & Malte Knüppel, 2021. "How far can we forecast? Statistical tests of the predictive content," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(4), pages 369-392, June.
    8. Fama, Eugene F. & French, Kenneth R., 1988. "Dividend yields and expected stock returns," Journal of Financial Economics, Elsevier, vol. 22(1), pages 3-25, October.
    9. Isiklar, Gultekin & Lahiri, Kajal, 2007. "How far ahead can we forecast? Evidence from cross-country surveys," International Journal of Forecasting, Elsevier, vol. 23(2), pages 167-187.
    10. James H. Stock & Mark W. Watson, 2007. "Why Has U.S. Inflation Become Harder to Forecast?," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 39(s1), pages 3-33, February.
    11. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 33(1), pages 125-132.
    12. Clements,Michael & Hendry,David, 1998. "Forecasting Economic Time Series," Cambridge Books, Cambridge University Press, number 9780521634809, June.
    13. Granger, C. W. J. & Newbold, Paul, 1986. "Forecasting Economic Time Series," Elsevier Monographs, Elsevier, edition 2, number 9780122951831 edited by Shell, Karl.
    14. Knüppel, Malte, 2018. "Forecast-error-based estimation of forecast uncertainty when the horizon is increased," International Journal of Forecasting, Elsevier, vol. 34(1), pages 105-116.
    15. Jacob A. Mincer & Victor Zarnowitz, 1969. "The Evaluation of Economic Forecasts," NBER Chapters, in: Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance, pages 3-46, National Bureau of Economic Research, Inc.
    16. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    17. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.
    18. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    19. Andrews, Donald W K, 1991. "Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation," Econometrica, Econometric Society, vol. 59(3), pages 817-858, May.
    20. Jacob A. Mincer, 1969. "Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance," NBER Books, National Bureau of Economic Research, Inc, number minc69-1, October.
    21. Michael W. McCracken, 2019. "Tests of Conditional Predictive Ability: Some Simulation Evidence," Working Papers 2019-11, Federal Reserve Bank of St. Louis.
    22. Harvey, David & Leybourne, Stephen & Newbold, Paul, 1997. "Testing the equality of prediction mean squared errors," International Journal of Forecasting, Elsevier, vol. 13(2), pages 281-291, June.
    23. repec:bla:econom:v:42:y:1975:i:166:p:123-38 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Stamer, Vincent, 2022. "Thinking Outside the Container: A Sparse Partial Least Squares Approach to Forecasting Trade Flows," VfS Annual Conference 2022 (Basel): Big Data in Economics 264096, Verein für Socialpolitik / German Economic Association.
    2. Stamer, Vincent, 2024. "Thinking outside the container: A sparse partial least squares approach to forecasting trade flows," International Journal of Forecasting, Elsevier, vol. 40(4), pages 1336-1358.
    3. Regmi, Hari & Kuethe, Todd H. & Foster, Kenneth A., 2022. "Evaluation of USDA's Agricultural Exports Projections," 2022 Annual Meeting, July 31-August 2, Anaheim, California 322363, Agricultural and Applied Economics Association.
    4. Gibbs, Christopher G. & Vasnev, Andrey L., 2024. "Conditionally optimal weights and forward-looking approaches to combining forecasts," International Journal of Forecasting, Elsevier, vol. 40(4), pages 1734-1751.
    5. Daniel Borup & Bent Jesper Christensen & Yunus Emre Ergemen, 2019. "Assessing predictive accuracy in panel data models with long-range dependence," CREATES Research Papers 2019-04, Department of Economics and Business Economics, Aarhus University.
    6. Jean-Yves Pitarakis, 2023. "Direct Multi-Step Forecast based Comparison of Nested Models via an Encompassing Test," Papers 2312.16099, arXiv.org.
    7. Zhu, Tiantian & Haugen, Stein & Liu, Yiliu & Yang, Xue, 2023. "A value of prediction model to estimate optimal response time to threats for accident prevention," Reliability Engineering and System Safety, Elsevier, vol. 232(C).
    8. Carola Binder & Tucker S. Mcelroy & Xuguang S. Sheng, 2022. "The Term Structure of Uncertainty: New Evidence from Survey Expectations," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 54(1), pages 39-71, February.
    9. Fabian Kruger & Hendrik Plett, 2022. "Prediction intervals for economic fixed-event forecasts," Papers 2210.13562, arXiv.org, revised Mar 2024.
    10. Cepni, Oguzhan & Clements, Michael P., 2024. "How local is the local inflation factor? Evidence from emerging European countries," International Journal of Forecasting, Elsevier, vol. 40(1), pages 160-183.
    11. Jörg Breitung & Malte Knüppel, 2021. "How far can we forecast? Statistical tests of the predictive content," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(4), pages 369-392, June.
    12. Jorge Abad & Javier Suarez, 2018. "The Procyclicality of Expected Credit Loss Provisions," Working Papers wp2018_1806, CEMFI.
    13. Günter Bamberg & Sebastian Heiden & Michael Krapp, 2023. "Langfrist-Prognose von Performance-Indizes: Vergleich einiger Verfahren [Long-term forecast of performance indices: comparison of some procedures]," AStA Wirtschafts- und Sozialstatistisches Archiv, Springer;Deutsche Statistische Gesellschaft - German Statistical Society, vol. 17(2), pages 161-181, June.
    14. Marc-Oliver Pohle, 2020. "The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation," Papers 2005.01835, arXiv.org.
    15. M. Chudý & S. Karmakar & W. B. Wu, 2020. "Long-term prediction intervals of economic time series," Empirical Economics, Springer, vol. 58(1), pages 191-222, January.
    16. repec:ags:aaea22:335690 is not listed on IDEAS
    17. Jörg Döpke & Karsten Müller & Lars Tegtmeier, 2023. "Moments of cross‐sectional stock market returns and the German business cycle," Economic Notes, Banca Monte dei Paschi di Siena SpA, vol. 52(2), July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    2. Pablo Pincheira Brown & Nicolás Hardy, 2024. "Correlation‐based tests of predictability," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 43(6), pages 1835-1858, September.
    3. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    4. Marc-Oliver Pohle, 2020. "The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation," Papers 2005.01835, arXiv.org.
    5. Barbara Rossi & Atsushi Inoue, 2012. "Out-of-Sample Forecast Tests Robust to the Choice of Window Size," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(3), pages 432-453, April.
    6. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    7. Cepni, Oguzhan & Clements, Michael P., 2024. "How local is the local inflation factor? Evidence from emerging European countries," International Journal of Forecasting, Elsevier, vol. 40(1), pages 160-183.
    8. Rossi, Barbara & Sekhposyan, Tatevik, 2011. "Understanding models' forecasting performance," Journal of Econometrics, Elsevier, vol. 164(1), pages 158-172, September.
    9. Gibbs, Christopher G. & Vasnev, Andrey L., 2024. "Conditionally optimal weights and forward-looking approaches to combining forecasts," International Journal of Forecasting, Elsevier, vol. 40(4), pages 1734-1751.
    10. Busetti, Fabio & Marcucci, Juri, 2013. "Comparing forecast accuracy: A Monte Carlo investigation," International Journal of Forecasting, Elsevier, vol. 29(1), pages 13-27.
    11. Nathan Goldstein & Ben‐Zion Zilberfarb, 2023. "The closer we get, the better we are?," Economic Inquiry, Western Economic Association International, vol. 61(2), pages 364-376, April.
    12. Barbara Rossi, 2021. "Forecasting in the Presence of Instabilities: How We Know Whether Models Predict Well and How to Improve Them," Journal of Economic Literature, American Economic Association, vol. 59(4), pages 1135-1190, December.
    13. Pablo Pincheira Brown & Nicolás Hardy, 2024. "The mean squared prediction error paradox," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 43(6), pages 2298-2321, September.
    14. Matei Demetrescu & Christoph Hanck & Robinson Kruse‐Becher, 2022. "Robust inference under time‐varying volatility: A real‐time evaluation of professional forecasters," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 1010-1030, August.
    15. West, Kenneth D., 2006. "Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 3, pages 99-134, Elsevier.
    16. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    17. Firmin Doko Tchatoka & Qazi Haque, 2023. "On bootstrapping tests of equal forecast accuracy for nested models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(7), pages 1844-1864, November.
    18. Kontogeorgos, Georgios & Lambrias, Kyriacos, 2019. "An analysis of the Eurosystem/ECB projections," Working Paper Series 2291, European Central Bank.
    19. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    20. Granziera, Eleonora & Sekhposyan, Tatevik, 2019. "Predicting relative forecasting performance: An empirical investigation," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1636-1657.

    More about this item

    Keywords

    Hypothesis Testing; Predictive Accuracy; Informativeness;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C32 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes; State Space Models
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:bubdps:072018. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/dbbgvde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.