IDEAS home Printed from https://ideas.repec.org/p/fip/fedkrw/rwp01-14.html
   My bibliography  Save this paper

Evaluating long-horizon forecasts

Author

Abstract

This paper examines the asymptotic and finite-sample properties of tests of equal forecast accuracy and encompassing applied to predictions from nested long-horizon regression models. We first derive the asymptotic distributions of a set of tests of equal forecast accuracy and encompassing, showing that the tests have non-standard distributions that depend on the parameters of the data-generating process. Using a simple parametric bootstrap for inference, we then conduct Monte Carlo simulations of a range of data-generating processes to examine the finite-sample size and power of the tests. In these simulations, the bootstrap yields tests with good finite-sample size and power properties, with the encompassing test proposed by Clark and McCracken (2001a) having superior power. The paper concludes with a reexamination of the predictive content of capacity utilization for core inflation.

Suggested Citation

  • Todd E. Clark & Michael W. McCracken, 2001. "Evaluating long-horizon forecasts," Research Working Paper RWP 01-14, Federal Reserve Bank of Kansas City.
    • Handle: RePEc:fip:fedkrw:rwp01-14
    as

    Download full text from publisher

    File URL: https://www.kansascityfed.org/documents/5407/pdf-RWP01-14.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Chao, John & Corradi, Valentina & Swanson, Norman R., 2001. "Out-Of-Sample Tests For Granger Causality," Macroeconomic Dynamics, Cambridge University Press, vol. 5(4), pages 598-620, September.
    2. Orphanides, Athanasios & van Norden, Simon, 2005. "The Reliability of Inflation Forecasts Based on Output Gap Estimates in Real Time," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 37(3), pages 583-601, June.
    3. Clarida, Richard H. & Sarno, Lucio & Taylor, Mark P. & Valente, Giorgio, 2003. "The out-of-sample success of term structure models as exchange rate predictors: a step beyond," Journal of International Economics, Elsevier, vol. 60(1), pages 61-83, May.
    4. Kilian, Lutz, 1999. "Exchange Rates and Monetary Fundamentals: What Do We Learn from Long-Horizon Regressions?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 14(5), pages 491-510, Sept.-Oct.
    5. Shintani, Mototsugu, 2005. "Nonlinear Forecasting Analysis Using Diffusion Indexes: An Application to Japan," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 37(3), pages 517-538, June.
    6. Kilian, Lutz & Taylor, Mark P., 2003. "Why is it so difficult to beat the random walk forecast of exchange rates?," Journal of International Economics, Elsevier, vol. 60(1), pages 85-107, May.
    7. Rossi, Barbara, 2005. "Optimal Tests For Nested Model Selection With Underlying Parameter Instability," Econometric Theory, Cambridge University Press, vol. 21(5), pages 962-990, October.
    8. Cheung, Yin-Wong & Chinn, Menzie D. & Pascual, Antonio Garcia, 2005. "Empirical exchange rate models of the nineties: Are any fit to survive?," Journal of International Money and Finance, Elsevier, vol. 24(7), pages 1150-1175, November.
    9. Jeremy Berkowitz & Lutz Kilian, 2000. "Recent developments in bootstrapping time series," Econometric Reviews, Taylor & Francis Journals, vol. 19(1), pages 1-48.
    10. repec:cup:macdyn:v:5:y:2001:i:4:p:598-620 is not listed on IDEAS
    11. Todd E. Clark, 2004. "Can out-of-sample forecast comparisons help prevent overfitting?," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 23(2), pages 115-139.
    12. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.
    13. Hodrick, Robert J, 1992. "Dividend Yields and Expected Stock Returns: Alternative Procedures for Inference and Measurement," The Review of Financial Studies, Society for Financial Studies, vol. 5(3), pages 357-386.
    14. Chevillon, Guillaume & Hendry, David F., 2005. "Non-parametric direct multi-step estimation for forecasting economic processes," International Journal of Forecasting, Elsevier, vol. 21(2), pages 201-218.
    15. Clive Granger & Yongil Jeon, 2004. "Forecasting Performance of Information Criteria with Many Macro Series," Journal of Applied Statistics, Taylor & Francis Journals, vol. 31(10), pages 1227-1240.
    16. West, Kenneth D, 2001. "Tests for Forecast Encompassing When Forecasts Depend on Estimated Regression Parameters," Journal of Business & Economic Statistics, American Statistical Association, vol. 19(1), pages 29-33, January.
    17. Jan J. J. Groen, 1999. "Long horizon predictability of exchange rates: Is it for real?," Empirical Economics, Springer, vol. 24(3), pages 451-469.
    18. Goncalves, Silvia & Kilian, Lutz, 2004. "Bootstrapping autoregressions with conditional heteroskedasticity of unknown form," Journal of Econometrics, Elsevier, vol. 123(1), pages 89-120, November.
    19. Harvey, David I & Leybourne, Stephen J & Newbold, Paul, 1998. "Tests for Forecast Encompassing," Journal of Business & Economic Statistics, American Statistical Association, vol. 16(2), pages 254-259, April.
    20. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    21. Clements, Michael P & Hendry, David F, 1996. "Multi-step Estimation for Forecasting," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 58(4), pages 657-684, November.
    22. Mark, Nelson & Sul, Donggyu, 2002. "Asymptotic Power Advantages of Long-Horizon Regressions," Working Papers 145, Department of Economics, The University of Auckland.
    23. Lutz Kilian, 1998. "Small-Sample Confidence Intervals For Impulse Response Functions," The Review of Economics and Statistics, MIT Press, vol. 80(2), pages 218-230, May.
    24. West, Kenneth D., 2006. "Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 3, pages 99-134, Elsevier.
    25. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    26. Stock, James H. & Watson, Mark W., 1999. "Forecasting inflation," Journal of Monetary Economics, Elsevier, vol. 44(2), pages 293-335, October.
    27. Estrella, Arturo & Hardouvelis, Gikas A, 1991. "The Term Structure as a Predictor of Real Economic Activity," Journal of Finance, American Finance Association, vol. 46(2), pages 555-576, June.
    28. Vuong, Quang H, 1989. "Likelihood Ratio Tests for Model Selection and Non-nested Hypotheses," Econometrica, Econometric Society, vol. 57(2), pages 307-333, March.
    29. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 33(1), pages 125-132.
    30. Massimiliano Marcellino, "undated". "Instability and non-linearity in the EMU," Working Papers 211, IGIER (Innocenzo Gasparini Institute for Economic Research), Bocconi University.
    31. repec:bla:jfinan:v:43:y:1988:i:4:p:933-48 is not listed on IDEAS
    32. Stephen G. Cecchetti, 1995. "Inflation Indicators and Inflation Policy," NBER Chapters, in: NBER Macroeconomics Annual 1995, Volume 10, pages 189-236, National Bureau of Economic Research, Inc.
    33. Hansen, Bruce E., 1992. "Convergence to Stochastic Integrals for Dependent Heterogeneous Processes," Econometric Theory, Cambridge University Press, vol. 8(4), pages 489-500, December.
    34. McCracken, Michael W., 2007. "Asymptotics for out of sample tests of Granger causality," Journal of Econometrics, Elsevier, vol. 140(2), pages 719-752, October.
    35. Arturo Estrella & Anthony P. Rodrigues & Sebastian Schich, 2003. "How Stable is the Predictive Power of the Yield Curve? Evidence from Germany and the United States," The Review of Economics and Statistics, MIT Press, vol. 85(3), pages 629-644, August.
    36. James H. Stock & Mark W.Watson, 2003. "Forecasting Output and Inflation: The Role of Asset Prices," Journal of Economic Literature, American Economic Association, vol. 41(3), pages 788-829, September.
    37. West, Kenneth D & McCracken, Michael W, 1998. "Regression-Based Tests of Predictive Ability," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 817-840, November.
    38. Corradi, Valentina & Swanson, Norman R. & Olivetti, Claudia, 2001. "Predictive ability with cointegrated variables," Journal of Econometrics, Elsevier, vol. 104(2), pages 315-358, September.
    39. Atsushi Inoue & Lutz Kilian, 2005. "In-Sample or Out-of-Sample Tests of Predictability: Which One Should We Use?," Econometric Reviews, Taylor & Francis Journals, vol. 23(4), pages 371-402.
    40. Schorfheide, Frank, 2005. "VAR forecasting under misspecification," Journal of Econometrics, Elsevier, vol. 128(1), pages 99-136, September.
    41. Meese, R. & Rogoff, K., 1988. "Was It Real? The Exchange Rate-Interest Differential Ralation Over The Modern Floating-Rate Period," Working papers 368, Wisconsin Madison - Social Systems.
    42. Mark, Nelson C, 1995. "Exchange Rates and Fundamentals: Evidence on Long-Horizon Predictability," American Economic Review, American Economic Association, vol. 85(1), pages 201-218, March.
    43. Ericsson, Neil R., 1992. "Parameter constancy, mean square forecast errors, and measuring forecast performance: An exposition, extensions, and illustration," Journal of Policy Modeling, Elsevier, vol. 14(4), pages 465-495, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Clark, Todd E. & West, Kenneth D., 2006. "Using out-of-sample mean squared prediction errors to test the martingale difference hypothesis," Journal of Econometrics, Elsevier, vol. 135(1-2), pages 155-186.
    2. Martin D. D. Evans & Richard K. Lyons, 2017. "Meese-Rogoff Redux: Micro-Based Exchange-Rate Forecasting," World Scientific Book Chapters, in: Studies in Foreign Exchange Economics, chapter 11, pages 457-475, World Scientific Publishing Co. Pte. Ltd..
    3. Clark, Todd E. & McCracken, Michael W., 2006. "The Predictive Content of the Output Gap for Inflation: Resolving In-Sample and Out-of-Sample Evidence," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 38(5), pages 1127-1148, August.
    4. Mehmet Balcilar & Rangan Gupta & Stephen M. Miller, 2015. "The out-of-sample forecasting performance of nonlinear models of regional housing prices in the US," Applied Economics, Taylor & Francis Journals, vol. 47(22), pages 2259-2277, May.
    5. Valentina Corradi & Norman R. Swanson, 2007. "Nonparametric Bootstrap Procedures For Predictive Inference Based On Recursive Estimation Schemes," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 48(1), pages 67-109, February.
    6. Bessec Marie & Bouabdallah Othman, 2005. "What Causes The Forecasting Failure of Markov-Switching Models? A Monte Carlo Study," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 9(2), pages 1-24, June.
    7. Todd E. Clark & Michael W. McCracken, 2002. "Forecast-based model selection in the presence of structural breaks," Research Working Paper RWP 02-05, Federal Reserve Bank of Kansas City.
    8. Nelson C. Mark & Donggyu Sul, 2004. "The Use of Predictive Regressions at Alternative Horizons in Finance and Economics," Finance 0409032, University Library of Munich, Germany.
    9. Clark, Todd E. & McCracken, Michael W., 2005. "The power of tests of predictive ability in the presence of structural breaks," Journal of Econometrics, Elsevier, vol. 124(1), pages 1-31, January.
    10. Alain Kabundi & Eliphas Ndou & Nombulelo Gumata, 2013. "Important Channels of Transmission Monetary Policy Shock in South Africa," Working Papers 375, Economic Research Southern Africa.
    11. Rapach, David E. & Wohar, Mark E. & Rangvid, Jesper, 2005. "Macro variables and international stock return predictability," International Journal of Forecasting, Elsevier, vol. 21(1), pages 137-166.
    12. Moser, Gabriel & Rumler, Fabio & Scharler, Johann, 2007. "Forecasting Austrian inflation," Economic Modelling, Elsevier, vol. 24(3), pages 470-480, May.
    13. O. De Bandt & E. Michaux & C. Bruneau & A. Flageollet, 2007. "Forecasting inflation using economic indicators: the case of France," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 26(1), pages 1-22.
    14. Michael Steiner, 2009. "Predicting premiums for the market, size, value, and momentum factors," Financial Markets and Portfolio Management, Springer;Swiss Society for Financial Market Research, vol. 23(2), pages 137-155, June.
    15. Rapach, David E. & Wohar, Mark E., 2006. "In-sample vs. out-of-sample tests of stock return predictability in the context of data mining," Journal of Empirical Finance, Elsevier, vol. 13(2), pages 231-247, March.
    16. Orphanides, Athanasios & van Norden, Simon, 2005. "The Reliability of Inflation Forecasts Based on Output Gap Estimates in Real Time," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 37(3), pages 583-601, June.
    17. Barbara Rossi, 2007. "Expectations hypotheses tests at Long Horizons," Econometrics Journal, Royal Economic Society, vol. 10(3), pages 554-579, November.
    18. Todd E. Clark & Kenneth D. West, 2005. "Using Out-of-Sample Mean Squared Prediction Errors to Test the Martingale Difference," NBER Technical Working Papers 0305, National Bureau of Economic Research, Inc.
    19. Boucher, Christophe, 2006. "Stock prices-inflation puzzle and the predictability of stock market returns," Economics Letters, Elsevier, vol. 90(2), pages 205-212, February.
    20. McCracken, Michael W., 2007. "Asymptotics for out of sample tests of Granger causality," Journal of Econometrics, Elsevier, vol. 140(2), pages 719-752, October.
    21. Rapach, David E. & Wohar, Mark E., 2006. "The out-of-sample forecasting performance of nonlinear models of real exchange rate behavior," International Journal of Forecasting, Elsevier, vol. 22(2), pages 341-361.
    22. Mestre, Ricardo, 2007. "Are survey-based inflation expections in the euro area informative?," Working Paper Series 721, European Central Bank.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    2. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.
    3. Todd E. Clark & Michael W. McCracken, 2002. "Forecast-based model selection in the presence of structural breaks," Research Working Paper RWP 02-05, Federal Reserve Bank of Kansas City.
    4. Todd E. Clark & Kenneth D. West, 2005. "Using Out-of-Sample Mean Squared Prediction Errors to Test the Martingale Difference," NBER Technical Working Papers 0305, National Bureau of Economic Research, Inc.
    5. Clark, Todd E. & McCracken, Michael W., 2006. "The Predictive Content of the Output Gap for Inflation: Resolving In-Sample and Out-of-Sample Evidence," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 38(5), pages 1127-1148, August.
    6. Clark, Todd E. & West, Kenneth D., 2006. "Using out-of-sample mean squared prediction errors to test the martingale difference hypothesis," Journal of Econometrics, Elsevier, vol. 135(1-2), pages 155-186.
    7. Clark, Todd E. & West, Kenneth D., 2007. "Approximately normal tests for equal predictive accuracy in nested models," Journal of Econometrics, Elsevier, vol. 138(1), pages 291-311, May.
    8. Atsushi Inoue & Lutz Kilian, 2005. "In-Sample or Out-of-Sample Tests of Predictability: Which One Should We Use?," Econometric Reviews, Taylor & Francis Journals, vol. 23(4), pages 371-402.
    9. Todd E. Clark & Michael W. McCracken, 2010. "Reality checks and nested forecast model comparisons," Working Papers 2010-032, Federal Reserve Bank of St. Louis.
    10. West, Kenneth D., 2006. "Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 3, pages 99-134, Elsevier.
    11. Atsushi Inoue & Lutz Kilian, 2005. "In-Sample or Out-of-Sample Tests of Predictability: Which One Should We Use?," Econometric Reviews, Taylor & Francis Journals, vol. 23(4), pages 371-402.
    12. Clark, Todd E. & McCracken, Michael W., 2005. "The power of tests of predictive ability in the presence of structural breaks," Journal of Econometrics, Elsevier, vol. 124(1), pages 1-31, January.
    13. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    14. Kenneth S. Rogoff & Vania Stavrakeva, 2008. "The Continuing Puzzle of Short Horizon Exchange Rate Forecasting," NBER Working Papers 14071, National Bureau of Economic Research, Inc.
    15. McCracken, Michael W., 2007. "Asymptotics for out of sample tests of Granger causality," Journal of Econometrics, Elsevier, vol. 140(2), pages 719-752, October.
    16. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    17. Clark, Todd E. & McCracken, Michael W., 2009. "Tests of Equal Predictive Ability With Real-Time Data," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(4), pages 441-454.
    18. Granziera, Eleonora & Hubrich, Kirstin & Moon, Hyungsik Roger, 2014. "A predictability test for a small number of nested models," Journal of Econometrics, Elsevier, vol. 182(1), pages 174-185.
    19. Barbara Rossi, 2013. "Exchange Rate Predictability," Journal of Economic Literature, American Economic Association, vol. 51(4), pages 1063-1119, December.
    20. Clark, Todd E. & McCracken, Michael W., 2015. "Nested forecast model comparisons: A new approach to testing equal accuracy," Journal of Econometrics, Elsevier, vol. 186(1), pages 160-177.

    More about this item

    Keywords

    Forecasting;

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fip:fedkrw:rwp01-14. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Zach Kastens (email available below). General contact details of provider: https://edirc.repec.org/data/frbkcus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.