IDEAS home Printed from https://ideas.repec.org/
MyIDEAS: Login to save this paper or follow this series

Tests of conditional predictive ability

  • Raffaella Giacomini

    ()

    (Boston College)

  • Halbert White

    (University of California, San Diego)

We argue that the current framework for predictive ability testing (e.g.,West, 1996) is not necessarily useful for real-time forecast selection, i.e., for assessing which of two competing forecasting methods will perform better in the future. We propose an alternative framework for out-of-sample comparison of predictive ability which delivers more practically relevant conclusions. Our approach is based on inference about conditional expectations of forecasts and forecast errors rather than the unconditional expectations that are the focus of the existing literature. We capture important determinants of forecast performance that are neglected in the existing literature by evaluating what we call the forecasting method (the model and the parameter estimation procedure), rather than just the forecasting model. Compared to previous approaches, our tests are valid under more general data assumptions (heterogeneity rather than stationarity) and estimation methods, and they can handle comparison of both nested and non-nested models, which is not currently possible. To illustrate the usefulness of the proposed tests, we compare the forecast performance of three leading parameter-reduction methods for macroeconomic forecasting using a large number of predictors: a sequential model selection approach, the "diffusion indexes" approach of Stock and Watson (2002), and the use of Bayesian shrinkage estimators.

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

File URL: http://fmwww.bc.edu/EC-P/wp572.pdf
File Function: main text
Download Restriction: no

Paper provided by Boston College Department of Economics in its series Boston College Working Papers in Economics with number 572.

as
in new window

Length: 38 pages
Date of creation: 01 Apr 2003
Date of revision:
Handle: RePEc:boc:bocoec:572
Contact details of provider: Postal: Boston College, 140 Commonwealth Avenue, Chestnut Hill MA 02467 USA
Phone: 617-552-3670
Fax: +1-617-552-2308
Web page: http://fmwww.bc.edu/EC/
Email:


More information through EDIRC

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:

as in new window
  1. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
  2. Francis X. Diebold & Jose A. Lopez, 1996. "Forecast Evaluation and Combination," NBER Technical Working Papers 0192, National Bureau of Economic Research, Inc.
  3. Donald W.K. Andrews, 1988. "Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation," Cowles Foundation Discussion Papers 877R, Cowles Foundation for Research in Economics, Yale University, revised Jul 1989.
  4. Kenneth D. West & Michael W. McCracken, 1998. "Regression-Based Tests of Predictive Ability," NBER Technical Working Papers 0226, National Bureau of Economic Research, Inc.
  5. Corradi, Valentina & Swanson, Norman R. & Olivetti, Claudia, 2001. "Predictive ability with cointegrated variables," Journal of Econometrics, Elsevier, vol. 104(2), pages 315-358, September.
  6. Harvey, David & Leybourne, Stephen & Newbold, Paul, 1997. "Testing the equality of prediction mean squared errors," International Journal of Forecasting, Elsevier, vol. 13(2), pages 281-291, June.
  7. Diebold, Francis X & Mariano, Roberto S, 1995. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 13(3), pages 253-63, July.
  8. Todd E. Clark & Michael W. McCracken, 2000. "Tests of Equal Forecast Accuracy and Encompassing for Nested Models," Econometric Society World Congress 2000 Contributed Papers 0319, Econometric Society.
  9. Donald W.K. Andrews, 1999. "Higher-Order Improvements of a Computationally Attractive-Step Bootstrap for Extremum Estimators," Cowles Foundation Discussion Papers 1230, Cowles Foundation for Research in Economics, Yale University.
  10. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Publishing House "SINERGIA PRESS", vol. 33(1), pages 125-132.
  11. Leitch, Gordon & Tanner, J Ernest, 1991. "Economic Forecast Evaluation: Profits versus the Conventional Error Measures," American Economic Review, American Economic Association, vol. 81(3), pages 580-90, June.
  12. White, Halbert & Domowitz, Ian, 1984. "Nonlinear Regression with Dependent Observations," Econometrica, Econometric Society, vol. 52(1), pages 143-61, January.
  13. Atsushi Inoue & Lutz Kilian, 2005. "In-Sample or Out-of-Sample Tests of Predictability: Which One Should We Use?," Econometric Reviews, Taylor & Francis Journals, vol. 23(4), pages 371-402.
  14. Kenneth D. West, 1994. "Asymptotic Inference About Predictive Ability," Macroeconomics 9410002, EconWPA.
  15. Christopher A. Sims & Tao Zha, 1996. "Bayesian methods for dynamic multivariate models," Working Paper 96-13, Federal Reserve Bank of Atlanta.
  16. Neil R. Ericsson, 2000. "Predictable uncertainty in economic forecasting," International Finance Discussion Papers 695, Board of Governors of the Federal Reserve System (U.S.).
  17. Stinchcombe, Maxwell B. & White, Halbert, 1998. "Consistent Specification Testing With Nuisance Parameters Present Only Under The Alternative," Econometric Theory, Cambridge University Press, vol. 14(03), pages 295-325, June.
  18. Norman R. Swanson, 2000. "An Out of Sample Test for Granger Causality," Econometric Society World Congress 2000 Contributed Papers 0362, Econometric Society.
  19. Giacomini, Raffaella & Komunjer, Ivana, 2002. "Evaluation and Combination of Conditional Quantile Forecasts," University of California at San Diego, Economics Working Paper Series qt4n99t4wz, Department of Economics, UC San Diego.
  20. Bierens, Herman J, 1990. "A Consistent Conditional Moment Test of Functional Form," Econometrica, Econometric Society, vol. 58(6), pages 1443-58, November.
  21. Mc Cracken, Michael W., 2000. "Robust out-of-sample inference," Journal of Econometrics, Elsevier, vol. 99(2), pages 195-223, December.
  22. Fisher, Franklin M, 1970. "Tests of Equality Between Sets of Coefficients in Two Linear Regressions: An Expository Note," Econometrica, Econometric Society, vol. 38(2), pages 361-66, March.
  23. Lopez, Jose A, 2001. "Evaluating the Predictive Accuracy of Volatility Models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 20(2), pages 87-109, March.
  24. Kenneth D. West & Hali J. Edison & Dongchul Cho, 1993. "A utility based comparison of some models of exchange rate volatility," International Finance Discussion Papers 441, Board of Governors of the Federal Reserve System (U.S.).
  25. Chao, John & Corradi, Valentina & Swanson, Norman R., 2001. "Out-Of-Sample Tests For Granger Causality," Macroeconomic Dynamics, Cambridge University Press, vol. 5(04), pages 598-620, September.
  26. Kevin Hoover & Stephen J. Perez, 2003. "Data Mining Reconsidered: Encompassing And The General-To-Specific Approach To Specification Search," Working Papers 9727, University of California, Davis, Department of Economics.
  27. Litterman, Robert B, 1986. "Forecasting with Bayesian Vector Autoregressions-Five Years of Experience," Journal of Business & Economic Statistics, American Statistical Association, vol. 4(1), pages 25-38, January.
  28. Wooldridge, Jeffrey M. & White, Halbert, 1988. "Some Invariance Principles and Central Limit Theorems for Dependent Heterogeneous Processes," Econometric Theory, Cambridge University Press, vol. 4(02), pages 210-230, August.
  29. repec:cup:cbooks:9780521632423 is not listed on IDEAS
  30. Michael P. Clements & David F. Hendry, 2001. "Forecasting Non-Stationary Economic Time Series," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262531895, June.
  31. Stock, James H & Watson, Mark W, 2002. "Macroeconomic Forecasting Using Diffusion Indexes," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(2), pages 147-62, April.
  32. repec:cup:cbooks:9780521634809 is not listed on IDEAS
  33. Raffaella Giacomini, 2002. "Comparing Density Forecasts via Weighted Likelihood Ratio Tests: Asymptotic and Bootstrap Methods," Boston College Working Papers in Economics 583, Boston College Department of Economics.
  34. Fama, Eugene F & MacBeth, James D, 1973. "Risk, Return, and Equilibrium: Empirical Tests," Journal of Political Economy, University of Chicago Press, vol. 81(3), pages 607-36, May-June.
  35. Gonedes, Nicholas J., 1973. "Evidence on the Information Content of Accounting Numbers: Accounting-based and Market-based Estimates of Systematic Risk," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 8(03), pages 407-443, June.
  36. McCulloch, Robert & Rossi, Peter E., 1990. "Posterior, predictive, and utility-based approaches to testing the arbitrage pricing theory," Journal of Financial Economics, Elsevier, vol. 28(1-2), pages 7-38.
Full references (including those not matched with items on IDEAS)

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

When requesting a correction, please mention this item's handle: RePEc:boc:bocoec:572. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Christopher F Baum)

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.

This information is provided to you by IDEAS at the Research Division of the Federal Reserve Bank of St. Louis using RePEc data.