IDEAS home Printed from https://ideas.repec.org/
MyIDEAS: Login to save this paper or follow this series

Tests of conditional predictive ability

  • Raffaella Giacomini

    ()

    (Boston College)

  • Halbert White

    (University of California, San Diego)

We argue that the current framework for predictive ability testing (e.g.,West, 1996) is not necessarily useful for real-time forecast selection, i.e., for assessing which of two competing forecasting methods will perform better in the future. We propose an alternative framework for out-of-sample comparison of predictive ability which delivers more practically relevant conclusions. Our approach is based on inference about conditional expectations of forecasts and forecast errors rather than the unconditional expectations that are the focus of the existing literature. We capture important determinants of forecast performance that are neglected in the existing literature by evaluating what we call the forecasting method (the model and the parameter estimation procedure), rather than just the forecasting model. Compared to previous approaches, our tests are valid under more general data assumptions (heterogeneity rather than stationarity) and estimation methods, and they can handle comparison of both nested and non-nested models, which is not currently possible. To illustrate the usefulness of the proposed tests, we compare the forecast performance of three leading parameter-reduction methods for macroeconomic forecasting using a large number of predictors: a sequential model selection approach, the "diffusion indexes" approach of Stock and Watson (2002), and the use of Bayesian shrinkage estimators.

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

File URL: http://fmwww.bc.edu/EC-P/wp572.pdf
File Function: main text
Download Restriction: no

Paper provided by Boston College Department of Economics in its series Boston College Working Papers in Economics with number 572.

as
in new window

Length: 38 pages
Date of creation: 01 Apr 2003
Date of revision:
Handle: RePEc:boc:bocoec:572
Contact details of provider: Postal: Boston College, 140 Commonwealth Avenue, Chestnut Hill MA 02467 USA
Phone: 617-552-3670
Fax: +1-617-552-2308
Web page: http://fmwww.bc.edu/EC/
Email:


More information through EDIRC

References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:

as in new window
  1. Donald W.K. Andrews, 1999. "Higher-Order Improvements of a Computationally Attractive-Step Bootstrap for Extremum Estimators," Cowles Foundation Discussion Papers 1230R, Cowles Foundation for Research in Economics, Yale University, revised Jan 2001.
  2. Mc Cracken, Michael W., 2000. "Robust out-of-sample inference," Journal of Econometrics, Elsevier, vol. 99(2), pages 195-223, December.
  3. Jose A. Lopez, 1995. "Evaluating the predictive accuracy of volatility models," Research Paper 9524, Federal Reserve Bank of New York.
  4. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-44, January.
  5. Andrews, Donald W K, 1991. "Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation," Econometrica, Econometric Society, vol. 59(3), pages 817-58, May.
  6. Michael P. Clements & David F. Hendry, 2001. "Forecasting Non-Stationary Economic Time Series," MIT Press Books, The MIT Press, edition 1, volume 1, number 0262531895, June.
  7. Raffaella Giacomini, 2002. "Comparing Density Forecasts via Weighted Likelihood Ratio Tests: Asymptotic and Bootstrap Methods," Boston College Working Papers in Economics 583, Boston College Department of Economics.
  8. Francis X. Diebold & Jose A. Lopez, 1996. "Forecast Evaluation and Combination," NBER Technical Working Papers 0192, National Bureau of Economic Research, Inc.
  9. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.
  10. Kevin Hoover & Stephen J. Perez, 2003. "Data Mining Reconsidered: Encompassing And The General-To-Specific Approach To Specification Search," Working Papers 9727, University of California, Davis, Department of Economics.
  11. Inoue, Atsushi & Kilian, Lutz, 2002. "In-Sample or Out-of-Sample Tests of Predictability: Which One Should We Use?," CEPR Discussion Papers 3671, C.E.P.R. Discussion Papers.
  12. Kenneth D. West & Michael W. McCracken, 1998. "Regression-Based Tests of Predictive Ability," NBER Technical Working Papers 0226, National Bureau of Economic Research, Inc.
  13. Giacomini, Raffaella & Komunjer, Ivana, 2002. "Evaluation and Combination of Conditional Quantile Forecasts," University of California at San Diego, Economics Working Paper Series qt4n99t4wz, Department of Economics, UC San Diego.
  14. Corradi, Valentina & Swanson, Norman R. & Olivetti, Claudia, 2001. "Predictive ability with cointegrated variables," Journal of Econometrics, Elsevier, vol. 104(2), pages 315-358, September.
  15. Fisher, Franklin M, 1970. "Tests of Equality Between Sets of Coefficients in Two Linear Regressions: An Expository Note," Econometrica, Econometric Society, vol. 38(2), pages 361-66, March.
  16. West, Kenneth D. & Edison, Hali J. & Cho, Dongchul, 1993. "A utility-based comparison of some models of exchange rate volatility," Journal of International Economics, Elsevier, vol. 35(1-2), pages 23-45, August.
  17. Stock, James H & Watson, Mark W, 2002. "Macroeconomic Forecasting Using Diffusion Indexes," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(2), pages 147-62, April.
  18. Neil R. Ericsson, 2000. "Predictable uncertainty in economic forecasting," International Finance Discussion Papers 695, Board of Governors of the Federal Reserve System (U.S.).
  19. Whitney K. Newey & Kenneth D. West, 1986. "A Simple, Positive Semi-Definite, Heteroskedasticity and AutocorrelationConsistent Covariance Matrix," NBER Technical Working Papers 0055, National Bureau of Economic Research, Inc.
  20. Bierens, Herman J, 1990. "A Consistent Conditional Moment Test of Functional Form," Econometrica, Econometric Society, vol. 58(6), pages 1443-58, November.
  21. White, Halbert & Domowitz, Ian, 1984. "Nonlinear Regression with Dependent Observations," Econometrica, Econometric Society, vol. 52(1), pages 143-61, January.
  22. McCulloch, Robert & Rossi, Peter E., 1990. "Posterior, predictive, and utility-based approaches to testing the arbitrage pricing theory," Journal of Financial Economics, Elsevier, vol. 28(1-2), pages 7-38.
  23. Robert B. Litterman, 1985. "Forecasting with Bayesian vector autoregressions five years of experience," Working Papers 274, Federal Reserve Bank of Minneapolis.
  24. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-84, September.
  25. repec:cup:cbooks:9780521634809 is not listed on IDEAS
  26. Stinchcombe, Maxwell B. & White, Halbert, 1998. "Consistent Specification Testing With Nuisance Parameters Present Only Under The Alternative," Econometric Theory, Cambridge University Press, vol. 14(03), pages 295-325, June.
  27. Chao, John & Corradi, Valentina & Swanson, Norman R., 2001. "Out-Of-Sample Tests For Granger Causality," Macroeconomic Dynamics, Cambridge University Press, vol. 5(04), pages 598-620, September.
  28. Fama, Eugene F & MacBeth, James D, 1973. "Risk, Return, and Equilibrium: Empirical Tests," Journal of Political Economy, University of Chicago Press, vol. 81(3), pages 607-36, May-June.
  29. Sims, Christopher A & Zha, Tao, 1998. "Bayesian Methods for Dynamic Multivariate Models," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 949-68, November.
  30. repec:cup:cbooks:9780521632423 is not listed on IDEAS
  31. Harvey, David & Leybourne, Stephen & Newbold, Paul, 1997. "Testing the equality of prediction mean squared errors," International Journal of Forecasting, Elsevier, vol. 13(2), pages 281-291, June.
  32. Wooldridge, Jeffrey M. & White, Halbert, 1988. "Some Invariance Principles and Central Limit Theorems for Dependent Heterogeneous Processes," Econometric Theory, Cambridge University Press, vol. 4(02), pages 210-230, August.
  33. Leitch, Gordon & Tanner, J Ernest, 1991. "Economic Forecast Evaluation: Profits versus the Conventional Error Measures," American Economic Review, American Economic Association, vol. 81(3), pages 580-90, June.
  34. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
  35. Gonedes, Nicholas J., 1973. "Evidence on the Information Content of Accounting Numbers: Accounting-based and Market-based Estimates of Systematic Risk," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 8(03), pages 407-443, June.
  36. Norman R. Swanson, 2000. "An Out of Sample Test for Granger Causality," Econometric Society World Congress 2000 Contributed Papers 0362, Econometric Society.
Full references (including those not matched with items on IDEAS)

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

When requesting a correction, please mention this item's handle: RePEc:boc:bocoec:572. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Christopher F Baum)

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.

This information is provided to you by IDEAS at the Research Division of the Federal Reserve Bank of St. Louis using RePEc data.