IDEAS home Printed from https://ideas.repec.org/p/uwa/wpaper/20-06.html
   My bibliography  Save this paper

On bootstrapping tests of equal forecast accuracy for nested models

Author

Listed:
  • Firmin Doko Tchatoka

    (School of Economics, The University of Adelaide)

  • Qazi Haque

    (Economics Discipline, Business School, University of Western Australia and Centre for Applied Macroeconomic Analysis, Australian National University)

Abstract

The asymptotic distributions of the recursive out‐of‐sample forecast accuracy test statistics depend on stochastic integrals of Brownian motion when the models under comparison are nested. This often complicates their implementation in practice because the computation of their asymptotic critical values is burdensome. Hansen and Timmermann (2015, Econometrica) propose a Wald approximation of the commonly used recursive F‐statistic and provide a simple characterization of the exact density of its asymptotic distribution. However, this characterization holds only when the larger model has one extra predictor or the forecast errors are homoscedastic. No such closed‐form characterization is readily available when the nesting involves more than one predictor and heteroscedasticity or serial correlation is present. We first show through Monte Carlo experiments that both the recursive F‐test and its Wald approximation have poor finite‐sample properties, especially when the forecast horizon is greater than one and forecast errors exhibit serial correlation. We then propose a hybrid bootstrap method consisting of a moving block bootstrap and a residual‐based bootstrap for both statistics and establish its validity. Simulations show that the hybrid bootstrap has good finite‐sample performance, even in multi‐step ahead forecasts with more than one predictor, and with heteroscedastic or autocorrelated forecast errors. The bootstrap method is illustrated on forecasting core inflation and GDP growth.
(This abstract was borrowed from another version of this item.)

Suggested Citation

  • Firmin Doko Tchatoka & Qazi Haque, 2020. "On bootstrapping tests of equal forecast accuracy for nested models," Economics Discussion / Working Papers 20-06, The University of Western Australia, Department of Economics.
    • Handle: RePEc:uwa:wpaper:20-06
    Note: MD5 = 3c916a03409aa5a951e956212eca18e2
    as

    Download full text from publisher

    File URL: https://ecompapers.biz.uwa.edu.au/paper/PDF%20of%20Discussion%20Papers/2020/DP%2020.06_Tchatoka%20and%20Haque.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Clark, Todd E. & McCracken, Michael W., 2015. "Nested forecast model comparisons: A new approach to testing equal accuracy," Journal of Econometrics, Elsevier, vol. 186(1), pages 160-177.
    2. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    3. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.
    4. Davidson, James, 1994. "Stochastic Limit Theory: An Introduction for Econometricians," OUP Catalogue, Oxford University Press, number 9780198774037, Decembrie.
    5. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    6. Whitney K. Newey & Kenneth D. West, 1994. "Automatic Lag Selection in Covariance Matrix Estimation," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 61(4), pages 631-653.
    7. Peter Reinhard Hansen & Allan Timmermann, 2012. "Choice of Sample Split in Out-of-Sample Forecast Evaluation," CREATES Research Papers 2012-43, Department of Economics and Business Economics, Aarhus University.
    8. Fitzenberger, Bernd, 1998. "The moving blocks bootstrap and robust inference for linear least squares and quantile regressions," Journal of Econometrics, Elsevier, vol. 82(2), pages 235-287, February.
    9. Todd Clark & Michael McCracken, 2012. "Reality Checks and Comparisons of Nested Predictive Models," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(1), pages 53-66.
    10. Todd Clark & Michael McCracken, 2005. "Evaluating Direct Multistep Forecasts," Econometric Reviews, Taylor & Francis Journals, vol. 24(4), pages 369-404.
    11. Peter Reinhard Hansen & Allan Timmermann, 2015. "Equivalence Between Out‐of‐Sample Forecast Comparisons and Wald Statistics," Econometrica, Econometric Society, vol. 83, pages 2485-2505, November.
    12. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    13. Andrews, Donald W K, 1991. "Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation," Econometrica, Econometric Society, vol. 59(3), pages 817-858, May.
    14. Todd E. Clark & Michael W. Mccracken, 2014. "Tests Of Equal Forecast Accuracy For Overlapping Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(3), pages 415-430, April.
    15. Clark, Todd E. & West, Kenneth D., 2007. "Approximately normal tests for equal predictive accuracy in nested models," Journal of Econometrics, Elsevier, vol. 138(1), pages 291-311, May.
    16. James H. Stock & Mark W.Watson, 2003. "Forecasting Output and Inflation: The Role of Asset Prices," Journal of Economic Literature, American Economic Association, vol. 41(3), pages 788-829, September.
    17. Barbara Rossi & Atsushi Inoue, 2012. "Out-of-Sample Forecast Tests Robust to the Choice of Window Size," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(3), pages 432-453, April.
    18. Andrews, Donald W K & Monahan, J Christopher, 1992. "An Improved Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimator," Econometrica, Econometric Society, vol. 60(4), pages 953-966, July.
    19. Lai, T. L. & Wei, C. Z., 1983. "Asymptotic properties of general autoregressive models and strong consistency of least-squares estimates of their parameters," Journal of Multivariate Analysis, Elsevier, vol. 13(1), pages 1-23, March.
    20. Valentina Corradi & Norman R. Swanson, 2007. "Nonparametric Bootstrap Procedures For Predictive Inference Based On Recursive Estimation Schemes," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 48(1), pages 67-109, February.
    21. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 33(1), pages 125-132.
    22. McCracken, Michael W., 2007. "Asymptotics for out of sample tests of Granger causality," Journal of Econometrics, Elsevier, vol. 140(2), pages 719-752, October.
    23. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    24. Jansson, Michael, 2002. "Consistent Covariance Matrix Estimation For Linear Processes," Econometric Theory, Cambridge University Press, vol. 18(6), pages 1449-1459, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    2. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    3. Granziera, Eleonora & Hubrich, Kirstin & Moon, Hyungsik Roger, 2014. "A predictability test for a small number of nested models," Journal of Econometrics, Elsevier, vol. 182(1), pages 174-185.
    4. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    5. Busetti, Fabio & Marcucci, Juri, 2013. "Comparing forecast accuracy: A Monte Carlo investigation," International Journal of Forecasting, Elsevier, vol. 29(1), pages 13-27.
    6. Li, Jia & Patton, Andrew J., 2018. "Asymptotic inference about predictive accuracy using high frequency data," Journal of Econometrics, Elsevier, vol. 203(2), pages 223-240.
    7. Todd E. Clark & Michael W. McCracken, 2010. "Reality checks and nested forecast model comparisons," Working Papers 2010-032, Federal Reserve Bank of St. Louis.
    8. Calhoun, Gray, 2014. "Out-Of-Sample Comparisons of Overfit Models," Staff General Research Papers Archive 32462, Iowa State University, Department of Economics.
    9. Barbara Rossi & Atsushi Inoue, 2012. "Out-of-Sample Forecast Tests Robust to the Choice of Window Size," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(3), pages 432-453, April.
    10. Todd E. Clark & Michael W. McCracken, 2010. "Testing for unconditional predictive ability," Working Papers 2010-031, Federal Reserve Bank of St. Louis.
    11. Clark, Todd E. & West, Kenneth D., 2007. "Approximately normal tests for equal predictive accuracy in nested models," Journal of Econometrics, Elsevier, vol. 138(1), pages 291-311, May.
    12. Mayer, Walter J. & Liu, Feng & Dang, Xin, 2017. "Improving the power of the Diebold–Mariano–West test for least squares predictions," International Journal of Forecasting, Elsevier, vol. 33(3), pages 618-626.
    13. West, Kenneth D., 2006. "Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 3, pages 99-134, Elsevier.
    14. Clark, Todd E. & McCracken, Michael W., 2015. "Nested forecast model comparisons: A new approach to testing equal accuracy," Journal of Econometrics, Elsevier, vol. 186(1), pages 160-177.
    15. Pincheira, Pablo & Hardy, Nicolas, 2022. "Correlation Based Tests of Predictability," MPRA Paper 112014, University Library of Munich, Germany.
    16. Daniel Borup & Jonas N. Eriksen & Mads M. Kjær & Martin Thyrsgaard, 2020. "Predicting bond return predictability," CREATES Research Papers 2020-09, Department of Economics and Business Economics, Aarhus University.
    17. Pablo Pincheira Brown, 2022. "A Power Booster Factor for Out-of-Sample Tests of Predictability," Revista Economía, Fondo Editorial - Pontificia Universidad Católica del Perú, vol. 45(89), pages 150-183.
    18. Rapach, David & Zhou, Guofu, 2013. "Forecasting Stock Returns," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 328-383, Elsevier.
    19. Mariano, Roberto S. & Preve, Daniel, 2012. "Statistical tests for multiple forecast comparison," Journal of Econometrics, Elsevier, vol. 169(1), pages 123-130.
    20. Ahmed, Shamim & Liu, Xiaoquan & Valente, Giorgio, 2016. "Can currency-based risk factors help forecast exchange rates?," International Journal of Forecasting, Elsevier, vol. 32(1), pages 75-97.

    More about this item

    Keywords

    Out-of-sample forecasts; HAC estimator; Moving block bootstrap; Bootstrap consistency;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General
    • C32 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes; State Space Models

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:uwa:wpaper:20-06. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sam Tang (email available below). General contact details of provider: https://edirc.repec.org/data/deuwaau.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.