IDEAS home Printed from https://ideas.repec.org/p/svk/wpaper/1034.html
   My bibliography  Save this paper

On a Bootstrap Test for Forecast Evaluations

Author

Listed:
  • Marian Vavra

    (National Bank of Slovakia, Research Department)

Abstract

This paper is concerned with the problem of testing for the equal forecast accuracy of competing models using a bootstrap-based Diebold-Mariano test statistic. The finite-sample properties of the test are assessed via Monte Carlo experiments. As an illustration, the forecast accuracy of the US Survey of Professional Forecasters is compared to that of an autoregressive model. The empirical results indicate that professionals beat AR models systematically only for a single economic variable – the unemployment rate

Suggested Citation

  • Marian Vavra, 2015. "On a Bootstrap Test for Forecast Evaluations," Working and Discussion Papers WP 5/2015, Research Department, National Bank of Slovakia.
    • Handle: RePEc:svk:wpaper:1034
    as

    Download full text from publisher

    File URL: http://www.nbs.sk/_img/Documents/PUBLIK/WP_5_2015_Vavra_Test_For_Forecast_Evaluations.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Fuertes, Ana-Maria, 2008. "Sieve bootstrap t-tests on long-run average parameters," Computational Statistics & Data Analysis, Elsevier, vol. 52(7), pages 3354-3370, March.
    2. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    3. Tom Engsted & Thomas Q. Pedersen, 2014. "Bias-Correction in Vector Autoregressive Models: A Simulation Study," Econometrics, MDPI, vol. 2(1), pages 1-27, March.
    4. Edwin Choi & Peter Hall, 2000. "Bootstrap confidence regions computed from autoregressions of arbitrary order," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 62(2), pages 461-477.
    5. D. S. Poskitt, 2008. "Properties of the Sieve Bootstrap for Fractionally Integrated and Non‐Invertible Processes," Journal of Time Series Analysis, Wiley Blackwell, vol. 29(2), pages 224-250, March.
    6. George Kapetanios & Zacharias Psaradakis, 2006. "Sieve Bootstrap for Strongly Dependent Stationary Processes," Working Papers 552, Queen Mary University of London, School of Economics and Finance.
    7. Whitney K. Newey & Kenneth D. West, 1994. "Automatic Lag Selection in Covariance Matrix Estimation," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 61(4), pages 631-653.
    8. Silvia Goncalves & Lutz Kilian, 2007. "Asymptotic and Bootstrap Inference for AR(∞) Processes with Conditional Heteroskedasticity," Econometric Reviews, Taylor & Francis Journals, vol. 26(6), pages 609-641.
    9. Kim, Hyeongwoo & Durmaz, Nazif, 2012. "Bias correction and out-of-sample forecast accuracy," International Journal of Forecasting, Elsevier, vol. 28(3), pages 575-586.
    10. Francis X. Diebold & Jose A. Lopez, 1995. "Forecast evaluation and combination," Research Paper 9525, Federal Reserve Bank of New York.
    11. Palm, Franz C. & Smeekes, Stephan & Urbain, Jean-Pierre, 2010. "A Sieve Bootstrap Test For Cointegration In A Conditional Error Correction Model," Econometric Theory, Cambridge University Press, vol. 26(3), pages 647-681, June.
      • Arnold Zellner & Franz C. Palm, 2000. "Correction," Econometrica, Econometric Society, vol. 68(5), pages 1293-1294, September.
    12. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    13. George Kapetanios & Zacharias Psaradakis, 2006. "Sieve Bootstrap for Strongly Dependent Stationary Processes," Working Papers 552, Queen Mary University of London, School of Economics and Finance.
    14. Mariano, Roberto S. & Preve, Daniel, 2012. "Statistical tests for multiple forecast comparison," Journal of Econometrics, Elsevier, vol. 169(1), pages 123-130.
    15. Hansen, Peter Reinhard, 2005. "A Test for Superior Predictive Ability," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 365-380, October.
    16. Yoosoon Chang & Joon Y. Park, 2003. "A Sieve Bootstrap For The Test Of A Unit Root," Journal of Time Series Analysis, Wiley Blackwell, vol. 24(4), pages 379-400, July.
    17. Kapetanios, George & Labhard, Vincent & Price, Simon, 2008. "Forecast combination and the Bank of England's suite of statistical forecasting models," Economic Modelling, Elsevier, vol. 25(4), pages 772-792, July.
    18. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    19. Andrews, Donald W K, 1991. "Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation," Econometrica, Econometric Society, vol. 59(3), pages 817-858, May.
    20. Gonçalves, Sílvia & Vogelsang, Timothy J., 2011. "Block Bootstrap Hac Robust Tests: The Sophistication Of The Naive Bootstrap," Econometric Theory, Cambridge University Press, vol. 27(4), pages 745-791, August.
    21. Andrews, Donald W K & Monahan, J Christopher, 1992. "An Improved Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimator," Econometrica, Econometric Society, vol. 60(4), pages 953-966, July.
    22. Harvey, David & Leybourne, Stephen & Newbold, Paul, 1997. "Testing the equality of prediction mean squared errors," International Journal of Forecasting, Elsevier, vol. 13(2), pages 281-291, June.
    23. Paparoditis, Efstathios, 1996. "Bootstrapping Autoregressive and Moving Average Parameter Estimates of Infinite Order Vector Autoregressive Processes," Journal of Multivariate Analysis, Elsevier, vol. 57(2), pages 277-296, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Psaradakis, Zacharias & Vávra, Marián, 2017. "A distance test of normality for a wide class of stationary processes," Econometrics and Statistics, Elsevier, vol. 2(C), pages 50-60.
    2. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    3. Firmin Doko Tchatoka & Qazi Haque, 2023. "On bootstrapping tests of equal forecast accuracy for nested models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(7), pages 1844-1864, November.
    4. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    5. Şener, Emrah & Baronyan, Sayad & Ali Mengütürk, Levent, 2012. "Ranking the predictive performances of value-at-risk estimation methods," International Journal of Forecasting, Elsevier, vol. 28(4), pages 849-873.
    6. Busetti, Fabio & Marcucci, Juri, 2013. "Comparing forecast accuracy: A Monte Carlo investigation," International Journal of Forecasting, Elsevier, vol. 29(1), pages 13-27.
    7. Margherita Gerolimetto & Stefano Magrini, 2020. "Testing for boundary conditions in case of fractionally integrated processes," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 29(2), pages 357-371, June.
    8. Kruse, Robinson & Leschinski, Christian & Will, Michael, 2016. "Comparing Predictive Accuracy under Long Memory - With an Application to Volatility Forecasting," Hannover Economic Papers (HEP) dp-571, Leibniz Universität Hannover, Wirtschaftswissenschaftliche Fakultät.
    9. Laura Coroneo & Fabrizio Iacone, 2020. "Comparing predictive accuracy in small samples using fixed‐smoothing asymptotics," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(4), pages 391-409, June.
    10. Håvard Hungnes, 2020. "Equal predictability test for multi-step-ahead system forecasts invariant to linear transformations," Discussion Papers 931, Statistics Norway, Research Department.
    11. Kontogeorgos, Georgios & Lambrias, Kyriacos, 2019. "An analysis of the Eurosystem/ECB projections," Working Paper Series 2291, European Central Bank.
    12. Mariano, Roberto S. & Preve, Daniel, 2012. "Statistical tests for multiple forecast comparison," Journal of Econometrics, Elsevier, vol. 169(1), pages 123-130.
    13. Giot, Pierre & Petitjean, Mikael, 2007. "The information content of the Bond-Equity Yield Ratio: Better than a random walk?," International Journal of Forecasting, Elsevier, vol. 23(2), pages 289-305.
    14. Harvey, David I. & Leybourne, Stephen J. & Whitehouse, Emily J., 2017. "Forecast evaluation tests and negative long-run variance estimates in small samples," International Journal of Forecasting, Elsevier, vol. 33(4), pages 833-847.
    15. West, Kenneth D., 2006. "Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 3, pages 99-134, Elsevier.
    16. Halkos, George E. & Tsirivis, Apostolos S., 2019. "Effective energy commodity risk management: Econometric modeling of price volatility," Economic Analysis and Policy, Elsevier, vol. 63(C), pages 234-250.
    17. Thomas A. Knetsch, 2005. "Evaluating the German Inventory Cycle Using Data from the Ifo Business Survey," Contributions to Economics, in: Jan-Egbert Sturm & Timo Wollmershäuser (ed.), Ifo Survey Data in Business Cycle and Monetary Policy Analysis, pages 61-92, Springer.
    18. Hirukawa, Masayuki, 2023. "Robust Covariance Matrix Estimation in Time Series: A Review," Econometrics and Statistics, Elsevier, vol. 27(C), pages 36-61.
    19. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    20. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.

    More about this item

    Keywords

    Forecast evaluation; Diebold-Mariano test; Sieve bootstrap;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General
    • C32 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes; State Space Models
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:svk:wpaper:1034. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nbsgvsk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.