IDEAS home Printed from https://ideas.repec.org/p/ssb/dispap/931.html

Equal predictability test for multi-step-ahead system forecasts invariant to linear transformations

Author

Listed:

Abstract

The paper derives a test for equal predictability of multi-step-ahead system forecasts that is invariant to linear transformations. The test is a multivariate version of the Diebold-Mariano test. An invariant metric for multi-step-ahead system forecasts is necessary as the conclusions otherwise can depend on how the forecasts are reported (e.g., as in levels or differences; or log-levels or growth rates). The test is used in comparing quarterly multi-step-ahead system forecasts made by Statistics Norway with similar forecasts made by Norges Bank.

Suggested Citation

  • Håvard Hungnes, 2020. "Equal predictability test for multi-step-ahead system forecasts invariant to linear transformations," Discussion Papers 931, Statistics Norway, Research Department.
    • Handle: RePEc:ssb:dispap:931
    as

    Download full text from publisher

    File URL: https://www.ssb.no/en/forskning/discussion-papers/_attachment/422355
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.
    2. Sinclair, Tara M. & Stekler, H.O., 2013. "Examining the quality of early GDP component estimates," International Journal of Forecasting, Elsevier, vol. 29(4), pages 736-750.
    3. Lawrence, Michael & Goodwin, Paul & O'Connor, Marcus & Onkal, Dilek, 2006. "Judgmental forecasting: A review of progress over the last 25 years," International Journal of Forecasting, Elsevier, vol. 22(3), pages 493-518.
    4. Granger, C. W. J. & Newbold, Paul, 1986. "Forecasting Economic Time Series," Elsevier Monographs, Elsevier, edition 2, number 9780122951831 edited by Shell, Karl.
    5. Clark, Todd E. & McCracken, Michael W., 2015. "Nested forecast model comparisons: A new approach to testing equal accuracy," Journal of Econometrics, Elsevier, vol. 186(1), pages 160-177.
    6. Dag Kolsrud, 2015. "A Time‐Simultaneous Prediction Box for a Multivariate Time Series," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 34(8), pages 675-693, December.
    7. Mariano, Roberto S. & Preve, Daniel, 2012. "Statistical tests for multiple forecast comparison," Journal of Econometrics, Elsevier, vol. 169(1), pages 123-130.
    8. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    9. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    10. Òscar Jordà & Massimiliano Marcellino, 2010. "Path forecast evaluation," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 635-662.
    11. Harvey, David & Leybourne, Stephen & Newbold, Paul, 1997. "Testing the equality of prediction mean squared errors," International Journal of Forecasting, Elsevier, vol. 13(2), pages 281-291, June.
    12. Andrew Martinez, 2017. "Testing for Differences in Path Forecast Accuracy: Forecast-Error Dynamics Matter," Working Papers (Old Series) 1717, Federal Reserve Bank of Cleveland.
    13. Tara M. Sinclair & H. O. Stekler & Warren Carnow, 2012. "A new approach for evaluating economic forecasts," Economics Bulletin, AccessEcon, vol. 32(3), pages 2332-2342.
    14. David Harvey & Paul Newbold, 2000. "Tests for multiple forecast encompassing," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 15(5), pages 471-482.
    15. Francis X. Diebold, 2015. "Comparing Predictive Accuracy, Twenty Years Later: A Personal Perspective on the Use and Abuse of Diebold-Mariano Tests," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 33(1), pages 1-1, January.
    16. Magnus Kvåle Helliesen & Håvard Hungnes & Terje Skjerpen, 2022. "Revisions in the Norwegian National Accounts: accuracy, unbiasedness and efficiency in preliminary figures," Empirical Economics, Springer, vol. 62(3), pages 1079-1121, March.
    17. Ericsson Neil R., 2008. "Comment on "Economic Forecasting in a Changing World" (by Michael Clements and David Hendry)," Capitalism and Society, De Gruyter, vol. 3(2), pages 1-18, October.
    18. Harvey, David I. & Leybourne, Stephen J. & Whitehouse, Emily J., 2017. "Forecast evaluation tests and negative long-run variance estimates in small samples," International Journal of Forecasting, Elsevier, vol. 33(4), pages 833-847.
    19. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    20. Oberhofer, W & Kmenta, J, 1974. "A General Procedure for Obtaining Maximum Likelihood Estimates in Generalized Regression Models," Econometrica, Econometric Society, vol. 42(3), pages 579-590, May.
    21. Bjørnland, Hilde C. & Ravazzolo, Francesco & Thorsrud, Leif Anders, 2017. "Forecasting GDP with global components: This time is different," International Journal of Forecasting, Elsevier, vol. 33(1), pages 153-173.
    22. Clements,Michael & Hendry,David, 1998. "Forecasting Economic Time Series," Cambridge Books, Cambridge University Press, number 9780521632423, November.
    23. G. Elliott & C. Granger & A. Timmermann (ed.), 2013. "Handbook of Economic Forecasting," Handbook of Economic Forecasting, Elsevier, edition 1, volume 2, number 2.
    24. Harvey, David I & Leybourne, Stephen J & Newbold, Paul, 1998. "Tests for Forecast Encompassing," Journal of Business & Economic Statistics, American Statistical Association, vol. 16(2), pages 254-259, April.
    25. Håvard Hungnes, 2018. "Encompassing tests for evaluating multi-step system forecasts invariant to linear transformations," Discussion Papers 871, Statistics Norway, Research Department.
    26. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    27. Hansen, Peter Reinhard, 2005. "A Test for Superior Predictive Ability," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 365-380, October.
    28. Hilde C. Bjørnland & Karsten Gerdrup & Anne Sofie Jore & Christie Smith & Leif Anders Thorsrud, 2012. "Does Forecast Combination Improve Norges Bank Inflation Forecasts?," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 74(2), pages 163-179, April.
    29. Anders Bredahl Kock & Timo Teräsvirta, 2016. "Forecasting Macroeconomic Variables Using Neural Network Models and Three Automated Model Selection Techniques," Econometric Reviews, Taylor & Francis Journals, vol. 35(8-10), pages 1753-1779, December.
    30. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    31. Andre Jungmittag, 2016. "Combination of Forecasts across Estimation Windows: An Application to Air Travel Demand," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 35(4), pages 373-380, July.
    32. Hendry, David F. & Martinez, Andrew B., 2017. "Evaluating multi-step system forecasts with relatively few forecast-error observations," International Journal of Forecasting, Elsevier, vol. 33(2), pages 359-372.
    33. Dag Kolsrud, 2007. "Time-simultaneous prediction band for a time series," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 26(3), pages 171-188.
    34. El-Shagi, Makram & Giesen, Sebastian & Jung, Alexander, 2016. "Revisiting the relative forecast performances of Fed staff and private forecasters: A dynamic approach," International Journal of Forecasting, Elsevier, vol. 32(2), pages 313-323.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Håvard Hungnes, 2020. "Predicting the exchange rate path. The importance of using up-to-date observations in the forecasts," Discussion Papers 934, Statistics Norway, Research Department.
    2. Magnus Kvåle Helliesen & Håvard Hungnes & Terje Skjerpen, 2022. "Revisions in the Norwegian National Accounts: accuracy, unbiasedness and efficiency in preliminary figures," Empirical Economics, Springer, vol. 62(3), pages 1079-1121, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Håvard Hungnes, 2018. "Encompassing tests for evaluating multi-step system forecasts invariant to linear transformations," Discussion Papers 871, Statistics Norway, Research Department.
    2. Håvard Hungnes, 2020. "Predicting the exchange rate path. The importance of using up-to-date observations in the forecasts," Discussion Papers 934, Statistics Norway, Research Department.
    3. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    4. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    5. Busetti, Fabio & Marcucci, Juri, 2013. "Comparing forecast accuracy: A Monte Carlo investigation," International Journal of Forecasting, Elsevier, vol. 29(1), pages 13-27.
    6. Laura Coroneo & Fabrizio Iacone, 2020. "Comparing predictive accuracy in small samples using fixed‐smoothing asymptotics," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(4), pages 391-409, June.
    7. Jin, Sainan & Corradi, Valentina & Swanson, Norman R., 2017. "Robust Forecast Comparison," Econometric Theory, Cambridge University Press, vol. 33(6), pages 1306-1351, December.
    8. Andrew Martinez, 2017. "Testing for Differences in Path Forecast Accuracy: Forecast-Error Dynamics Matter," Working Papers (Old Series) 1717, Federal Reserve Bank of Cleveland.
    9. Barbara Rossi & Atsushi Inoue, 2012. "Out-of-Sample Forecast Tests Robust to the Choice of Window Size," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(3), pages 432-453, April.
    10. Harvey, David I. & Leybourne, Stephen J. & Whitehouse, Emily J., 2017. "Forecast evaluation tests and negative long-run variance estimates in small samples," International Journal of Forecasting, Elsevier, vol. 33(4), pages 833-847.
    11. Barbara Rossi, 2021. "Forecasting in the Presence of Instabilities: How We Know Whether Models Predict Well and How to Improve Them," Journal of Economic Literature, American Economic Association, vol. 59(4), pages 1135-1190, December.
    12. Saša ŽIKOVIÆ & Randall K. FILER, 2013. "Ranking of VaR and ES Models: Performance in Developed and Emerging Markets," Czech Journal of Economics and Finance (Finance a uver), Charles University Prague, Faculty of Social Sciences, vol. 63(4), pages 327-359, August.
    13. Mariano, Roberto S. & Preve, Daniel, 2012. "Statistical tests for multiple forecast comparison," Journal of Econometrics, Elsevier, vol. 169(1), pages 123-130.
    14. Graham Elliott & Allan Timmermann, 2016. "Forecasting in Economics and Finance," Annual Review of Economics, Annual Reviews, vol. 8(1), pages 81-110, October.
    15. Todd E. Clark & Michael W. McCracken, 2010. "Reality checks and nested forecast model comparisons," Working Papers 2010-032, Federal Reserve Bank of St. Louis.
    16. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    17. Dichtl, Hubert & Drobetz, Wolfgang & Neuhierl, Andreas & Wendt, Viktoria-Sophie, 2021. "Data snooping in equity premium prediction," International Journal of Forecasting, Elsevier, vol. 37(1), pages 72-94.
    18. Mark E. Wohar & David E. Rapach, 2007. "Forecasting the recent behavior of US business fixed investment spending: an analysis of competing models This is a significantly revised version of our previous paper, 'Forecasting US Business Fixed Investment Spending'. The results reported in this," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 26(1), pages 33-51.
    19. Valentina Corradi & Norman Swanson, 2013. "A Survey of Recent Advances in Forecast Accuracy Comparison Testing, with an Extension to Stochastic Dominance," Departmental Working Papers 201309, Rutgers University, Department of Economics.
    20. Marian Vavra, 2015. "On a Bootstrap Test for Forecast Evaluations," Working and Discussion Papers WP 5/2015, Research Department, National Bank of Slovakia.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    JEL classification:

    • C32 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes; State Space Models
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ssb:dispap:931. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: L Maasø (email available below). General contact details of provider: https://edirc.repec.org/data/ssbgvno.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.