IDEAS home Printed from https://ideas.repec.org/p/fip/fedcwp/1717.html
   My bibliography  Save this paper

Testing for Differences in Path Forecast Accuracy: Forecast-Error Dynamics Matter

Author

Listed:
  • Andrew Martinez

Abstract

Although the trajectory and path of future outcomes plays an important role in policy decisions, analyses of forecast accuracy typically focus on individual point forecasts. However, it is important to examine the path forecasts errors since they include the forecast dynamics. We use the link between path forecast evaluation methods and the joint predictive density to propose a test for differences in system path forecast accuracy. We also demonstrate how our test relates to and extends existing joint testing approaches. Simulations highlight both the advantages and disadvantages of path forecast accuracy tests in detecting a broad range of differences in forecast errors. We compare the Federal Reserve?s Greenbook point and path forecasts against four DSGE model forecasts. The results show that differences in forecast-error dynamics can play an important role in the assessment of forecast accuracy.

Suggested Citation

  • Andrew Martinez, 2017. "Testing for Differences in Path Forecast Accuracy: Forecast-Error Dynamics Matter," Working Papers (Old Series) 1717, Federal Reserve Bank of Cleveland.
  • Handle: RePEc:fip:fedcwp:1717
    DOI: 10.26509/frbc-wp-201717
    as

    Download full text from publisher

    File URL: https://www.clevelandfed.org/en/newsroom-and-events/publications/working-papers/2017-working-papers/wp-1717-testing-for-differences-in-path-forecast-accuracy.aspx
    File Function: Full text
    Download Restriction: no

    File URL: https://libkey.io/10.26509/frbc-wp-201717?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    2. Hendry, David F. & Mizon, Grayham E., 2014. "Unpredictability in economic analysis, econometric modeling and forecasting," Journal of Econometrics, Elsevier, vol. 182(1), pages 186-195.
    3. Stewart, Kenneth G., 1995. "The functional equivalence of the W, LR, and LM statistics," Economics Letters, Elsevier, vol. 49(2), pages 109-112, August.
    4. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.
    5. Ivana Komunjer & Michael T. Owyang, 2012. "Multivariate Forecast Evaluation and Rationality Testing," The Review of Economics and Statistics, MIT Press, vol. 94(4), pages 1066-1080, November.
    6. Hendry, David F. & Martinez, Andrew B., 2017. "Evaluating multi-step system forecasts with relatively few forecast-error observations," International Journal of Forecasting, Elsevier, vol. 33(2), pages 359-372.
    7. Tim Oliver Berg, 2016. "Multivariate Forecasting with BVARs and DSGE Models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 35(8), pages 718-740, December.
    8. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    9. Frank Schorfheide & Dongho Song, 2015. "Real-Time Forecasting With a Mixed-Frequency VAR," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 33(3), pages 366-380, July.
    10. Òscar Jordà & Massimiliano Marcellino, 2010. "Path forecast evaluation," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 635-662.
    11. Peter Reinhard Hansen & Allan Timmermann, 2015. "Equivalence Between Out‐of‐Sample Forecast Comparisons and Wald Statistics," Econometrica, Econometric Society, vol. 83, pages 2485-2505, November.
    12. Maik H. Wolters, 2015. "Evaluating Point and Density Forecasts of DSGE Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 30(1), pages 74-96, January.
    13. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    14. Vuong, Quang H, 1989. "Likelihood Ratio Tests for Model Selection and Non-nested Hypotheses," Econometrica, Econometric Society, vol. 57(2), pages 307-333, March.
    15. Wolters, Maik H., 2011. "Forecasting under Model Uncertainty," VfS Annual Conference 2011 (Frankfurt, Main): The Order of the World Economy - Lessons from the Crisis 48723, Verein für Socialpolitik / German Economic Association.
    16. Clive W.J. Granger, 1999. "Outline of forecast theory using generalized cost functions," Spanish Economic Review, Springer;Spanish Economic Association, vol. 1(2), pages 161-173.
    17. Andrews, Donald W K, 1991. "Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation," Econometrica, Econometric Society, vol. 59(3), pages 817-858, May.
    18. Gelper, Sarah & Croux, Christophe, 2007. "Multivariate out-of-sample tests for Granger causality," Computational Statistics & Data Analysis, Elsevier, vol. 51(7), pages 3319-3329, April.
    19. Yock Y. Chong & David F. Hendry, 1986. "Econometric Evaluation of Linear Macro-Economic Models," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 53(4), pages 671-690.
    20. Clements, Michael P & Hendry, David F, 1995. "Forecasting in Cointegration Systems," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 10(2), pages 127-146, April-Jun.
    21. Dr. James Mitchell, 2005. "Evaluating, comparing and combining density forecasts using the KLIC with an application to the Bank of England and NIESR ÔfanÕ charts of inflation," National Institute of Economic and Social Research (NIESR) Discussion Papers 253, National Institute of Economic and Social Research.
    22. Peter R. Hansen & Asger Lunde & James M. Nason, 2011. "The Model Confidence Set," Econometrica, Econometric Society, vol. 79(2), pages 453-497, March.
    23. Amisano, Gianni & Giacomini, Raffaella, 2007. "Comparing Density Forecasts via Weighted Likelihood Ratio Tests," Journal of Business & Economic Statistics, American Statistical Association, vol. 25, pages 177-190, April.
    24. Faust, Jon & Wright, Jonathan H., 2009. "Comparing Greenbook and Reduced Form Forecasts Using a Large Realtime Dataset," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(4), pages 468-479.
    25. Del Negro, Marco & Schorfheide, Frank & Smets, Frank & Wouters, Rafael, 2007. "On the Fit of New Keynesian Models," Journal of Business & Economic Statistics, American Statistical Association, vol. 25, pages 123-143, April.
    26. Clements,Michael & Hendry,David, 1998. "Forecasting Economic Time Series," Cambridge Books, Cambridge University Press, number 9780521632423.
    27. Chen, Songxi, 2012. "Two Sample Tests for High Dimensional Covariance Matrices," MPRA Paper 46026, University Library of Munich, Germany.
    28. James Mitchell & Stephen G. Hall, 2005. "Evaluating, Comparing and Combining Density Forecasts Using the KLIC with an Application to the Bank of England and NIESR ‘Fan’ Charts of Inflation," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 67(s1), pages 995-1033, December.
    29. Tony Cai & Weidong Liu & Yin Xia, 2013. "Two-Sample Covariance Matrix Testing and Support Recovery in High-Dimensional and Sparse Settings," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 108(501), pages 265-277, March.
    30. Marco Del Negro & Frank Schorfheide, 2004. "Priors from General Equilibrium Models for VARS," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 45(2), pages 643-673, May.
    31. Negro, Marco Del & Schorfheide, Frank, 2013. "DSGE Model-Based Forecasting," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 57-140, Elsevier.
    32. Capistran, Carlos, 2006. "On comparing multi-horizon forecasts," Economics Letters, Elsevier, vol. 93(2), pages 176-181, November.
    33. Cai, T. Tony & Liang, Tengyuan & Zhou, Harrison H., 2015. "Law of log determinant of sample covariance matrix and optimal estimation of differential entropy for high-dimensional Gaussian distributions," Journal of Multivariate Analysis, Elsevier, vol. 137(C), pages 161-172.
    34. Frank Smets & Rafael Wouters, 2007. "Shocks and Frictions in US Business Cycles: A Bayesian DSGE Approach," American Economic Review, American Economic Association, vol. 97(3), pages 586-606, June.
    35. Clements, Michael P. & Hendry, David F., 1997. "An empirical study of seasonal unit roots in forecasting," International Journal of Forecasting, Elsevier, vol. 13(3), pages 341-355, September.
    36. White, Halbert, 1980. "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity," Econometrica, Econometric Society, vol. 48(4), pages 817-838, May.
    37. Rogier Quaedvlieg, 2021. "Multi-Horizon Forecast Comparison," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(1), pages 40-53, January.
    38. David F Hendry & John N J Muellbauer, 2018. "The future of macroeconomics: macro theory and models at the Bank of England," Oxford Review of Economic Policy, Oxford University Press and Oxford Review of Economic Policy Limited, vol. 34(1-2), pages 287-328.
    39. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    40. Abadir, Karim M. & Distaso, Walter & Žikeš, Filip, 2014. "Design-free estimation of variance matrices," Journal of Econometrics, Elsevier, vol. 181(2), pages 165-180.
    41. Engle, Robert F., 1984. "Wald, likelihood ratio, and Lagrange multiplier tests in econometrics," Handbook of Econometrics, in: Z. Griliches† & M. D. Intriligator (ed.), Handbook of Econometrics, edition 1, volume 2, chapter 13, pages 775-826, Elsevier.
    42. Martinez, Andrew B., 2015. "How good are US government forecasts of the federal debt?," International Journal of Forecasting, Elsevier, vol. 31(2), pages 312-324.
    43. Eben Lazarus & Daniel J. Lewis & James H. Stock, 2021. "The Size‐Power Tradeoff in HAR Inference," Econometrica, Econometric Society, vol. 89(5), pages 2497-2516, September.
    44. Edge, Rochelle M. & Kiley, Michael T. & Laforte, Jean-Philippe, 2008. "Natural rate measures in an estimated DSGE model of the U.S. economy," Journal of Economic Dynamics and Control, Elsevier, vol. 32(8), pages 2512-2535, August.
    45. Fuhrer, Jeffrey C, 1997. "Inflation/Output Variance Trade-Offs and Optimal Monetary Policy," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 29(2), pages 214-234, May.
    46. G. Elliott & C. Granger & A. Timmermann (ed.), 2013. "Handbook of Economic Forecasting," Handbook of Economic Forecasting, Elsevier, edition 1, volume 2, number 2.
    47. Dr. James Mitchell, 2005. "Evaluating, comparing and combining density forecasts using the KLIC with an application to the Bank of England and NIESR ÔfanÕ charts of inflation," National Institute of Economic and Social Research (NIESR) Discussion Papers 253, National Institute of Economic and Social Research.
    48. Michael Wolf & Dan Wunderli, 2012. "Bootstrap joint prediction regions," ECON - Working Papers 064, Department of Economics - University of Zurich, revised May 2013.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bora, Siddhartha & Katchova, Ani & Kuethe, Todd, 2020. "How Accurate are the USDA’s Baseline Projections?," 2020 Agricultural and Rural Finance Markets in Transition (Virtual Meeting), October 22, 2020 309147, Regional Research Committee NC-1177 (formerly NC-1014): Agricultural and Rural Finance Markets in Transition.
    2. Håvard Hungnes, 2020. "Equal predictability test for multi-step-ahead system forecasts invariant to linear transformations," Discussion Papers 931, Statistics Norway, Research Department.
    3. Andrew B. Martinez, 2020. "Extracting Information from Different Expectations," Working Papers 2020-008, The George Washington University, Department of Economics, H. O. Stekler Research Program on Forecasting.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    2. Kolasa, Marcin & Rubaszek, Michał, 2015. "Forecasting using DSGE models with financial frictions," International Journal of Forecasting, Elsevier, vol. 31(1), pages 1-19.
    3. Barbara Rossi, 2019. "Forecasting in the presence of instabilities: How do we know whether models predict well and how to improve them," Economics Working Papers 1711, Department of Economics and Business, Universitat Pompeu Fabra, revised Jul 2021.
    4. Raffaella Giacomini, 2015. "Economic theory and forecasting: lessons from the literature," Econometrics Journal, Royal Economic Society, vol. 18(2), pages 22-41, June.
    5. Diks, Cees & Panchenko, Valentyn & Sokolinskiy, Oleg & van Dijk, Dick, 2014. "Comparing the accuracy of multivariate density forecasts in selected regions of the copula support," Journal of Economic Dynamics and Control, Elsevier, vol. 48(C), pages 79-94.
    6. Cees Diks & Valentyn Panchenko & Oleg Sokolinskiy, & Dick van Dijk, 2013. "Comparing the Accuracy of Copula-Based Multivariate Density Forecasts in Selected Regions of Support," Tinbergen Institute Discussion Papers 13-061/III, Tinbergen Institute.
    7. Fabian Krüger & Todd E. Clark & Francesco Ravazzolo, 2017. "Using Entropic Tilting to Combine BVAR Forecasts With External Nowcasts," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 35(3), pages 470-485, July.
    8. Diks, Cees & Fang, Hao, 2020. "Comparing density forecasts in a risk management context," International Journal of Forecasting, Elsevier, vol. 36(2), pages 531-551.
    9. Knüppel, Malte & Schultefrankenfeld, Guido, 2019. "Assessing the uncertainty in central banks’ inflation outlooks," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1748-1769.
    10. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    11. Raffaella Giacomini, 2014. "Economic theory and forecasting: lessons from the literature," CeMMAP working papers 41/14, Institute for Fiscal Studies.
    12. Berg, Tim O. & Henzel, Steffen R., 2015. "Point and density forecasts for the euro area using Bayesian VARs," International Journal of Forecasting, Elsevier, vol. 31(4), pages 1067-1095.
    13. Knut Are Aastveit & Claudia Foroni & Francesco Ravazzolo, 2017. "Density Forecasts With Midas Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(4), pages 783-801, June.
    14. Diks, Cees & Panchenko, Valentyn & van Dijk, Dick, 2011. "Likelihood-based scoring rules for comparing density forecasts in tails," Journal of Econometrics, Elsevier, vol. 163(2), pages 215-230, August.
    15. Gelain, Paolo & Iskrev, Nikolay & J. Lansing, Kevin & Mendicino, Caterina, 2019. "Inflation dynamics and adaptive expectations in an estimated DSGE model," Journal of Macroeconomics, Elsevier, vol. 59(C), pages 258-277.
    16. Tae-Hwy Lee & Yong Bao & Burak Saltoğlu, 2007. "Comparing density forecast models Previous versions of this paper have been circulated with the title, 'A Test for Density Forecast Comparison with Applications to Risk Management' since October 2003;," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 26(3), pages 203-225.
    17. Mu-Chun Wang, 2009. "Comparing the DSGE model with the factor model: an out-of-sample forecasting experiment," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 28(2), pages 167-182.
    18. Maik H. Wolters, 2015. "Evaluating Point and Density Forecasts of DSGE Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 30(1), pages 74-96, January.
    19. repec:hal:journl:peer-00834423 is not listed on IDEAS
    20. Anders Warne & Günter Coenen & Kai Christoffel, 2017. "Marginalized Predictive Likelihood Comparisons of Linear Gaussian State‐Space Models with Applications to DSGE, DSGE‐VAR, and VAR Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(1), pages 103-119, January.
    21. Capistrán Carlos, 2007. "Optimality Tests for Multi-Horizon Forecasts," Working Papers 2007-14, Banco de México.

    More about this item

    Keywords

    GFESM; log determinant; log score; mean square error;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C22 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fip:fedcwp:1717. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: 4D Library (email available below). General contact details of provider: https://edirc.repec.org/data/frbclus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.