IDEAS home Printed from https://ideas.repec.org/a/wly/japmet/v35y2020i4p391-409.html
   My bibliography  Save this article

Comparing predictive accuracy in small samples using fixed‐smoothing asymptotics

Author

Listed:
  • Laura Coroneo
  • Fabrizio Iacone

Abstract

We consider fixed‐smoothing asymptotics for the Diebold and Mariano (Journal of Business and Economic Statistics, 1995, 13(3), 253–263) test of predictive accuracy. We show that this approach delivers predictive accuracy tests that are correctly sized even when only a small number of out‐of‐sample observations is available. We apply the fixed‐smoothing asymptotics to the Diebold and Mariano test to evaluate the predictive accuracy of the Survey of Professional Forecasters (SPF) and of the European Central Bank Survey of Professional Forecasters (ECB SPF) against a simple random walk. Our results show that the predictive abilities of the SPF and of the ECB SPF were partially spurious.

Suggested Citation

  • Laura Coroneo & Fabrizio Iacone, 2020. "Comparing predictive accuracy in small samples using fixed‐smoothing asymptotics," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(4), pages 391-409, June.
    • Handle: RePEc:wly:japmet:v:35:y:2020:i:4:p:391-409
    DOI: 10.1002/jae.2756
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/jae.2756
    Download Restriction: no
    File URL: https://libkey.io/10.1002/jae.2756?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Phillips, Peter C.B., 2005. "Hac Estimation By Automated Regression," Econometric Theory, Cambridge University Press, vol. 21(1), pages 116-142, February.
    2. Yixiao Sun, 2013. "A heteroskedasticity and autocorrelation robust F test using an orthonormal series variance estimator," Econometrics Journal, Royal Economic Society, vol. 16(1), pages 1-26, February.
    3. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    4. Hualde, Javier & Iacone, Fabrizio, 2017. "Fixed bandwidth asymptotics for the studentized mean of fractionally integrated processes," Economics Letters, Elsevier, vol. 150(C), pages 39-43.
    5. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    6. Raffaella Giacomini & Barbara Rossi, 2009. "Detecting and Predicting Forecast Breakdowns," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 76(2), pages 669-705.
    7. Garcí­a, Juan Angel, 2003. "An introduction to the ECB's survey of professional forecasters," Occasional Paper Series 8, European Central Bank.
    8. Kiefer, Nicholas M. & Vogelsang, Timothy J., 2005. "A New Asymptotic Theory For Heteroskedasticity-Autocorrelation Robust Tests," Econometric Theory, Cambridge University Press, vol. 21(6), pages 1130-1164, December.
    9. Whitney K. Newey & Kenneth D. West, 1994. "Automatic Lag Selection in Covariance Matrix Estimation," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 61(4), pages 631-653.
    10. Javier Hualde & Fabrizio Iacone, 2015. "Autocorrelation robust inference using the Daniell kernel with fixed bandwidth," Discussion Papers 15/14, Department of Economics, University of York.
    11. Juan Angel Garcia, 2003. "An introduction to the ECB’s survey of professional forecasters," Occasional Paper Series 08, European Central Bank.
    12. Francis X. Diebold & Jose A. Lopez, 1995. "Forecast evaluation and combination," Research Paper 9525, Federal Reserve Bank of New York.
    13. Eben Lazarus & Daniel J. Lewis & James H. Stock & Mark W. Watson, 2018. "HAR Inference: Recommendations for Practice Rejoinder," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 36(4), pages 574-575, October.
    14. Li, Jia & Patton, Andrew J., 2018. "Asymptotic inference about predictive accuracy using high frequency data," Journal of Econometrics, Elsevier, vol. 203(2), pages 223-240.
    15. Granger, C. W. J. & Newbold, Paul, 1986. "Forecasting Economic Time Series," Elsevier Monographs, Elsevier, edition 2, number 9780122951831 edited by Shell, Karl.
    16. Conflitti, Cristina & De Mol, Christine & Giannone, Domenico, 2015. "Optimal combination of survey forecasts," International Journal of Forecasting, Elsevier, vol. 31(4), pages 1096-1103.
    17. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    18. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.
    19. Yixiao Sun & Peter C. B. Phillips & Sainan Jin, 2008. "Optimal Bandwidth Selection in Heteroskedasticity-Autocorrelation Robust Testing," Econometrica, Econometric Society, vol. 76(1), pages 175-194, January.
    20. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    21. Harvey, David I. & Leybourne, Stephen J. & Whitehouse, Emily J., 2017. "Forecast evaluation tests and negative long-run variance estimates in small samples," International Journal of Forecasting, Elsevier, vol. 33(4), pages 833-847.
    22. Francis X. Diebold, 2015. "Comparing Predictive Accuracy, Twenty Years Later: A Personal Perspective on the Use and Abuse of Diebold-Mariano Tests," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 33(1), pages 1-1, January.
    23. Wooldridge, Jeffrey M. & White, Halbert, 1988. "Some Invariance Principles and Central Limit Theorems for Dependent Heterogeneous Processes," Econometric Theory, Cambridge University Press, vol. 4(2), pages 210-230, August.
    24. Carlos Bowles & Roberta Friz & Veronique Genre & Geoff Kenny & Aidan Meyler & Tuomas Rautanen, 2010. "An Evaluation of the Growth and Unemployment Forecasts in the ECB Survey of Professional Forecasters," OECD Journal: Journal of Business Cycle Measurement and Analysis, OECD Publishing, Centre for International Research on Economic Tendency Surveys, vol. 2010(2), pages 1-28.
    25. Gonçalves, Sílvia & Vogelsang, Timothy J., 2011. "Block Bootstrap Hac Robust Tests: The Sophistication Of The Naive Bootstrap," Econometric Theory, Cambridge University Press, vol. 27(4), pages 745-791, August.
    26. Eben Lazarus & Daniel J. Lewis & James H. Stock & Mark W. Watson, 2018. "HAR Inference: Recommendations for Practice," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 36(4), pages 541-559, October.
    27. Laura Coroneo & Fabrizio Iacone & Fabio Profumo, 2019. "A Real-time Density Forecast Evaluation of the ECB Survey of Professional Forecasters," Discussion Papers 19/14, Department of Economics, University of York.
    28. Raffaella Giacomini & Barbara Rossi, 2010. "Forecast comparisons in unstable environments," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 595-620.
    29. Kenny, Geoff & Genre, Véronique & Bowles, Carlos & Friz, Roberta & Meyler, Aidan & Rautanen, Tuomas, 2007. "The ECB survey of professional forecasters (SPF) - A review after eight years' experience," Occasional Paper Series 59, European Central Bank.
    30. Harvey, David & Leybourne, Stephen & Newbold, Paul, 1997. "Testing the equality of prediction mean squared errors," International Journal of Forecasting, Elsevier, vol. 13(2), pages 281-291, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Alberto Caruso & Laura Coroneo, 2023. "Does Real‐Time Macroeconomic Information Help to Predict Interest Rates?," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 55(8), pages 2027-2059, December.
    2. Andrea Bastianin & Elisabetta Mirto & Yan Qin & Luca Rossini, 2024. "What drives the European carbon market? Macroeconomic factors and forecasts," Working Papers 2024.02, Fondazione Eni Enrico Mattei.
    3. Coroneo, Laura & Iacone, Fabrizio & Paccagnini, Alessia & Santos Monteiro, Paulo, 2023. "Testing the predictive accuracy of COVID-19 forecasts," International Journal of Forecasting, Elsevier, vol. 39(2), pages 606-622.
    4. Christis Katsouris, 2023. "High Dimensional Time Series Regression Models: Applications to Statistical Learning Methods," Papers 2308.16192, arXiv.org.
    5. Bastianin, Andrea & Mirto, Elisabetta & Qin, Yan & Rossini, Luca, 2024. "What drives the European carbon market? Macroeconomic factors and forecasts," FEEM Working Papers 339740, Fondazione Eni Enrico Mattei (FEEM).
    6. Yu Bai & Andrea Carriero & Todd E. Clark & Massimiliano Marcellino, 2022. "Macroeconomic forecasting in a multi‐country context," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(6), pages 1230-1255, September.
    7. Christian Glocker & Serguei Kaniovski, 2022. "Macroeconometric forecasting using a cluster of dynamic factor models," Empirical Economics, Springer, vol. 63(1), pages 43-91, July.
    8. Matei Demetrescu & Christoph Hanck & Robinson Kruse‐Becher, 2022. "Robust inference under time‐varying volatility: A real‐time evaluation of professional forecasters," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 1010-1030, August.
    9. Raïsa Basselier & David Antonio Liedo & Geert Langenus, 2018. "Nowcasting Real Economic Activity in the Euro Area: Assessing the Impact of Qualitative Surveys," Journal of Business Cycle Research, Springer;Centre for International Research on Economic Tendency Surveys (CIRET), vol. 14(1), pages 1-46, April.
    10. Katleho Makatjane & Tshepiso Tsoku, 2022. "Bootstrapping Time-Varying Uncertainty Intervals for Extreme Daily Return Periods," IJFS, MDPI, vol. 10(1), pages 1-23, January.
    11. Jack Fosten & Daniel Gutknecht, 2021. "Horizon confidence sets," Empirical Economics, Springer, vol. 61(2), pages 667-692, August.
    12. Alberto Caruso & Laura Coroneo, 2019. "Predicting interest rates in real-time," Discussion Papers 19/18, Department of Economics, University of York.
    13. Tobias Fissler & Yannick Hoga, 2021. "Backtesting Systemic Risk Forecasts using Multi-Objective Elicitability," Papers 2104.10673, arXiv.org, revised Feb 2022.
    14. Jasiński, Tomasz, 2020. "Use of new variables based on air temperature for forecasting day-ahead spot electricity prices using deep neural networks: A new approach," Energy, Elsevier, vol. 213(C).
    15. Norman R. Swanson & Weiqi Xiong & Xiye Yang, 2020. "Predicting interest rates using shrinkage methods, real‐time diffusion indexes, and model combinations," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(5), pages 587-613, August.
    16. Andrea Bastianin & Elisabetta Mirto & Yan Qin & Luca Rossini, 2024. "What drives the European carbon market? Macroeconomic factors and forecasts," Working Papers 2024.02, Fondazione Eni Enrico Mattei.
    17. Wegmüller, Philipp & Glocker, Christian & Guggia, Valentino, 2023. "Weekly economic activity: Measurement and informational content," International Journal of Forecasting, Elsevier, vol. 39(1), pages 228-243.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Laura Coroneo & Fabrizio Iacone, 2015. "Comparing predictive accuracy in small samples," Discussion Papers 15/15, Department of Economics, University of York.
    2. Coroneo, Laura & Iacone, Fabrizio & Paccagnini, Alessia & Santos Monteiro, Paulo, 2023. "Testing the predictive accuracy of COVID-19 forecasts," International Journal of Forecasting, Elsevier, vol. 39(2), pages 606-622.
    3. Casini, Alessandro, 2023. "Theory of evolutionary spectra for heteroskedasticity and autocorrelation robust inference in possibly misspecified and nonstationary models," Journal of Econometrics, Elsevier, vol. 235(2), pages 372-392.
    4. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    5. Federico Belotti & Alessandro Casini & Leopoldo Catania & Stefano Grassi & Pierre Perron, 2023. "Simultaneous bandwidths determination for DK-HAC estimators and long-run variance estimation in nonparametric settings," Econometric Reviews, Taylor & Francis Journals, vol. 42(3), pages 281-306, February.
    6. Hirukawa, Masayuki, 2023. "Robust Covariance Matrix Estimation in Time Series: A Review," Econometrics and Statistics, Elsevier, vol. 27(C), pages 36-61.
    7. Barbara Rossi & Atsushi Inoue, 2012. "Out-of-Sample Forecast Tests Robust to the Choice of Window Size," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(3), pages 432-453, April.
    8. Daniel Borup & Jonas N. Eriksen & Mads M. Kjær & Martin Thyrsgaard, 2020. "Predicting bond return predictability," CREATES Research Papers 2020-09, Department of Economics and Business Economics, Aarhus University.
    9. Håvard Hungnes, 2020. "Equal predictability test for multi-step-ahead system forecasts invariant to linear transformations," Discussion Papers 931, Statistics Norway, Research Department.
    10. Matei Demetrescu & Christoph Hanck & Robinson Kruse‐Becher, 2022. "Robust inference under time‐varying volatility: A real‐time evaluation of professional forecasters," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 1010-1030, August.
    11. Rossi, Barbara & Sekhposyan, Tatevik, 2011. "Understanding models' forecasting performance," Journal of Econometrics, Elsevier, vol. 164(1), pages 158-172, September.
    12. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    13. Pincheira, Pablo & Hardy, Nicolas, 2022. "Correlation Based Tests of Predictability," MPRA Paper 112014, University Library of Munich, Germany.
    14. Costantini, Mauro & Kunst, Robert M., 2021. "On using predictive-ability tests in the selection of time-series prediction models: A Monte Carlo evaluation," International Journal of Forecasting, Elsevier, vol. 37(2), pages 445-460.
    15. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    16. Rho, Seunghwa & Vogelsang, Timothy J., 2021. "Inference in time series models using smoothed-clustered standard errors," Journal of Econometrics, Elsevier, vol. 224(1), pages 113-133.
    17. Busetti, Fabio & Marcucci, Juri, 2013. "Comparing forecast accuracy: A Monte Carlo investigation," International Journal of Forecasting, Elsevier, vol. 29(1), pages 13-27.
    18. Pellatt, Daniel F. & Sun, Yixiao, 2023. "Asymptotic F test in regressions with observations collected at high frequency over long span," Journal of Econometrics, Elsevier, vol. 235(2), pages 1281-1309.
    19. Kruse, Robinson & Leschinski, Christian & Will, Michael, 2016. "Comparing Predictive Accuracy under Long Memory - With an Application to Volatility Forecasting," Hannover Economic Papers (HEP) dp-571, Leibniz Universität Hannover, Wirtschaftswissenschaftliche Fakultät.
    20. Filip Stanek, 2021. "Optimal Out-of-Sample Forecast Evaluation under Stationarity," CERGE-EI Working Papers wp712, The Center for Economic Research and Graduate Education - Economics Institute, Prague.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:japmet:v:35:y:2020:i:4:p:391-409. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.interscience.wiley.com/jpages/0883-7252/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.