IDEAS home Printed from https://ideas.repec.org/p/yor/yorken/21-03.html

Testing for equal predictive accuracy with strong dependence

Author

Listed:
  • Laura Coroneo
  • Fabrizio Iacone

Abstract

We revisit the Diebold and Mariano (1995) test, investigating the consequences of having autocorrelation in the loss differential. This situation can arise not only when a forecast is sub-optimal under MSE loss, but also when it is optimal under an alternative loss, or it is evaluated on a short sample, or when a forecast with weakly dependent forecast errors is compared to a naive benchmark. We show that the power of the Diebold and Mariano (1995) test decreases as the dependence increases, making it more difficult to obtain statistically significant evidence of superior predictive ability against less accurate benchmarks. Moreover, we find that after a certain threshold the test has no power and the correct null hypothesis is spuriously rejected. Taken together, these results caution to seriously consider the dependence properties of the selected forecast and of the loss differential before the application of the Diebold and Mariano (1995) test, especially when naive benchmarks are considered.

Suggested Citation

  • Laura Coroneo & Fabrizio Iacone, 2021. "Testing for equal predictive accuracy with strong dependence," Discussion Papers 21/03, Department of Economics, University of York.
    • Handle: RePEc:yor:yorken:21/03
    as

    Download full text from publisher

    File URL: https://www.york.ac.uk/media/economics/documents/discussionpapers/2021/2103.pdf
    File Function: Main text
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Askitas, Nikos & Tatsiramos, Konstantinos & Verheyden, Bertrand, 2020. "Lockdown Strategies, Mobility Patterns and COVID-19," IZA Discussion Papers 13293, Institute of Labor Economics (IZA).
    2. Dittmann, Ingolf & Granger, Clive W. J., 2002. "Properties of nonlinear transformations of fractionally integrated processes," Journal of Econometrics, Elsevier, vol. 110(2), pages 113-133, October.
    3. Marinucci, D. & Robinson, Peter M., 2001. "Narrow-band analysis of nonstationary processes," LSE Research Online Documents on Economics 303, London School of Economics and Political Science, LSE Library.
    4. Whitney K. Newey & Kenneth D. West, 1994. "Automatic Lag Selection in Covariance Matrix Estimation," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 61(4), pages 631-653.
    5. Frijters, Paul & Lalji, Chitwan & Pakrashi, Debayan, 2020. "Daily weather only has small effects on wellbeing in the US," Journal of Economic Behavior & Organization, Elsevier, vol. 176(C), pages 747-762.
    6. Apostolos Davillas & Andrew M Jones, 2021. "The first wave of the COVID‐19 pandemic and its impact on socioeconomic inequality in psychological distress in the UK," Health Economics, John Wiley & Sons, Ltd., vol. 30(7), pages 1668-1683, July.
    7. Michael W. McCracken, 2020. "Diverging Tests of Equal Predictive Ability," Econometrica, Econometric Society, vol. 88(4), pages 1753-1754, July.
    8. Barbara Rossi, 2005. "Testing Long-Horizon Predictive Ability With High Persistence, And The Meese-Rogoff Puzzle," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 46(1), pages 61-92, February.
    9. Hualde, Javier & Iacone, Fabrizio, 2017. "Fixed bandwidth asymptotics for the studentized mean of fractionally integrated processes," Economics Letters, Elsevier, vol. 150(C), pages 39-43.
    10. Gonçalves, Sílvia & Vogelsang, Timothy J., 2011. "Block Bootstrap Hac Robust Tests: The Sophistication Of The Naive Bootstrap," Econometric Theory, Cambridge University Press, vol. 27(4), pages 745-791, August.
    11. Marcellino, Massimiliano & Stock, James H. & Watson, Mark W., 2003. "Macroeconomic forecasting in the Euro area: Country specific versus area-wide information," European Economic Review, Elsevier, vol. 47(1), pages 1-18, February.
    12. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    13. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    14. Eben Lazarus & Daniel J. Lewis & James H. Stock & Mark W. Watson, 2018. "HAR Inference: Recommendations for Practice," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 36(4), pages 541-559, October.
    15. Khalaf, Lynda & Saunders, Charles J., 2017. "Monte Carlo forecast evaluation with persistent data," International Journal of Forecasting, Elsevier, vol. 33(1), pages 1-10.
    16. Kiefer, Nicholas M. & Vogelsang, Timothy J., 2005. "A New Asymptotic Theory For Heteroskedasticity-Autocorrelation Robust Tests," Econometric Theory, Cambridge University Press, vol. 21(6), pages 1130-1164, December.
    17. Marinucci, D & Robinson, Peter, 2001. "Narrow-band analysis of nonstationary processes," LSE Research Online Documents on Economics 2015, London School of Economics and Political Science, LSE Library.
    18. Patton, Andrew J. & Timmermann, Allan, 2007. "Properties of optimal forecasts under asymmetric loss and nonlinearity," Journal of Econometrics, Elsevier, vol. 140(2), pages 884-918, October.
    19. Phillips, Peter C.B. & Magdalinos, Tassos, 2007. "Limit theory for moderate deviations from a unit root," Journal of Econometrics, Elsevier, vol. 136(1), pages 115-130, January.
    20. Eben Lazarus & Daniel J. Lewis & James H. Stock & Mark W. Watson, 2018. "HAR Inference: Recommendations for Practice Rejoinder," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 36(4), pages 574-575, October.
    21. Francis X. Diebold, 2015. "Comparing Predictive Accuracy, Twenty Years Later: A Personal Perspective on the Use and Abuse of Diebold-Mariano Tests," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 33(1), pages 1-1, January.
    22. Forni, Mario & Hallin, Marc & Lippi, Marco & Reichlin, Lucrezia, 2003. "Do financial variables help forecasting inflation and real activity in the euro area?," Journal of Monetary Economics, Elsevier, vol. 50(6), pages 1243-1255, September.
    23. Yixiao Sun, 2014. "Fixed-smoothing Asymptotics and AsymptoticFandtTests in the Presence of Strong Autocorrelation," Advances in Econometrics, in: Essays in Honor of Peter C. B. Phillips, volume 14, pages 23-63, Emerald Group Publishing Limited.
    24. Laura Coroneo & Fabrizio Iacone, 2020. "Comparing predictive accuracy in small samples using fixed‐smoothing asymptotics," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(4), pages 391-409, June.
    25. Giraitis, Liudas & Phillips, Peter C.B., 2012. "Mean and autocovariance function estimation near the boundary of stationarity," Journal of Econometrics, Elsevier, vol. 169(2), pages 166-178.
    26. James Banks & Xiaowei Xu, 2020. "The Mental Health Effects of the First Two Months of Lockdown during the COVID‐19 Pandemic in the UK," Fiscal Studies, John Wiley & Sons, vol. 41(3), pages 685-708, September.
    27. Corradi, Valentina & Swanson, Norman R. & Olivetti, Claudia, 2001. "Predictive ability with cointegrated variables," Journal of Econometrics, Elsevier, vol. 104(2), pages 315-358, September.
    28. Yixiao Sun, 2014. "Fixed‐Smoothing Asymptotics in a Two‐Step Generalized Method of Moments Framework," Econometrica, Econometric Society, vol. 82, pages 2327-2370, November.
    29. D Marinucci & Peter M Robinson, 2001. "Narrow-Band Analysis of Nonstationary Processes," STICERD - Econometrics Paper Series 421, Suntory and Toyota International Centres for Economics and Related Disciplines, LSE.
    30. Xiaofeng Shao, 2015. "Self-Normalization for Time Series: A Review of Recent Developments," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(512), pages 1797-1817, December.
    31. McElroy, Tucker & Politis, Dimitris N., 2012. "Fixed-B Asymptotics For The Studentized Mean From Time Series With Short, Long, Or Negative Memory," Econometric Theory, Cambridge University Press, vol. 28(2), pages 471-481, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Nikolaos T. Giannakopoulos & Damianos P. Sakas & Kanellos Toudas & Panagiotis Karountzos, 2025. "Implication of Digital Marketing in the Supply Chain Finance of the Beverage Industry," IJFS, MDPI, vol. 13(4), pages 1-23, October.
    2. Coroneo, Laura & Iacone, Fabrizio & Profumo, Fabio, 2024. "Survey density forecast comparison in small samples," International Journal of Forecasting, Elsevier, vol. 40(4), pages 1486-1504.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Coroneo, Laura & Iacone, Fabrizio & Paccagnini, Alessia & Santos Monteiro, Paulo, 2023. "Testing the predictive accuracy of COVID-19 forecasts," International Journal of Forecasting, Elsevier, vol. 39(2), pages 606-622.
    2. Laura Coroneo & Fabrizio Iacone, 2020. "Comparing predictive accuracy in small samples using fixed‐smoothing asymptotics," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(4), pages 391-409, June.
    3. Coroneo, Laura & Iacone, Fabrizio & Profumo, Fabio, 2024. "Survey density forecast comparison in small samples," International Journal of Forecasting, Elsevier, vol. 40(4), pages 1486-1504.
    4. Kruse, Robinson & Leschinski, Christian & Will, Michael, 2016. "Comparing Predictive Accuracy under Long Memory - With an Application to Volatility Forecasting," Hannover Economic Papers (HEP) dp-571, Leibniz Universität Hannover, Wirtschaftswissenschaftliche Fakultät.
    5. Hirukawa, Masayuki, 2023. "Robust Covariance Matrix Estimation in Time Series: A Review," Econometrics and Statistics, Elsevier, vol. 27(C), pages 36-61.
    6. Casini, Alessandro & Perron, Pierre, 2024. "Prewhitened long-run variance estimation robust to nonstationarity," Journal of Econometrics, Elsevier, vol. 242(1).
    7. Filip Staněk, 2023. "Optimal out‐of‐sample forecast evaluation under stationarity," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(8), pages 2249-2279, December.
    8. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    9. Oguzhan Akgun & Alain Pirotte & Giovanni Urga & Zhenlin Yang, 2025. "Testing Clustered Equal Predictive Ability with Unknown Clusters," Papers 2507.14621, arXiv.org, revised Jul 2025.
    10. Federico Belotti & Alessandro Casini & Leopoldo Catania & Stefano Grassi & Pierre Perron, 2023. "Simultaneous bandwidths determination for DK-HAC estimators and long-run variance estimation in nonparametric settings," Econometric Reviews, Taylor & Francis Journals, vol. 42(3), pages 281-306, February.
    11. Matei Demetrescu & Christoph Hanck & Robinson Kruse, 2016. "Fixed-b Inference in the Presence of Time-Varying Volatility," CREATES Research Papers 2016-01, Department of Economics and Business Economics, Aarhus University.
    12. Jungbin Hwang & Gonzalo Valdés, 2025. "HAR Inference for Quantile Regression in Time Series," Working papers 2025-03, University of Connecticut, Department of Economics.
    13. Pellatt, Daniel F. & Sun, Yixiao, 2023. "Asymptotic F test in regressions with observations collected at high frequency over long span," Journal of Econometrics, Elsevier, vol. 235(2), pages 1281-1309.
    14. Javier Hualde & Fabrizio Iacone, 2015. "Autocorrelation robust inference using the Daniell kernel with fixed bandwidth," Discussion Papers 15/14, Department of Economics, University of York.
    15. Filip Stanek, 2021. "Optimal Out-of-Sample Forecast Evaluation under Stationarity," CERGE-EI Working Papers wp712, The Center for Economic Research and Graduate Education - Economics Institute, Prague.
    16. David T. Frazier & Donald S. Poskitt, 2025. "Sequential Scoring Rule Evaluation for Forecast Method Selection," Papers 2505.09090, arXiv.org.
    17. Casini, Alessandro, 2023. "Theory of evolutionary spectra for heteroskedasticity and autocorrelation robust inference in possibly misspecified and nonstationary models," Journal of Econometrics, Elsevier, vol. 235(2), pages 372-392.
    18. Hwang, Taeyoon & Vogelsang, Timothy J., 2024. "Some fixed-b results for regressions with high frequency data over long spans," Journal of Econometrics, Elsevier, vol. 244(2).
    19. Pellatt , Daniel & Sun, Yixiao, 2020. "Asymptotic F test in Regressions with Observations Collected at High Frequency over Long Span," University of California at San Diego, Economics Working Paper Series qt19f0d9wz, Department of Economics, UC San Diego.
    20. Yu‐Sheng Lai, 2022. "Use of high‐frequency data to evaluate the performance of dynamic hedging strategies," Journal of Futures Markets, John Wiley & Sons, Ltd., vol. 42(1), pages 104-124, January.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C32 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes; State Space Models
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:yor:yorken:21/03. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Paul Hodgson (email available below). General contact details of provider: https://edirc.repec.org/data/deyoruk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.