IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v33y2017i1p1-10.html
   My bibliography  Save this article

Monte Carlo forecast evaluation with persistent data

Author

Listed:
  • Khalaf, Lynda
  • Saunders, Charles J.

Abstract

Persistent processes, including local-to-unity and random walks, are commonly considered as forecasting models of interest. However, the associated forecast errors follow non-standard distributions that complicate forecast evaluation tests. We propose a finite sample simulation-based solution to this problem. The method requires a flexible parametric null model that can be simulated as long as a finite dimension nuisance parameter can be specified. The size control of our method is robust to non-standard limiting distributions, such as degenerate asymptotic distribution problems that arise from nested and unit root models. Our simulation studies demonstrate that many of the existing forecast evaluation methods, including various bootstraps, over-reject for highly persistent data. In contrast, our method is level correct and has good power. We extend our approach to the inversion of forecast evaluation statistics in order to construct exact confidence sets for the benchmark model. Confidence sets provide much more information than tests, particularly in the case of the persistence-adjusted relevance of predictive regressors (Rossi, 2005).

Suggested Citation

  • Khalaf, Lynda & Saunders, Charles J., 2017. "Monte Carlo forecast evaluation with persistent data," International Journal of Forecasting, Elsevier, vol. 33(1), pages 1-10.
  • Handle: RePEc:eee:intfor:v:33:y:2017:i:1:p:1-10
    DOI: 10.1016/j.ijforecast.2016.06.004
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S016920701630070X
    Download Restriction: Full text for ScienceDirect subscribers only

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kirstin Hubrich & Kenneth D. West, 2010. "Forecast evaluation of small nested model sets," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 574-594.
    2. David F. Hendry & Kirstin Hubrich, 2011. "Combining Disaggregate Forecasts or Combining Disaggregate Information to Forecast an Aggregate," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(2), pages 216-227, April.
    3. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    4. Marie-Claude Beaulieu & Jean-Marie Dufour & Lynda Khalaf, 2013. "Identification-Robust Estimation and Testing of the Zero-Beta CAPM," Review of Economic Studies, Oxford University Press, vol. 80(3), pages 892-924.
    5. Ron Alquist & Lutz Kilian, 2010. "What do we learn from the price of crude oil futures?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 539-573.
    6. Dufour, Jean-Marie & Khalaf, Lynda & Bernard, Jean-Thomas & Genest, Ian, 2004. "Simulation-based finite-sample tests for heteroskedasticity and ARCH effects," Journal of Econometrics, Elsevier, vol. 122(2), pages 317-347, October.
    7. Jean‐Thomas Bernard & Jean‐Marie Dufour & Lynda Khalaf & Maral Kichian, 2012. "An identification‐robust test for time‐varying parameters in the dynamics of energy prices," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 27(4), pages 603-624, June.
    8. Donald W. K. Andrews, 2000. "Inconsistency of the Bootstrap when a Parameter Is on the Boundary of the Parameter Space," Econometrica, Econometric Society, vol. 68(2), pages 399-406, March.
    9. Beaulieu, Marie-Claude & Dufour, Jean-Marie & Khalaf, Lynda, 2010. "Asset-pricing anomalies and spanning: Multivariate and multifactor tests with heavy-tailed distributions," Journal of Empirical Finance, Elsevier, vol. 17(4), pages 763-782, September.
    10. Hansen, Peter Reinhard, 2005. "A Test for Superior Predictive Ability," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 365-380, October.
    11. repec:taf:jnlbes:v:30:y:2012:i:1:p:53-66 is not listed on IDEAS
    12. Bernard, Jean-Thomas & Idoudi, Nadhem & Khalaf, Lynda & Yelou, Clement, 2007. "Finite sample multivariate structural change tests with application to energy demand models," Journal of Econometrics, Elsevier, vol. 141(2), pages 1219-1244, December.
    13. James H. Stock & Mark W. Watson, 2007. "Why Has U.S. Inflation Become Harder to Forecast?," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 39(s1), pages 3-33, February.
    14. Jean-Marie Dufour & Lynda Khalaf & Marie-Claude Beaulieu, 2010. "Multivariate residual-based finite-sample tests for serial dependence and ARCH effects with applications to asset pricing models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(2), pages 263-285.
    15. Andrew Harvey & Alessandra Luati, 2014. "Filtering With Heavy Tails," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(507), pages 1112-1122, September.
    16. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    17. Jeremy Berkowitz & Lorenzo Giorgianni, 2001. "Long-Horizon Exchange Rate Predictability?," The Review of Economics and Statistics, MIT Press, vol. 83(1), pages 81-91, February.
    18. Barbara Rossi, 2005. "Testing Long-Horizon Predictive Ability With High Persistence, And The Meese-Rogoff Puzzle," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 46(1), pages 61-92, February.
    19. Dufour, Jean-Marie, 2006. "Monte Carlo tests with nuisance parameters: A general approach to finite-sample inference and nonstandard asymptotics," Journal of Econometrics, Elsevier, vol. 133(2), pages 443-477, August.
    20. Peter R. Hansen & Asger Lunde & James M. Nason, 2011. "The Model Confidence Set," Econometrica, Econometric Society, vol. 79(2), pages 453-497, March.
    21. Peter C. B. Phillips, 2014. "On Confidence Intervals for Autoregressive Roots and Predictive Regression," Econometrica, Econometric Society, vol. 82(3), pages 1177-1195, May.
    22. Clark, Todd E. & West, Kenneth D., 2007. "Approximately normal tests for equal predictive accuracy in nested models," Journal of Econometrics, Elsevier, vol. 138(1), pages 291-311, May.
    23. David Harvey & Paul Newbold, 2000. "Tests for multiple forecast encompassing," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 15(5), pages 471-482.
    24. Goffe, William L. & Ferrier, Gary D. & Rogers, John, 1994. "Global optimization of statistical functions with simulated annealing," Journal of Econometrics, Elsevier, vol. 60(1-2), pages 65-99.
    25. Phillips, Peter C. B., 1998. "Impulse response and forecast error variance asymptotics in nonstationary VARs," Journal of Econometrics, Elsevier, vol. 83(1-2), pages 21-56.
    26. Diebold, Francis X & Kilian, Lutz, 2000. "Unit-Root Tests Are Useful for Selecting Forecasting Models," Journal of Business & Economic Statistics, American Statistical Association, vol. 18(3), pages 265-273, July.
    27. Corradi, Valentina & Swanson, Norman R. & Olivetti, Claudia, 2001. "Predictive ability with cointegrated variables," Journal of Econometrics, Elsevier, vol. 104(2), pages 315-358, September.
    28. Kemp, Gordon C.R., 1999. "The Behavior Of Forecast Errors From A Nearly Integrated Ar(1) Model As Both Sample Size And Forecast Horizon Become Large," Econometric Theory, Cambridge University Press, vol. 15(02), pages 238-256, April.
    29. Stock, James H., 1991. "Confidence intervals for the largest autoregressive root in U.S. macroeconomic time series," Journal of Monetary Economics, Elsevier, vol. 28(3), pages 435-459, December.
    30. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    31. Cavanagh, Christopher L. & Elliott, Graham & Stock, James H., 1995. "Inference in Models with Nearly Integrated Regressors," Econometric Theory, Cambridge University Press, vol. 11(05), pages 1131-1147, October.
    32. Elise Coudin & Jean-Marie Dufour, 2009. "Finite-sample distribution-free inference in linear median regressions under heteroscedasticity and non-linear dependence of unknown form," Econometrics Journal, Royal Economic Society, vol. 12(s1), pages 19-49, January.
    33. Beaulieu, Marie-Claude & Dufour, Jean-Marie & Khalaf, Lynda, 2007. "Multivariate Tests of MeanVariance Efficiency With Possibly Non-Gaussian Errors: An Exact Simulation-Based Approach," Journal of Business & Economic Statistics, American Statistical Association, vol. 25, pages 398-410, October.
    34. Anna Mikusheva, 2007. "Uniform Inference in Autoregressive Models," Econometrica, Econometric Society, vol. 75(5), pages 1411-1452, September.
    Full references (including those not matched with items on IDEAS)

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:33:y:2017:i:1:p:1-10. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Dana Niculescu). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.