IDEAS home Printed from https://ideas.repec.org/a/eee/econom/v226y2022i1p139-154.html
   My bibliography  Save this article

Analyzing cross-validation for forecasting with structural instability

Author

Listed:
  • Hirano, Keisuke
  • Wright, Jonathan H.

Abstract

When forecasting with economic time series data, researchers often use a restricted window of observations or downweight past observations in order to mitigate the potential effects of parameter instability. In this paper, we study the problem of selecting a window for point forecasts made at the end of the sample. We develop asymptotic approximations to the sampling properties of window selection methods, and post-window selection point forecasts, where there is local parameter instability of various sorts. We examine risk properties of point forecasts made after cross-validation to select the window, and compare this approach to some alternative methods of selecting the window. We also propose a quasi-Bayesian form of cross-validation that we find to have good risk properties.

Suggested Citation

  • Hirano, Keisuke & Wright, Jonathan H., 2022. "Analyzing cross-validation for forecasting with structural instability," Journal of Econometrics, Elsevier, vol. 226(1), pages 139-154.
  • Handle: RePEc:eee:econom:v:226:y:2022:i:1:p:139-154
    DOI: 10.1016/j.jeconom.2020.10.009
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0304407620304024
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.jeconom.2020.10.009?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Ploberger, Werner & Krämer;, Walter, 1990. "The Local Power of the CUSUM and CUSUM of Squares Tests," Econometric Theory, Cambridge University Press, vol. 6(3), pages 335-347, September.
    2. Atsushi Inoue & Mototsugu Shintani, 2018. "Quasi‐Bayesian model selection," Quantitative Economics, Econometric Society, vol. 9(3), pages 1265-1297, November.
    3. Inoue, Atsushi & Jin, Lu & Rossi, Barbara, 2017. "Rolling window selection for out-of-sample forecasting with time-varying parameters," Journal of Econometrics, Elsevier, vol. 196(1), pages 55-67.
    4. Koo, Bonsoo & Seo, Myung Hwan, 2015. "Structural-break models under mis-specification: Implications for forecasting," Journal of Econometrics, Elsevier, vol. 188(1), pages 166-181.
    5. Michael W. McCracken & Serena Ng, 2016. "FRED-MD: A Monthly Database for Macroeconomic Research," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 34(4), pages 574-589, October.
    6. Elliott, Graham & Muller, Ulrich K., 2007. "Confidence sets for the date of a single break in linear time series regressions," Journal of Econometrics, Elsevier, vol. 141(2), pages 1196-1218, December.
    7. Chernozhukov, Victor & Hong, Han, 2003. "An MCMC approach to classical estimation," Journal of Econometrics, Elsevier, vol. 115(2), pages 293-346, August.
    8. Pesaran, M. Hashem & Pick, Andreas & Pranovich, Mikhail, 2013. "Optimal forecasts in the presence of structural breaks," Journal of Econometrics, Elsevier, vol. 177(2), pages 134-152.
    9. Davis, Richard A. & Dunsmuir, William T.M., 1996. "Maximum Likelihood Estimation for MA(1) Processes with a Root on or near the Unit Circle," Econometric Theory, Cambridge University Press, vol. 12(1), pages 1-29, March.
    10. Gary Koop & Simon M. Potter, 2007. "Estimation and Forecasting in Models with Multiple Breaks," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 74(3), pages 763-789.
    11. Jushan Bai & Pierre Perron, 1998. "Estimating and Testing Linear Models with Multiple Structural Changes," Econometrica, Econometric Society, vol. 66(1), pages 47-78, January.
    12. Shephard, Neil, 1993. "Distribution of the ML Estimator of an MA(1) and a local level model," Econometric Theory, Cambridge University Press, vol. 9(3), pages 377-401, June.
    13. Geweke, John & Amisano, Gianni, 2011. "Optimal prediction pools," Journal of Econometrics, Elsevier, vol. 164(1), pages 130-141, September.
    14. Giraitis, Liudas & Kapetanios, George & Price, Simon, 2013. "Adaptive forecasting in the presence of recent and ongoing structural change," Journal of Econometrics, Elsevier, vol. 177(2), pages 153-170.
    15. Xiaohong Chen & Timothy M. Christensen & Elie Tamer, 2018. "Monte Carlo Confidence Sets for Identified Sets," Econometrica, Econometric Society, vol. 86(6), pages 1965-2018, November.
    16. James H. Stock & Mark W. Watson, 2010. "Modeling inflation after the crisis," Proceedings - Economic Policy Symposium - Jackson Hole, Federal Reserve Bank of Kansas City, pages 173-220.
    17. G. Elliott & C. Granger & A. Timmermann (ed.), 2013. "Handbook of Economic Forecasting," Handbook of Economic Forecasting, Elsevier, edition 1, volume 2, number 2.
    18. Pesaran, M. Hashem & Timmermann, Allan, 2007. "Selection of estimation window in the presence of breaks," Journal of Econometrics, Elsevier, vol. 137(1), pages 134-161, March.
    19. Faust, Jon & Wright, Jonathan H., 2013. "Forecasting Inflation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 2-56, Elsevier.
    20. Müller, Ulrich K. & Wang, Yulong, 2019. "Nearly weighted risk minimal unbiased estimation," Journal of Econometrics, Elsevier, vol. 209(1), pages 18-34.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Barbara Rossi, 2019. "Forecasting in the presence of instabilities: How do we know whether models predict well and how to improve them," Economics Working Papers 1711, Department of Economics and Business, Universitat Pompeu Fabra, revised Jul 2021.
    2. Boot, Tom & Pick, Andreas, 2020. "Does modeling a structural break improve forecast accuracy?," Journal of Econometrics, Elsevier, vol. 215(1), pages 35-59.
    3. Gantungalag Altansukh & Denise R. Osborn, 2022. "Using structural break inference for forecasting time series," Empirical Economics, Springer, vol. 63(1), pages 1-41, July.
    4. Hännikäinen Jari, 2017. "Selection of an Estimation Window in the Presence of Data Revisions and Recent Structural Breaks," Journal of Econometric Methods, De Gruyter, vol. 6(1), pages 1-22, January.
    5. Raffaella Giacomini & Barbara Rossi, 2015. "Forecasting in Nonstationary Environments: What Works and What Doesn't in Reduced-Form and Structural Models," Annual Review of Economics, Annual Reviews, vol. 7(1), pages 207-229, August.
    6. Davide De Gaetano, 2018. "Forecast Combinations in the Presence of Structural Breaks: Evidence from U.S. Equity Markets," Mathematics, MDPI, vol. 6(3), pages 1-19, March.
    7. Tae‐Hwy Lee & Shahnaz Parsaeian & Aman Ullah, 2022. "Optimal forecast under structural breaks," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 965-987, August.
    8. Inoue, Atsushi & Jin, Lu & Rossi, Barbara, 2017. "Rolling window selection for out-of-sample forecasting with time-varying parameters," Journal of Econometrics, Elsevier, vol. 196(1), pages 55-67.
    9. Tom Boot & Andreas Pick, 2017. "A near optimal test for structural breaks when forecasting under square error loss," Tinbergen Institute Discussion Papers 17-039/III, Tinbergen Institute.
    10. Dellas, Harris & Gibson, Heather D. & Hall, Stephen G. & Tavlas, George S., 2018. "The macroeconomic and fiscal implications of inflation forecast errors," Journal of Economic Dynamics and Control, Elsevier, vol. 93(C), pages 203-217.
    11. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    12. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    13. Philippe Goulet Coulombe & Maxime Leroux & Dalibor Stevanovic & Stéphane Surprenant, 2022. "How is machine learning useful for macroeconomic forecasting?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 920-964, August.
    14. Bańbura, Marta & Bobeica, Elena, 2023. "Does the Phillips curve help to forecast euro area inflation?," International Journal of Forecasting, Elsevier, vol. 39(1), pages 364-390.
    15. Delle Monache, Davide & Petrella, Ivan, 2017. "Adaptive models and heavy tails with an application to inflation forecasting," International Journal of Forecasting, Elsevier, vol. 33(2), pages 482-501.
    16. Tae‐Hwy Lee & Shahnaz Parsaeian & Aman Ullah, 2022. "Forecasting Under Structural Breaks Using Improved Weighted Estimation," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 84(6), pages 1485-1501, December.
    17. Davide De Gaetano, 2017. "Forecasting With Garch Models Under Structural Breaks: An Approach Based On Combinations Across Estimation Windows," Departmental Working Papers of Economics - University 'Roma Tre' 0219, Department of Economics - University Roma Tre.
    18. Mwasi Paza Mboya & Philipp Sibbertsen, 2023. "Optimal forecasts in the presence of discrete structural breaks under long memory," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(7), pages 1889-1908, November.
    19. Byron Botha & Rulof Burger & Kevin Kotzé & Neil Rankin & Daan Steenkamp, 2023. "Big data forecasting of South African inflation," Empirical Economics, Springer, vol. 65(1), pages 149-188, July.
    20. Yousuf, Kashif & Ng, Serena, 2021. "Boosting high dimensional predictive regressions with time varying parameters," Journal of Econometrics, Elsevier, vol. 224(1), pages 60-87.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:econom:v:226:y:2022:i:1:p:139-154. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jeconom .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.