IDEAS home Printed from https://ideas.repec.org/p/hit/econdp/2018-03.html
   My bibliography  Save this paper

Testing for Changes in Forecasting Performance

Author

Listed:
  • PERRON, Pierre
  • YAMAMOTO, Yohei

Abstract

We consider the issue of forecast failure (or breakdown) and propose methods to assess retrospectively whether a given forecasting model provides forecasts which show evidence of changes with respect to some loss function. We adapt the classical structural change tests to the forecast failure context. First, we recommend that all tests should be carried with a fixed scheme to have best power. This ensures a maximum difference between the fitted in and out-of-sample means of the losses and avoids contamination issues under the rolling and recursive schemes. With a fixed scheme, Giacomini and Rossi's (2009) (GR) test is simply a Wald test for a one-time change in the mean of the total (the in-sample plus out-of-sample) losses at a known break date, say m, the value that separates the in and out-of-sample periods. To alleviate this problem, we consider a variety of tests: maximizing the GR test over all possible values of m within a pre-specified range; a Double sup-Wald (DSW) test which for each m performs a sup-Wald test for a change in the mean of the out-of-sample losses and takes the maximum of such tests over some range; we also propose to work directly with the total loss series to define the Total Loss sup-Wald (TLSW) test and the Total Loss UDmax (TLUD) test. Using extensive simulations, we show that with forecasting models potentially involving lagged dependent variables, the only tests having a monotonic power function for all data-generating processes are the DSW and TLUD tests, constructed with a fixed forecasting window scheme. Some explanations are provided and two empirical applications illustrate the relevance of our findings in practice.

Suggested Citation

  • PERRON, Pierre & YAMAMOTO, Yohei, 2018. "Testing for Changes in Forecasting Performance," Discussion Papers 2018-03, Graduate School of Economics, Hitotsubashi University.
  • Handle: RePEc:hit:econdp:2018-03
    as

    Download full text from publisher

    File URL: http://hermes-ir.lib.hit-u.ac.jp/rs/bitstream/10086/29252/1/070econDP18-03.pdf
    Download Restriction: no

    References listed on IDEAS

    as
    1. Raffaella Giacomini & Barbara Rossi, 2009. "Detecting and Predicting Forecast Breakdowns," Review of Economic Studies, Oxford University Press, vol. 76(2), pages 669-705.
    2. Deng, Ai & Perron, Pierre, 2008. "A non-local perspective on the power properties of the CUSUM and CUSUM of squares tests for structural change," Journal of Econometrics, Elsevier, vol. 142(1), pages 212-240, January.
    3. Kim, Dukpa & Perron, Pierre, 2009. "Assessing the relative power of structural break tests using a framework based on the approximate Bahadur slope," Journal of Econometrics, Elsevier, vol. 149(1), pages 26-51, April.
    4. Luis Filipe Martins & Pierre Perron, 2016. "Improved Tests for Forecast Comparisons in the Presence of Instabilities," Journal of Time Series Analysis, Wiley Blackwell, vol. 37(5), pages 650-659, September.
    5. Pierre Perron & Yohei Yamamoto, 2016. "On the Usefulness or Lack Thereof of Optimality Criteria for Structural Change Tests," Econometric Reviews, Taylor & Francis Journals, vol. 35(5), pages 782-844, May.
    6. Jushan Bai & Pierre Perron, 1998. "Estimating and Testing Linear Models with Multiple Structural Changes," Econometrica, Econometric Society, vol. 66(1), pages 47-78, January.
    7. Perron, Pierre, 1989. "The Great Crash, the Oil Price Shock, and the Unit Root Hypothesis," Econometrica, Econometric Society, vol. 57(6), pages 1361-1401, November.
    8. Perron, P., 1991. "A Test for Changes in a Polynomial Trend Functions for a Dynamioc Time Series," Papers 363, Princeton, Department of Economics - Econometric Research Program.
    9. Andrews, Donald W K, 1993. "Tests for Parameter Instability and Structural Change with Unknown Change Point," Econometrica, Econometric Society, vol. 61(4), pages 821-856, July.
    10. Barbara Rossi & Atsushi Inoue, 2012. "Out-of-Sample Forecast Tests Robust to the Choice of Window Size," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(3), pages 432-453, April.
    11. Zhongjun Qu, 2007. "Searching for cointegration in a dynamic system," Econometrics Journal, Royal Economic Society, vol. 10(3), pages 580-604, November.
    12. Perron, Pierre, 1990. "Testing for a Unit Root in a Time Series with a Changing Mean," Journal of Business & Economic Statistics, American Statistical Association, vol. 8(2), pages 153-162, April.
    Full references (including those not matched with items on IDEAS)

    More about this item

    Keywords

    forecast failure; non-monotonic power; structural change; out-of-sample method;

    JEL classification:

    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General
    • C22 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:hit:econdp:2018-03. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Digital Resources Section, Hitotsubashi University Library). General contact details of provider: http://edirc.repec.org/data/fehitjp.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.