IDEAS home Printed from https://ideas.repec.org/p/bos/wpaper/wp2014-003.html
   My bibliography  Save this paper

Improved Tests for Forecast Comparisons in the Presence of Instabilities

Author

Listed:
  • Luis Filipe Martins

    (Lisbon University Institute)

  • Pierre Perron

    (Boston University)

Abstract

Giacomini and Rossi (2010) proposed a aÌ uctuations test and a one-time reversal test for comparing the out-of-sample forecasting performance of two competing models in the presence of possible instabilities. In the simulations and empirical applications, they use a version of their test based on the sample variance of the loss di§erences instead of the relevant, and advocated, long-run variance (HAC estimate). We repli- cate the power properties of their tests with the appropriate HAC correction using exactly the same design they used. We show that the power functions of the tests are substantially lower than what they report. More importantly the power functions are non-monotonic. To alleviate the power problems of their tests in the presence of instabilities in the di§erences of the loss functions, we suggest using simple tests (sup- Wald and UDmax) for changes in the mean of the loss-di§erences. These are shown to have higher monotonic power, especially the UDmax version. We use their empirical examples to show that the practical relevance of the issue raised.

Suggested Citation

  • Luis Filipe Martins & Pierre Perron, 2014. "Improved Tests for Forecast Comparisons in the Presence of Instabilities," Boston University - Department of Economics - Working Papers Series 2014-003, Boston University - Department of Economics.
    • Handle: RePEc:bos:wpaper:wp2014-003
    as

    Download full text from publisher

    File URL: http://www.bu.edu/econ/files/2009/07/Perron-Improved-Tests-for-Forecast-Comparisons-Jan-2014.pdf
    File Function: First version, 2014
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    2. Clark, Todd E. & West, Kenneth D., 2006. "Using out-of-sample mean squared prediction errors to test the martingale difference hypothesis," Journal of Econometrics, Elsevier, vol. 135(1-2), pages 155-186.
    3. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    4. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    5. Andrews, Donald W K, 1991. "Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation," Econometrica, Econometric Society, vol. 59(3), pages 817-858, May.
    6. Kim, Dukpa & Perron, Pierre, 2009. "Assessing the relative power of structural break tests using a framework based on the approximate Bahadur slope," Journal of Econometrics, Elsevier, vol. 149(1), pages 26-51, April.
    7. Jushan Bai & Pierre Perron, 1998. "Estimating and Testing Linear Models with Multiple Structural Changes," Econometrica, Econometric Society, vol. 66(1), pages 47-78, January.
    8. Perron, Pierre, 1989. "The Great Crash, the Oil Price Shock, and the Unit Root Hypothesis," Econometrica, Econometric Society, vol. 57(6), pages 1361-1401, November.
    9. Donald W. K. Andrews, 2003. "Tests for Parameter Instability and Structural Change with Unknown Change Point: A Corrigendum," Econometrica, Econometric Society, vol. 71(1), pages 395-397, January.
    10. Andrews, Donald W K, 1993. "Tests for Parameter Instability and Structural Change with Unknown Change Point," Econometrica, Econometric Society, vol. 61(4), pages 821-856, July.
    11. Perron, Pierre & Vogelsang, Timothy J, 1992. "Testing for a Unit Root in a Time Series with a Changing Mean: Corrections and Extensions," Journal of Business & Economic Statistics, American Statistical Association, vol. 10(4), pages 467-470, October.
    12. Raffaella Giacomini & Barbara Rossi, 2010. "Forecast comparisons in unstable environments," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 595-620.
    13. Perron, P, 1993. "Erratum [The Great Crash, the Oil Price Shock and the Unit Root Hypothesis]," Econometrica, Econometric Society, vol. 61(1), pages 248-249, January.
    14. Perron, Pierre, 1990. "Testing for a Unit Root in a Time Series with a Changing Mean," Journal of Business & Economic Statistics, American Statistical Association, vol. 8(2), pages 153-162, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Casini, Alessandro & Perron, Pierre, 2021. "Continuous record Laplace-based inference about the break date in structural change models," Journal of Econometrics, Elsevier, vol. 224(1), pages 3-21.
    2. Casini, Alessandro, 2023. "Theory of evolutionary spectra for heteroskedasticity and autocorrelation robust inference in possibly misspecified and nonstationary models," Journal of Econometrics, Elsevier, vol. 235(2), pages 372-392.
    3. Casini, Alessandro & Perron, Pierre, 2022. "Generalized Laplace Inference In Multiple Change-Points Models," Econometric Theory, Cambridge University Press, vol. 38(1), pages 35-65, February.
    4. Alessandro Casini & Pierre Perron, 2018. "Structural Breaks in Time Series," Papers 1805.03807, arXiv.org.
    5. Alessandro Casini, 2018. "Tests for Forecast Instability and Forecast Failure under a Continuous Record Asymptotic Framework," Papers 1803.10883, arXiv.org, revised Dec 2018.
    6. Casini, Alessandro & Perron, Pierre, 2024. "Prewhitened long-run variance estimation robust to nonstationarity," Journal of Econometrics, Elsevier, vol. 242(1).
    7. Pierre Perron & Yohei Yamamoto, 2021. "Testing for Changes in Forecasting Performance," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(1), pages 148-165, January.
    8. Federico Belotti & Alessandro Casini & Leopoldo Catania & Stefano Grassi & Pierre Perron, 2023. "Simultaneous bandwidths determination for DK-HAC estimators and long-run variance estimation in nonparametric settings," Econometric Reviews, Taylor & Francis Journals, vol. 42(3), pages 281-306, February.
    9. Alessandro Casini, 2021. "Theory of Evolutionary Spectra for Heteroskedasticity and Autocorrelation Robust Inference in Possibly Misspecified and Nonstationary Models," Papers 2103.02981, arXiv.org, revised Aug 2024.
    10. Ye Li & Pierre Perron & Jiawen Xu, 2017. "Modelling exchange rate volatility with random level shifts," Applied Economics, Taylor & Francis Journals, vol. 49(26), pages 2579-2589, June.
    11. Alessandro Casini & Taosong Deng & Pierre Perron, 2021. "Theory of Low Frequency Contamination from Nonstationarity and Misspecification: Consequences for HAR Inference," Papers 2103.01604, arXiv.org, revised Sep 2024.
    12. Ciner, Cetin, 2017. "Predicting white metal prices by a commodity sensitive exchange rate," International Review of Financial Analysis, Elsevier, vol. 52(C), pages 309-315.
    13. Sebastian Fossati, 2017. "Testing for State-Dependent Predictive Ability," Working Papers 2017-9, University of Alberta, Department of Economics.
    14. Emilio Blanco & Fiorella Dogliolo & Lorena Garegnani, 2022. "Nowcasting during the Pandemic: Lessons from Argentina," BCRA Working Paper Series 202299, Central Bank of Argentina, Economic Research Department.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alessandro Casini & Taosong Deng & Pierre Perron, 2021. "Theory of Low Frequency Contamination from Nonstationarity and Misspecification: Consequences for HAR Inference," Papers 2103.01604, arXiv.org, revised Sep 2024.
    2. Casini, Alessandro & Perron, Pierre, 2024. "Prewhitened long-run variance estimation robust to nonstationarity," Journal of Econometrics, Elsevier, vol. 242(1).
    3. Pierre Perron & Yohei Yamamoto, 2022. "Structural change tests under heteroskedasticity: Joint estimation versus two‐steps methods," Journal of Time Series Analysis, Wiley Blackwell, vol. 43(3), pages 389-411, May.
    4. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    5. Pierre Perron & Yohei Yamamoto, 2016. "On the Usefulness or Lack Thereof of Optimality Criteria for Structural Change Tests," Econometric Reviews, Taylor & Francis Journals, vol. 35(5), pages 782-844, May.
    6. Rossi, Barbara & Sekhposyan, Tatevik, 2011. "Understanding models' forecasting performance," Journal of Econometrics, Elsevier, vol. 164(1), pages 158-172, September.
    7. Alessandro Casini, 2018. "Tests for Forecast Instability and Forecast Failure under a Continuous Record Asymptotic Framework," Papers 1803.10883, arXiv.org, revised Dec 2018.
    8. Raffaella Giacomini & Barbara Rossi, 2010. "Forecast comparisons in unstable environments," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 595-620.
    9. Casini, Alessandro, 2023. "Theory of evolutionary spectra for heteroskedasticity and autocorrelation robust inference in possibly misspecified and nonstationary models," Journal of Econometrics, Elsevier, vol. 235(2), pages 372-392.
    10. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    11. Pierre Perron & Yohei Yamamoto, 2021. "Testing for Changes in Forecasting Performance," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(1), pages 148-165, January.
    12. Vicente Esteve, 2004. "Política fiscal y productividad del trabajo en la economía española: un análisis de series temporales," Revista de Analisis Economico – Economic Analysis Review, Universidad Alberto Hurtado/School of Economics and Business, vol. 19(1), pages 3-29, June.
    13. Alessandro Casini, 2021. "Theory of Evolutionary Spectra for Heteroskedasticity and Autocorrelation Robust Inference in Possibly Misspecified and Nonstationary Models," Papers 2103.02981, arXiv.org, revised Aug 2024.
    14. Bill Russell & Dooruj Rambaccussing, 2019. "Breaks and the statistical process of inflation: the case of estimating the ‘modern’ long-run Phillips curve," Empirical Economics, Springer, vol. 56(5), pages 1455-1475, May.
    15. Fiteni, Inmaculada, 2004. "[tau]-estimators of regression models with structural change of unknown location," Journal of Econometrics, Elsevier, vol. 119(1), pages 19-44, March.
    16. Yamamoto, Yohei & Tanaka, Shinya, 2015. "Testing for factor loading structural change under common breaks," Journal of Econometrics, Elsevier, vol. 189(1), pages 187-206.
    17. Oka, Tatsushi & Perron, Pierre, 2018. "Testing for common breaks in a multiple equations system," Journal of Econometrics, Elsevier, vol. 204(1), pages 66-85.
    18. Clements, Michael & Smith, Jeremy, 1997. "Forecasting Seasonal Uk Consumption Components," Economic Research Papers 268761, University of Warwick - Department of Economics.
    19. Foster, Neil & Stehrer, Robert, 2007. "Modeling transformation in CEECs using smooth transitions," Journal of Comparative Economics, Elsevier, vol. 35(1), pages 57-86, March.
    20. Kurozumi, Eiji, 2002. "Testing for stationarity with a break," Journal of Econometrics, Elsevier, vol. 108(1), pages 63-99, May.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    JEL classification:

    • C22 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bos:wpaper:wp2014-003. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Program Coordinator (email available below). General contact details of provider: https://edirc.repec.org/data/decbuus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.