IDEAS home Printed from https://ideas.repec.org/a/jae/japmet/v25y2010i4p574-594.html
   My bibliography  Save this article

Forecast evaluation of small nested model sets

Author

Listed:
  • Kirstin Hubrich

    (European Central Bank, Frankfurt, Germany)

  • Kenneth D. West

    (Department of Economics, University of Wisconsin, Madison, WI, USA)

Abstract

We propose two new procedures for comparing the mean squared prediction error (MSPE) of a benchmark model to the MSPEs of a small set of alternative models that nest the benchmark. Our procedures compare the benchmark to all the alternative models simultaneously rather than sequentially, and do not require re-estimation of models as part of a bootstrap procedure. Both procedures adjust MSPE differences in accordance with Clark and West (2007); one procedure then examines the maximum t-statistic, while the other computes a chi-squared statistic. Our simulations examine the proposed procedures and two existing procedures that do not adjust the MSPE differences: a chi-squared statistic and White's (2000) reality check. In these simulations, the two statistics that adjust MSPE differences have the most accurate size, and the procedure that looks at the maximum t-statistic has the best power. We illustrate our procedures by comparing forecasts of different models for US inflation. Copyright © 2010 John Wiley & Sons, Ltd.

Suggested Citation

  • Kirstin Hubrich & Kenneth D. West, 2010. "Forecast evaluation of small nested model sets," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 574-594.
  • Handle: RePEc:jae:japmet:v:25:y:2010:i:4:p:574-594
    DOI: 10.1002/jae.1176
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1002/jae.1176
    File Function: Link to full text; subscription required
    Download Restriction: no

    File URL: http://qed.econ.queensu.ca:80/jae/2010-v25.4/
    File Function: Supporting data files and programs
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Orphanides, Athanasios & van Norden, Simon, 2005. "The Reliability of Inflation Forecasts Based on Output Gap Estimates in Real Time," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 37(3), pages 583-601, June.
    2. Kirstin Hubrich & David F. Hendry, 2005. "Forecasting Aggregates by Disaggregates," Computing in Economics and Finance 2005 270, Society for Computational Economics.
    3. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    4. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.
    5. Atsushi Inoue & Lutz Kilian, 2005. "In-Sample or Out-of-Sample Tests of Predictability: Which One Should We Use?," Econometric Reviews, Taylor & Francis Journals, vol. 23(4), pages 371-402.
    6. Clark, Todd E. & West, Kenneth D., 2006. "Using out-of-sample mean squared prediction errors to test the martingale difference hypothesis," Journal of Econometrics, Elsevier, vol. 135(1-2), pages 155-186.
    7. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    8. Hansen, Peter Reinhard, 2005. "A Test for Superior Predictive Ability," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 365-380, October.
    9. West, Kenneth D. & Cho, Dongchul, 1995. "The predictive ability of several models of exchange rate volatility," Journal of Econometrics, Elsevier, vol. 69(2), pages 367-391, October.
    10. West, Kenneth D. & Edison, Hali J. & Cho, Dongchul, 1993. "A utility-based comparison of some models of exchange rate volatility," Journal of International Economics, Elsevier, vol. 35(1-2), pages 23-45, August.
    11. Hendry, David F. & Hubrich, Kirstin, 2011. "Combining Disaggregate Forecasts or Combining Disaggregate Information to Forecast an Aggregate," Journal of Business & Economic Statistics, American Statistical Association, vol. 29(2), pages 216-227.
    12. Todd Clark & Michael McCracken, 2005. "Evaluating Direct Multistep Forecasts," Econometric Reviews, Taylor & Francis Journals, vol. 24(4), pages 369-404.
    13. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    14. David Harvey & Paul Newbold, 2000. "Tests for multiple forecast encompassing," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 15(5), pages 471-482.
    15. Sarno, Lucio & Thornton, Daniel L & Valente, Giorgio, 2005. "Federal Funds Rate Prediction," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 37(3), pages 449-471, June.
    16. Clark, Todd E. & West, Kenneth D., 2007. "Approximately normal tests for equal predictive accuracy in nested models," Journal of Econometrics, Elsevier, vol. 138(1), pages 291-311, May.
    17. Inoue, Atsushi & Kilian, Lutz, 2006. "On the selection of forecasting models," Journal of Econometrics, Elsevier, vol. 130(2), pages 273-306, February.
    18. West, Kenneth D, 2001. "Tests for Forecast Encompassing When Forecasts Depend on Estimated Regression Parameters," Journal of Business & Economic Statistics, American Statistical Association, vol. 19(1), pages 29-33, January.
    19. Ashley, R & Granger, C W J & Schmalensee, R, 1980. "Advertising and Aggregate Consumption: An Analysis of Causality," Econometrica, Econometric Society, vol. 48(5), pages 1149-1167, July.
    20. Rapach, David E. & Wohar, Mark E., 2006. "In-sample vs. out-of-sample tests of stock return predictability in the context of data mining," Journal of Empirical Finance, Elsevier, vol. 13(2), pages 231-247, March.
    21. Todd E. Clark & Michael W. McCracken, 2010. "Reality checks and nested forecast model comparisons," Working Papers 2010-032, Federal Reserve Bank of St. Louis.
    22. Yongmiao Hong & Tae-Hwy Lee, 2003. "Inference on Predictability of Foreign Exchange Rates via Generalized Spectrum and Nonlinear Time Series Models," The Review of Economics and Statistics, MIT Press, vol. 85(4), pages 1048-1062, November.
    23. James H. Stock & Mark W. Watson, 2007. "Why Has U.S. Inflation Become Harder to Forecast?," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 39(s1), pages 3-33, February.
    24. Hubrich, Kirstin, 2005. "Forecasting euro area inflation: Does aggregating forecasts by HICP component improve forecast accuracy?," International Journal of Forecasting, Elsevier, vol. 21(1), pages 119-136.
    25. McCracken, Michael W., 2007. "Asymptotics for out of sample tests of Granger causality," Journal of Econometrics, Elsevier, vol. 140(2), pages 719-752, October.
    26. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    27. Lutkepohl, Helmut, 1984. "Forecasting Contemporaneously Aggregated Vector ARMA Processes," Journal of Business & Economic Statistics, American Statistical Association, vol. 2(3), pages 201-214, July.
    28. Andrew Atkeson & Lee E. Ohanian, 2001. "Are Phillips curves useful for forecasting inflation?," Quarterly Review, Federal Reserve Bank of Minneapolis, issue Win, pages 2-11.
    29. Andreas Billmeier, 2004. "Ghostbusting; Which Output Gap Measure Really Matters?," IMF Working Papers 04/146, International Monetary Fund.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ivana Komunjer & Michael T. Owyang, 2012. "Multivariate Forecast Evaluation and Rationality Testing," The Review of Economics and Statistics, MIT Press, vol. 94(4), pages 1066-1080, November.
    2. D'Amuri, Francesco & Marcucci, Juri, 2009. "'Google it!' Forecasting the US unemployment rate with a Google job search index," ISER Working Paper Series 2009-32, Institute for Social and Economic Research.
    3. Bekaert, Geert & Hoerova, Marie & Lo Duca, Marco, 2013. "Risk, uncertainty and monetary policy," Journal of Monetary Economics, Elsevier, vol. 60(7), pages 771-788.
    4. Kollmann, Robert & Zeugner, Stefan, 2012. "Leverage as a predictor for real activity and volatility," Journal of Economic Dynamics and Control, Elsevier, vol. 36(8), pages 1267-1283.
    5. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, Elsevier.
    6. Francesco Ravazzolo & Philip Rothman, 2013. "Oil and U.S. GDP: A Real‐Time Out‐of‐Sample Examination," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 45(2-3), pages 449-463, March.
    7. Daniel Andrés Jaimes Cárdenas & Jair Ojeda Joya, 2010. "Reglas de Taylor y previsibilidad fuera de muestra de la tasa de cambio en Latinoamérica," Borradores de Economia 619, Banco de la Republica de Colombia.
    8. Todd E. Clark & Michael W. McCracken, 2010. "Testing for unconditional predictive ability," Working Papers 2010-031, Federal Reserve Bank of St. Louis.
    9. Todd E. Clark & Michael W. McCracken, 2010. "Reality checks and nested forecast model comparisons," Working Papers 2010-032, Federal Reserve Bank of St. Louis.
    10. Aron, Janine & Muellbauer, John, 2012. "Improving forecasting in an emerging economy, South Africa: Changing trends, long run restrictions and disaggregation," International Journal of Forecasting, Elsevier, vol. 28(2), pages 456-476.
    11. Hendry, David F. & Hubrich, Kirstin, 2011. "Combining Disaggregate Forecasts or Combining Disaggregate Information to Forecast an Aggregate," Journal of Business & Economic Statistics, American Statistical Association, vol. 29(2), pages 216-227.
    12. Günter Coenen & Juha Kilponen & Mathias Trabandt, 2010. "When does fiscal stimulus work?," Research Bulletin, European Central Bank, vol. 10, pages 6-10.
    13. Khalaf, Lynda & Saunders, Charles J., 2017. "Monte Carlo forecast evaluation with persistent data," International Journal of Forecasting, Elsevier, vol. 33(1), pages 1-10.
    14. repec:eee:jbfina:v:84:y:2017:i:c:p:188-201 is not listed on IDEAS
    15. Molodtsova, Tanya & Papell, David H., 2009. "Out-of-sample exchange rate predictability with Taylor rule fundamentals," Journal of International Economics, Elsevier, vol. 77(2), pages 167-180, April.
    16. Biswas, Anindya, 2014. "The output gap and expected security returns," Review of Financial Economics, Elsevier, vol. 23(3), pages 131-140.
    17. Ana Lamo & Frank Smets, 2010. "Wage dynamics in Europe: some new findings," Research Bulletin, European Central Bank, vol. 10, pages 2-5.
    18. Granziera, Eleonora & Hubrich, Kirstin & Moon, Hyungsik Roger, 2014. "A predictability test for a small number of nested models," Journal of Econometrics, Elsevier, vol. 182(1), pages 174-185.
    19. Daniel Borup & Martin Thyrsgaard, 1705. "Statistical tests for equal predictive ability across multiple forecasting methods," CREATES Research Papers 2017-19, Department of Economics and Business Economics, Aarhus University.
    20. Busetti, Fabio & Marcucci, Juri, 2013. "Comparing forecast accuracy: A Monte Carlo investigation," International Journal of Forecasting, Elsevier, vol. 29(1), pages 13-27.
    21. Mariano, Roberto S. & Preve, Daniel, 2012. "Statistical tests for multiple forecast comparison," Journal of Econometrics, Elsevier, vol. 169(1), pages 123-130.

    More about this item

    JEL classification:

    • C32 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes; State Space Models
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods
    • E37 - Macroeconomics and Monetary Economics - - Prices, Business Fluctuations, and Cycles - - - Forecasting and Simulation: Models and Applications

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:jae:japmet:v:25:y:2010:i:4:p:574-594. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Wiley-Blackwell Digital Licensing) or (Christopher F. Baum). General contact details of provider: http://www.interscience.wiley.com/jpages/0883-7252/ .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.