A Bunch of Models, a Bunch of Nulls and Inference About Predictive Ability
AbstractInference about predictive ability is usually carried out in the form of pairwise comparisons between two competing forecasting methods. Nevertheless, some interesting questions are concerned with families of models and not just with a couple of forecasting strategies. An example of this would be the question about the predictive accuracy of pure time-series models versus models based on economic fundamentals. It is clear that an appropriate answer to this question requires comparing families of models, which may include a number of different forecasting strategies. Another usual approach in the literature consists of comparing the accuracy of a new forecasting method with a natural benchmark. Nevertheless, unless the econometrician is completely sure about the superiority of the benchmark over the rest of the methods available in the literature, he/she may want to compare the accuracy of his/her new forecasting model, and its extensions, against a broader set of methods. In this article we present a simple methodology to test the null hypothesis of equal predictive ability between two families of forecasting methods. Our approach corresponds to a natural extension of the White (2000) reality check in which we allow for the families being compared to be populated by a large number of forecasting methods. We illustrate our testing approach with an empirical application comparing the ability of two families of models to predict headline inflation in Chile, the US, Sweden and Mexico. With this illustration we show that comparing families of models using the usual approach based on pairwise comparisons of the best ex-post performing models in each family, may lead to conclusions that are at odds with those suggested by our approach.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
Bibliographic InfoPaper provided by Central Bank of Chile in its series Working Papers Central Bank of Chile with number 607.
Date of creation: Jan 2011
Date of revision:
Other versions of this item:
- Pincheira, Pablo, 2013. "A Bunch of Models, a Bunch of Nulls and Inference about Predictive Ability," Journal for Economic Forecasting, Institute for Economic Forecasting, vol. 0(3), pages 26-43, October.
- C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
- C22 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models &bull Diffusion Processes
- C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
- C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods
- E3 - Macroeconomics and Monetary Economics - - Prices, Business Fluctuations, and Cycles
- E4 - Macroeconomics and Monetary Economics - - Money and Interest Rates
- F31 - International Economics - - International Finance - - - Foreign Exchange
This paper has been announced in the following NEP Reports:
- NEP-ALL-2011-04-23 (All new papers)
- NEP-CBA-2011-04-23 (Central Banking)
- NEP-ECM-2011-04-23 (Econometrics)
- NEP-FOR-2011-04-23 (Forecasting)
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- West, Kenneth D, 1996.
"Asymptotic Inference about Predictive Ability,"
Econometric Society, vol. 64(5), pages 1067-84, September.
- Andrew Atkeson & Lee E. Ohanian., 2001. "Are Phillips curves useful for forecasting inflation?," Quarterly Review, Federal Reserve Bank of Minneapolis, issue Win, pages 2-11.
- Diebold, Francis X & Mariano, Roberto S, 1995.
"Comparing Predictive Accuracy,"
Journal of Business & Economic Statistics,
American Statistical Association, vol. 13(3), pages 253-63, July.
- Tom Doan, . "DMARIANO: RATS procedure to compute Diebold-Mariano Forecast Comparison Test," Statistical Software Components RTS00055, Boston College Department of Economics.
- Francis X. Diebold & Robert S. Mariano, 1994. "Comparing Predictive Accuracy," NBER Technical Working Papers 0169, National Bureau of Economic Research, Inc.
- Raffaella Giacomini & Halbert White, 2003.
"Tests of conditional predictive ability,"
Boston College Working Papers in Economics
572, Boston College Department of Economics.
- Giacomini, Raffaella & White, Halbert, 2003. "Tests of Conditional Predictive Ability," University of California at San Diego, Economics Working Paper Series qt5jk0j5jh, Department of Economics, UC San Diego.
- Raffaella Giacomini & Halbert White, 2003. "Tests of Conditional Predictive Ability," Econometrics 0308001, EconWPA.
- Ghysels, Eric & Osborn, Denise R. & Rodrigues, Paulo M.M., 2006. "Forecasting Seasonal Time Series," Handbook of Economic Forecasting, Elsevier.
- Hansen, Peter Reinhard, 2005. "A Test for Superior Predictive Ability," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 365-380, October.
- Croushore Dean, 2010.
"An Evaluation of Inflation Forecasts from Surveys Using Real-Time Data,"
The B.E. Journal of Macroeconomics,
De Gruyter, vol. 10(1), pages 1-32, May.
- Dean Croushore, 2006. "An evaluation of inflation forecasts from surveys using real-time data," Working Papers 06-19, Federal Reserve Bank of Philadelphia.
- James H. Stock & Mark W. Watson, 2008.
"Phillips Curve Inflation Forecasts,"
NBER Working Papers
14322, National Bureau of Economic Research, Inc.
- West, Kenneth D., 2006. "Forecast Evaluation," Handbook of Economic Forecasting, Elsevier.
- Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
- Groen, Jan J.J. & Kapetanios, George & Price, Simon, 2009. "A real time evaluation of Bank of England forecasts of inflation and growth," International Journal of Forecasting, Elsevier, vol. 25(1), pages 74-80.
- Pincheira, Pablo & García, Álvaro, 2012. "En busca de un buen marco de referencia predictivo para la inflación en Chile," El Trimestre Económico, Fondo de Cultura Económica, vol. 0(313), pages 85-123, enero-mar.
- Carlos A. Medel, 2013.
"How informative are in-sample information criteria to forecasting? The case of Chilean GDP,"
Latin American Journal of Economics-formerly Cuadernos de Economía,
Instituto de Economía. Pontificia Universidad Católica de Chile., vol. 50(1), pages 133-161, May.
- Carlos Medel, 2012. "How Informative are In–Sample Information Criteria to Forecasting? The Case of Chilean GDP," Working Papers Central Bank of Chile 657, Central Bank of Chile.
- Medel, Carlos A., 2012. "How informative are in-sample information criteria to forecasting? the case of Chilean GDP," MPRA Paper 35949, University Library of Munich, Germany.
- Pablo Pincheira & Carlos A. Medel, 2012. "Forecasting Inflation with a Simple and Accurate Benchmark: a Cross-Country Analysis," Working Papers Central Bank of Chile 677, Central Bank of Chile.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Claudio Sepulveda).
If references are entirely missing, you can add them using this form.