IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v33y2017i1p48-60.html
   My bibliography  Save this article

Model Confidence Sets and forecast combination

Author

Listed:
  • Samuels, Jon D.
  • Sekkel, Rodrigo M.

Abstract

A longstanding finding in the forecasting literature is that averaging the forecasts from a range of models often improves upon forecasts based on a single model, with equal weight averaging working particularly well. This paper analyzes the effects of trimming the set of models prior to averaging. We compare different trimming schemes and propose a new approach based on Model Confidence Sets that takes into account the statistical significance of the out-of-sample forecasting performance. In an empirical application to the forecasting of U.S. macroeconomic indicators, we find significant gains in out-of-sample forecast accuracy from using the proposed trimming method.

Suggested Citation

  • Samuels, Jon D. & Sekkel, Rodrigo M., 2017. "Model Confidence Sets and forecast combination," International Journal of Forecasting, Elsevier, vol. 33(1), pages 48-60.
  • Handle: RePEc:eee:intfor:v:33:y:2017:i:1:p:48-60
    DOI: 10.1016/j.ijforecast.2016.07.004
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169207016300747
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.ijforecast.2016.07.004?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Inoue, Atsushi & Kilian, Lutz, 2008. "How Useful Is Bagging in Forecasting Economic Time Series? A Case Study of U.S. Consumer Price Inflation," Journal of the American Statistical Association, American Statistical Association, vol. 103, pages 511-522, June.
    2. Wright, Jonathan H., 2008. "Bayesian Model Averaging and exchange rate forecasts," Journal of Econometrics, Elsevier, vol. 146(2), pages 329-341, October.
    3. Peter R. Hansen & Asger Lunde & James M. Nason, 2011. "The Model Confidence Set," Econometrica, Econometric Society, vol. 79(2), pages 453-497, March.
    4. Jonathan H. Wright, 2009. "Forecasting US inflation by Bayesian model averaging," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 28(2), pages 131-144.
    5. Bernanke, Ben S. & Boivin, Jean, 2003. "Monetary policy in a data-rich environment," Journal of Monetary Economics, Elsevier, vol. 50(3), pages 525-546, April.
    6. Faust, Jon & Wright, Jonathan H., 2009. "Comparing Greenbook and Reduced Form Forecasts Using a Large Realtime Dataset," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(4), pages 468-479.
    7. Timmermann, Allan, 2006. "Forecast Combinations," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 4, pages 135-196, Elsevier.
    8. Peter Tulip, 2009. "Has the Economy Become More Predictable? Changes in Greenbook Forecast Accuracy," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 41(6), pages 1217-1231, September.
    9. Marco Aiolfi & Carlo Ambrogio Favero, "undated". "Model Uncertainty, Thick Modelling and the predictability of Stock Returns," Working Papers 221, IGIER (Innocenzo Gasparini Institute for Economic Research), Bocconi University.
    10. Jon Faust & Simon Gilchrist & Jonathan H. Wright & Egon Zakrajšsek, 2013. "Credit Spreads as Predictors of Real-Time Economic Activity: A Bayesian Model-Averaging Approach," The Review of Economics and Statistics, MIT Press, vol. 95(5), pages 1501-1519, December.
    11. Hansen, Bruce E., 2008. "Least-squares forecast averaging," Journal of Econometrics, Elsevier, vol. 146(2), pages 342-350, October.
    12. Pérez-Quirós, Gabriel & Camacho, Máximo & Alvarez, Rocio, 2012. "Finite sample performance of small versus large scale dynamic factor models," CEPR Discussion Papers 8867, C.E.P.R. Discussion Papers.
    13. Capistrán, Carlos & Timmermann, Allan, 2009. "Forecast Combination With Entry and Exit of Experts," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(4), pages 428-440.
    14. Clemon, Robert T & Winkler, Robert L, 1986. "Combining Economic Forecasts," Journal of Business & Economic Statistics, American Statistical Association, vol. 4(1), pages 39-46, January.
    15. Clemen, Robert T., 1989. "Combining forecasts: A review and annotated bibliography," International Journal of Forecasting, Elsevier, vol. 5(4), pages 559-583.
    16. Spyros Makridakis & Robert L. Winkler, 1983. "Averages of Forecasts: Some Empirical Results," Management Science, INFORMS, vol. 29(9), pages 987-996, September.
    17. James H. Stock & Mark W. Watson, 2007. "Why Has U.S. Inflation Become Harder to Forecast?," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 39(s1), pages 3-33, February.
    18. Stock, James H & Watson, Mark W, 2002. "Macroeconomic Forecasting Using Diffusion Indexes," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(2), pages 147-162, April.
    19. David F. Hendry & Michael P. Clements, 2004. "Pooling of forecasts," Econometrics Journal, Royal Economic Society, vol. 7(1), pages 1-31, June.
    20. Aiolfi, Marco & Timmermann, Allan, 2006. "Persistence in forecasting performance and conditional combination strategies," Journal of Econometrics, Elsevier, vol. 135(1-2), pages 31-53.
    21. G. Elliott & C. Granger & A. Timmermann (ed.), 2006. "Handbook of Economic Forecasting," Handbook of Economic Forecasting, Elsevier, edition 1, volume 1, number 1.
    22. Mark W. Watson & James H. Stock, 2004. "Combination forecasts of output growth in a seven-country data set," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 23(6), pages 405-430.
    23. Bruce E. Hansen, 2007. "Least Squares Model Averaging," Econometrica, Econometric Society, vol. 75(4), pages 1175-1189, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bravo, Jorge M. & Ayuso, Mercedes & Holzmann, Robert & Palmer, Edward, 2021. "Addressing the life expectancy gap in pension policy," Insurance: Mathematics and Economics, Elsevier, vol. 99(C), pages 200-221.
    2. Roccazzella, Francesco & Gambetti, Paolo & Vrins, Frédéric, 2022. "Optimal and robust combination of forecasts via constrained optimization and shrinkage," International Journal of Forecasting, Elsevier, vol. 38(1), pages 97-116.
    3. Afanasyev, Dmitriy O. & Fedorova, Elena A., 2019. "On the impact of outlier filtering on the electricity price forecasting accuracy," Applied Energy, Elsevier, vol. 236(C), pages 196-210.
    4. WenJie Wang & Qi Xu & Dandan Fan, 2018. "Stein-Rule Combination Forecasting on RFID Based Supply Chain," Asia-Pacific Journal of Operational Research (APJOR), World Scientific Publishing Co. Pte. Ltd., vol. 35(02), pages 1-13, April.
    5. Amendola, Alessandra & Braione, Manuela & Candila, Vincenzo & Storti, Giuseppe, 2020. "A Model Confidence Set approach to the combination of multivariate volatility forecasts," International Journal of Forecasting, Elsevier, vol. 36(3), pages 873-891.
    6. James Younker, 2022. "Calculating Effective Degrees of Freedom for Forecast Combinations and Ensemble Models," Discussion Papers 2022-19, Bank of Canada.
    7. Chiang, I-Hsuan Ethan & Liao, Yin & Zhou, Qing, 2021. "Modeling the cross-section of stock returns using sensible models in a model pool," Journal of Empirical Finance, Elsevier, vol. 60(C), pages 56-73.
    8. Wei, Yu & Liang, Chao & Li, Yan & Zhang, Xunhui & Wei, Guiwu, 2020. "Can CBOE gold and silver implied volatility help to forecast gold futures volatility in China? Evidence based on HAR and Ridge regression models," Finance Research Letters, Elsevier, vol. 35(C).
    9. Chao Liang & Yongan Xu & Zhonglu Chen & Xiafei Li, 2023. "Forecasting China's stock market volatility with shrinkage method: Can Adaptive Lasso select stronger predictors from numerous predictors?," International Journal of Finance & Economics, John Wiley & Sons, Ltd., vol. 28(4), pages 3689-3699, October.
    10. Wenting Zhao & Juanjuan Zhao & Xilong Yao & Zhixin Jin & Pan Wang, 2019. "A Novel Adaptive Intelligent Ensemble Model for Forecasting Primary Energy Demand," Energies, MDPI, vol. 12(7), pages 1-28, April.
    11. Carlos Henrique Dias Cordeiro de Castro & Fernando Antonio Lucena Aiube, 2023. "Forecasting inflation time series using score‐driven dynamic models and combination methods: The case of Brazil," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(2), pages 369-401, March.
    12. Mercedes Ayuso & Jorge M. Bravo & Robert Holzmann & Edward Palmer, 2021. "Automatic Indexation of the Pension Age to Life Expectancy: When Policy Design Matters," Risks, MDPI, vol. 9(5), pages 1-28, May.
    13. Jun Hao & Xiaolei Sun & Qianqian Feng, 2020. "A Novel Ensemble Approach for the Forecasting of Energy Demand Based on the Artificial Bee Colony Algorithm," Energies, MDPI, vol. 13(3), pages 1-25, January.
    14. Diebold, Francis X. & Shin, Minchul, 2019. "Machine learning for regularized survey forecast combination: Partially-egalitarian LASSO and its derivatives," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1679-1691.
    15. Garcia, Márcio G.P. & Medeiros, Marcelo C. & Vasconcelos, Gabriel F.R., 2017. "Real-time inflation forecasting with high-dimensional models: The case of Brazil," International Journal of Forecasting, Elsevier, vol. 33(3), pages 679-693.
    16. Ding, Lili & Zhao, Zhongchao & Wang, Lei, 2022. "Probability density forecasts for natural gas demand in China: Do mixed-frequency dynamic factors matter?," Applied Energy, Elsevier, vol. 312(C).
    17. Daniel Borup & Jonas N. Eriksen & Mads M. Kjær & Martin Thyrsgaard, 2020. "Predicting bond return predictability," CREATES Research Papers 2020-09, Department of Economics and Business Economics, Aarhus University.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jon D. Samuels & Rodrigo Sekkel, 2013. "Forecasting with Many Models: Model Confidence Sets and Forecast Combination," Staff Working Papers 13-11, Bank of Canada.
    2. Timmermann, Allan, 2006. "Forecast Combinations," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 4, pages 135-196, Elsevier.
    3. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    4. Cheng, Xu & Hansen, Bruce E., 2015. "Forecasting with factor-augmented regression: A frequentist model averaging approach," Journal of Econometrics, Elsevier, vol. 186(2), pages 280-293.
    5. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    6. Kim, Hyun Hak & Swanson, Norman R., 2018. "Mining big data using parsimonious factor, machine learning, variable selection and shrinkage methods," International Journal of Forecasting, Elsevier, vol. 34(2), pages 339-354.
    7. Diebold, Francis X. & Shin, Minchul, 2019. "Machine learning for regularized survey forecast combination: Partially-egalitarian LASSO and its derivatives," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1679-1691.
    8. Xiaojie Xu, 2020. "Corn Cash Price Forecasting," American Journal of Agricultural Economics, John Wiley & Sons, vol. 102(4), pages 1297-1320, August.
    9. Soybilgen, Barış & Yazgan, Ege, 2018. "Evaluating nowcasts of bridge equations with advanced combination schemes for the Turkish unemployment rate," Economic Modelling, Elsevier, vol. 72(C), pages 99-108.
    10. Chan, Felix & Pauwels, Laurent L., 2018. "Some theoretical results on forecast combinations," International Journal of Forecasting, Elsevier, vol. 34(1), pages 64-74.
    11. Knut Are Aastveit & Karsten R. Gerdrup & Anne Sofie Jore & Leif Anders Thorsrud, 2014. "Nowcasting GDP in Real Time: A Density Combination Approach," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 32(1), pages 48-68, January.
    12. Genre, Véronique & Kenny, Geoff & Meyler, Aidan & Timmermann, Allan, 2013. "Combining expert forecasts: Can anything beat the simple average?," International Journal of Forecasting, Elsevier, vol. 29(1), pages 108-121.
    13. Conflitti, Cristina & De Mol, Christine & Giannone, Domenico, 2015. "Optimal combination of survey forecasts," International Journal of Forecasting, Elsevier, vol. 31(4), pages 1096-1103.
    14. Bec, Frédérique & Mogliani, Matteo, 2015. "Nowcasting French GDP in real-time with surveys and “blocked” regressions: Combining forecasts or pooling information?," International Journal of Forecasting, Elsevier, vol. 31(4), pages 1021-1042.
    15. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    16. Bayer, Sebastian, 2018. "Combining Value-at-Risk forecasts using penalized quantile regressions," Econometrics and Statistics, Elsevier, vol. 8(C), pages 56-77.
    17. Christopher G. Gibbs, 2017. "Forecast combination, non-linear dynamics, and the macroeconomy," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 63(3), pages 653-686, March.
    18. Hansen, Bruce E., 2008. "Least-squares forecast averaging," Journal of Econometrics, Elsevier, vol. 146(2), pages 342-350, October.
    19. Groen, Jan J.J. & Kapetanios, George, 2016. "Revisiting useful approaches to data-rich macroeconomic forecasting," Computational Statistics & Data Analysis, Elsevier, vol. 100(C), pages 221-239.
    20. Carlos Henrique Dias Cordeiro de Castro & Fernando Antonio Lucena Aiube, 2023. "Forecasting inflation time series using score‐driven dynamic models and combination methods: The case of Brazil," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(2), pages 369-401, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:33:y:2017:i:1:p:48-60. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.