IDEAS home Printed from https://ideas.repec.org/p/wrk/wrkemf/10.html
   My bibliography  Save this paper

A comprehensive evaluation of macroeconomic forecasting methods

Author

Listed:
  • Andrea Carriero

    (Queen Mary University of London)

  • Galvao, Ana Beatriz

    (University of Warwick)

  • Kapetanios, George

    (King's College London)

Abstract

This paper contributes to the academic literature and the practice of macroeconomic forecasting. Our evaluation compares the performance of four classes of state-of-art forecasting models : Factor-Augmented Distributed Lag (FADL) Models, Mixed Data Sampling (MIDAS) Models, Bayesian Vector Autoregressive (BVAR) Models and a medium-sized Dynamic Stochastic General Equilibrium Model (DSGE). We look at these models to predict output growth and inflation with datasets from the US, UK, Euro Area, Germany, France, Italy and Japan. We evaluate the accuracy of point and density forecasts, and compare models with a large set of predictors with models that employ a medium-sized dataset. Our empirical results shed light on how the predictive ability of economic indicators for output growth and inflation changes with horizon, on the impact of dataset size on the calibration of density forecasts, and how the choice of the multivariate forecasting model depends on the forecasting horizon.

Suggested Citation

  • Andrea Carriero & Galvao, Ana Beatriz & Kapetanios, George, 2016. "A comprehensive evaluation of macroeconomic forecasting methods," EMF Research Papers 10, Economic Modelling and Forecasting Group.
  • Handle: RePEc:wrk:wrkemf:10
    as

    Download full text from publisher

    File URL: https://warwick.ac.uk/fac/soc/wbs/subjects/finance/mpf/working-papers/esrc_paper_8.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Gary M. Koop, 2013. "Forecasting with Medium and Large Bayesian VARS," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 28(2), pages 177-203, March.
    2. Marta Bańbura, 2008. "Large Bayesian VARs," 2008 Meeting Papers 334, Society for Economic Dynamics.
    3. Diebold, Francis X. & Schorfheide, Frank & Shin, Minchul, 2017. "Real-time forecast evaluation of DSGE models with stochastic volatility," Journal of Econometrics, Elsevier, vol. 201(2), pages 322-332.
    4. Berkowitz, Jeremy, 2001. "Testing Density Forecasts, with Applications to Risk Management," Journal of Business & Economic Statistics, American Statistical Association, vol. 19(4), pages 465-474, October.
    5. Jan J. J. Groen & George Kapetanios, 2013. "Model Selection Criteria for Factor-Augmented Regressions-super-," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 75(1), pages 37-63, February.
    6. Warne, Anders & Coenen, Günter & Christoffel, Kai, 2008. "The new area-wide model of the euro area: a micro-founded open-economy model for forecasting and policy analysis," Working Paper Series 944, European Central Bank.
    7. Andrew J. Patton & Allan Timmermann, 2011. "Predictability of Output Growth and Inflation: A Multi-Horizon Survey Approach," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(3), pages 397-410, July.
    8. Ferrara, Laurent & Marcellino, Massimiliano & Mogliani, Matteo, 2015. "Macroeconomic forecasting during the Great Recession: The return of non-linearity?," International Journal of Forecasting, Elsevier, vol. 31(3), pages 664-679.
    9. Frank Schorfheide & Dongho Song, 2015. "Real-Time Forecasting With a Mixed-Frequency VAR," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 33(3), pages 366-380, July.
    10. Marta Banbura & Domenico Giannone & Lucrezia Reichlin, 2010. "Large Bayesian vector auto regressions," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(1), pages 71-92.
    11. Bańbura, Marta & Giannone, Domenico & Modugno, Michele & Reichlin, Lucrezia, 2013. "Now-Casting and the Real-Time Data Flow," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 195-237, Elsevier.
    12. Maik H. Wolters, 2015. "Evaluating Point and Density Forecasts of DSGE Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 30(1), pages 74-96, January.
    13. Michael P. Clements, 2004. "Evaluating the Bank of England Density Forecasts of Inflation," Economic Journal, Royal Economic Society, vol. 114(498), pages 844-866, October.
    14. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    15. Rossi, Barbara & Sekhposyan, Tatevik, 2014. "Evaluating predictive densities of US output growth and inflation in a large macroeconomic data set," International Journal of Forecasting, Elsevier, vol. 30(3), pages 662-682.
    16. Wolters, Maik H., 2011. "Forecasting under Model Uncertainty," VfS Annual Conference 2011 (Frankfurt, Main): The Order of the World Economy - Lessons from the Crisis 48723, Verein für Socialpolitik / German Economic Association.
    17. Andrea Carriero & Todd E. Clark & Massimiliano Marcellino, 2015. "Bayesian VARs: Specification Choices and Forecast Accuracy," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 30(1), pages 46-73, January.
    18. Koop, Gary & Korobilis, Dimitris, 2013. "Large time-varying parameter VARs," Journal of Econometrics, Elsevier, vol. 177(2), pages 185-198.
    19. Rochelle M. Edge & Refet S. Gürkaynak, 2011. "How useful are estimated DSGE model forecasts?," Finance and Economics Discussion Series 2011-11, Board of Governors of the Federal Reserve System (U.S.).
    20. Herbst, Edward & Schorfheide, Frank, 2012. "Evaluating DSGE model forecasts of comovements," Journal of Econometrics, Elsevier, vol. 171(2), pages 152-166.
    21. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    22. Christopher A. Sims, 1993. "A Nine-Variable Probabilistic Macroeconomic Forecasting Model," NBER Chapters, in: Business Cycles, Indicators, and Forecasting, pages 179-212, National Bureau of Economic Research, Inc.
    23. Antonello D'Agostino & Luca Gambetti & Domenico Giannone, 2013. "Macroeconomic forecasting and structural change," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 28(1), pages 82-101, January.
    24. Ferrara, Laurent & Marcellino, Massimiliano & Mogliani, Matteo, 2015. "Macroeconomic forecasting during the Great Recession: The return of non-linearity?," International Journal of Forecasting, Elsevier, vol. 31(3), pages 664-679.
    25. James H. Stock & Mark W.Watson, 2003. "Forecasting Output and Inflation: The Role of Asset Prices," Journal of Economic Literature, American Economic Association, vol. 41(3), pages 788-829, September.
    26. Domenico Giannone & Michele Lenza & Giorgio E. Primiceri, 2015. "Prior Selection for Vector Autoregressions," The Review of Economics and Statistics, MIT Press, vol. 97(2), pages 436-451, May.
    27. Knut Are Aastveit & Claudia Foroni & Francesco Ravazzolo, 2017. "Density Forecasts With Midas Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(4), pages 783-801, June.
    28. Clements, Michael P. & Taylor, Nick, 2001. "Bootstrapping prediction intervals for autoregressive models," International Journal of Forecasting, Elsevier, vol. 17(2), pages 247-267.
    29. Chauvet, Marcelle, 1998. "An Econometric Characterization of Business Cycle Dynamics with Factor Structure and Regime Switching," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 969-996, November.
    30. Negro, Marco Del & Schorfheide, Frank, 2013. "DSGE Model-Based Forecasting," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 57-140, Elsevier.
    31. Marta Banbura & Domenico Giannone & Lucrezia Reichlin, 2010. "Large Bayesian vector auto regressions," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(1), pages 71-92.
    32. Frank Smets & Rafael Wouters, 2007. "Shocks and Frictions in US Business Cycles: A Bayesian DSGE Approach," American Economic Review, American Economic Association, vol. 97(3), pages 586-606, June.
    33. Warne, Anders & Coenen, Günter & Christoffel, Kai, 2010. "Forecasting with DSGE models," Working Paper Series 1185, European Central Bank.
    34. Todd E. Clark, 2011. "Real-Time Density Forecasts From Bayesian Vector Autoregressions With Stochastic Volatility," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(3), pages 327-341, July.
    35. Sims, Christopher A., 2000. "Using a likelihood perspective to sharpen econometric discourse: Three examples," Journal of Econometrics, Elsevier, vol. 95(2), pages 443-462, April.
    36. Raffaella Giacomini & Barbara Rossi, 2010. "Forecast comparisons in unstable environments," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 595-620.
    37. James H. Stock & Mark W. Watson, 2007. "Why Has U.S. Inflation Become Harder to Forecast?," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 39(s1), pages 3-33, February.
    38. Stock, James H & Watson, Mark W, 2002. "Macroeconomic Forecasting Using Diffusion Indexes," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(2), pages 147-162, April.
    39. Antonello D'Agostino & Paolo Surico, 2012. "A Century of Inflation Forecasts," The Review of Economics and Statistics, MIT Press, vol. 94(4), pages 1097-1106, November.
    40. Vladimir Kuzin & Massimiliano Marcellino & Christian Schumacher, 2013. "Pooling Versus Model Selection For Nowcasting Gdp With Many Predictors: Empirical Evidence For Six Industrialized Countries," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 28(3), pages 392-411, April.
    41. James Mitchell & Kenneth F. Wallis, 2011. "Evaluating density forecasts: forecast combinations, model mixtures, calibration and sharpness," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 26(6), pages 1023-1040, September.
    42. Faust, Jon & Wright, Jonathan H., 2013. "Forecasting Inflation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 2-56, Elsevier.
    43. Smets, Frank & Warne, Anders & Wouters, Rafael, 2014. "Professional forecasters and real-time forecasting with a DSGE model," International Journal of Forecasting, Elsevier, vol. 30(4), pages 981-995.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tallman, Ellis W. & Zaman, Saeed, 2020. "Combining survey long-run forecasts and nowcasts with BVAR forecasts using relative entropy," International Journal of Forecasting, Elsevier, vol. 36(2), pages 373-398.
    2. Magnus Reif, 2020. "Macroeconomics, Nonlinearities, and the Business Cycle," ifo Beiträge zur Wirtschaftsforschung, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, number 87.
    3. Raffaella Giacomini & Barbara Rossi, 2015. "Forecasting in Nonstationary Environments: What Works and What Doesn't in Reduced-Form and Structural Models," Annual Review of Economics, Annual Reviews, vol. 7(1), pages 207-229, August.
    4. Barbara Rossi, 2019. "Forecasting in the presence of instabilities: How do we know whether models predict well and how to improve them," Economics Working Papers 1711, Department of Economics and Business, Universitat Pompeu Fabra, revised Jul 2021.
    5. Knotek, Edward S. & Zaman, Saeed, 2019. "Financial nowcasts and their usefulness in macroeconomic forecasting," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1708-1724.
    6. Chauvet, Marcelle & Potter, Simon, 2013. "Forecasting Output," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 141-194, Elsevier.
    7. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    8. Tim Oliver Berg, 2016. "Multivariate Forecasting with BVARs and DSGE Models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 35(8), pages 718-740, December.
    9. Mandalinci, Zeyyad, 2017. "Forecasting inflation in emerging markets: An evaluation of alternative models," International Journal of Forecasting, Elsevier, vol. 33(4), pages 1082-1104.
    10. Berg, Tim O. & Henzel, Steffen R., 2015. "Point and density forecasts for the euro area using Bayesian VARs," International Journal of Forecasting, Elsevier, vol. 31(4), pages 1067-1095.
    11. Gael M. Martin & David T. Frazier & Ruben Loaiza-Maya & Florian Huber & Gary Koop & John Maheu & Didier Nibbering & Anastasios Panagiotelis, 2023. "Bayesian Forecasting in the 21st Century: A Modern Review," Monash Econometrics and Business Statistics Working Papers 1/23, Monash University, Department of Econometrics and Business Statistics.
    12. Bekiros Stelios & Paccagnini Alessia, 2015. "Estimating point and density forecasts for the US economy with a factor-augmented vector autoregressive DSGE model," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 19(2), pages 107-136, April.
    13. Gael M. Martin & David T. Frazier & Worapree Maneesoonthorn & Ruben Loaiza-Maya & Florian Huber & Gary Koop & John Maheu & Didier Nibbering & Anastasios Panagiotelis, 2022. "Bayesian Forecasting in Economics and Finance: A Modern Review," Papers 2212.03471, arXiv.org, revised Jul 2023.
    14. Chan, Joshua C.C., 2021. "Minnesota-type adaptive hierarchical priors for large Bayesian VARs," International Journal of Forecasting, Elsevier, vol. 37(3), pages 1212-1226.
    15. Galvão, Ana Beatriz, 2017. "Data revisions and DSGE models," Journal of Econometrics, Elsevier, vol. 196(1), pages 215-232.
    16. Carriero, Andrea & Mumtaz, Haroon & Theophilopoulou, Angeliki, 2015. "Macroeconomic information, structural change, and the prediction of fiscal aggregates," International Journal of Forecasting, Elsevier, vol. 31(2), pages 325-348.
    17. Reif Magnus, 2021. "Macroeconomic uncertainty and forecasting macroeconomic aggregates," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 25(2), pages 1-20, April.
    18. Fabian Krüger & Todd E. Clark & Francesco Ravazzolo, 2017. "Using Entropic Tilting to Combine BVAR Forecasts With External Nowcasts," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 35(3), pages 470-485, July.
    19. Anders Warne & Günter Coenen & Kai Christoffel, 2017. "Marginalized Predictive Likelihood Comparisons of Linear Gaussian State‐Space Models with Applications to DSGE, DSGE‐VAR, and VAR Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(1), pages 103-119, January.
    20. Andrea Carriero & Todd E. Clark & Massimiliano Marcellino, 2015. "Realtime nowcasting with a Bayesian mixed frequency model with stochastic volatility," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 178(4), pages 837-862, October.

    More about this item

    Keywords

    factor models ; BVAR models ; MIDAS models ; DSGE models ; density forecasts JEL Classification Numbers: C53;
    All these keywords.

    JEL classification:

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wrk:wrkemf:10. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Ana Galvão (email available below). General contact details of provider: https://edirc.repec.org/data/emwaruk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.