IDEAS home Printed from https://ideas.repec.org/p/cir/cirwor/2008s-28.html
   My bibliography  Save this paper

The Calibration of Probabilistic Economic Forecasts

Author

Listed:
  • John W. Galbraith
  • Simon van Norden

Abstract

A probabilistic forecast is the estimated probability with which a future event will satisfy a particular criterion. One interesting feature of such forecasts is their calibration, or the match between predicted probabilities and actual outcome probabilities. Calibration has been evaluated in the past by gropuing probability forecasts into discrete categories. Here we show that we can do so without discrete groupings; the kernel estimators that we use produce efficiency gains and smooth estimated curves relating predicted and actual probabilities. We use such estimates to evaluate the empirical evidence on calibration error in a number of economic applications including recession and inflation prediction, using both forecasts made and stored in real time and pseudo-forecasts made using the data vintage available at the forecast date. We evaluate outcomes using both first-release outcome measures as well as later, thoroughly revised data. We find strong evidence of incorrect calibration in professional forecasts of recessions and inflation. We also present evidence of asymmetries in the performace of inflation forecasts based on real-time output gaps.
(This abstract was borrowed from another version of this item.)

Suggested Citation

  • John W. Galbraith & Simon van Norden, 2008. "The Calibration of Probabilistic Economic Forecasts," CIRANO Working Papers 2008s-28, CIRANO.
  • Handle: RePEc:cir:cirwor:2008s-28
    as

    Download full text from publisher

    File URL: https://cirano.qc.ca/files/publications/2008s-28.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Orphanides, Athanasios & van Norden, Simon, 2005. "The Reliability of Inflation Forecasts Based on Output Gap Estimates in Real Time," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 37(3), pages 583-601, June.
    2. Athanasios Orphanides & Simon van Norden, 2002. "The Unreliability of Output-Gap Estimates in Real Time," The Review of Economics and Statistics, MIT Press, vol. 84(4), pages 569-583, November.
    3. Enno Mammen, "undated". "Comparing nonparametric versus parametric regression fits," Statistic und Oekonometrie 9205, Humboldt Universitaet Berlin.
    4. Terasvirta, T & Anderson, H M, 1992. "Characterizing Nonlinearities in Business Cycles Using Smooth Transition Autoregressive Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 7(S), pages 119-136, Suppl. De.
    5. Pagan,Adrian & Ullah,Aman, 1999. "Nonparametric Econometrics," Cambridge Books, Cambridge University Press, number 9780521355643, October.
    6. John W. Galbraith & Greg Tkacz, 2007. "Forecast content and content horizons for some important macroeconomic time series," Canadian Journal of Economics, Canadian Economics Association, vol. 40(3), pages 935-953, August.
    7. Dean Croushore & Tom Stark, 2003. "A Real-Time Data Set for Macroeconomists: Does the Data Vintage Matter?," The Review of Economics and Statistics, MIT Press, vol. 85(3), pages 605-617, August.
    8. Ang, Andrew & Bekaert, Geert & Wei, Min, 2007. "Do macro variables, asset markets, or surveys forecast inflation better?," Journal of Monetary Economics, Elsevier, vol. 54(4), pages 1163-1212, May.
    9. P. M. Robinson, 1983. "Nonparametric Estimators For Time Series," Journal of Time Series Analysis, Wiley Blackwell, vol. 4(3), pages 185-207, May.
    10. Tilmann Gneiting & Fadoua Balabdaoui & Adrian E. Raftery, 2007. "Probabilistic forecasts, calibration and sharpness," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 69(2), pages 243-268, April.
    11. Watson, Mark W., 1986. "Univariate detrending methods with stochastic trends," Journal of Monetary Economics, Elsevier, vol. 18(1), pages 49-75, July.
    12. Lahiri, Kajal & Wang, J. George, 2013. "Evaluating probability forecasts for GDP declines using alternative methodologies," International Journal of Forecasting, Elsevier, vol. 29(1), pages 175-190.
    13. Rudebusch, Glenn D. & Williams, John C., 2009. "Forecasting Recessions: The Puzzle of the Enduring Power of the Yield Curve," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(4), pages 492-503.
    14. Linton, O. & Whang, Yoon-Jae, 2007. "The quantilogram: With an application to evaluating directional predictability," Journal of Econometrics, Elsevier, vol. 141(1), pages 250-282, November.
    15. Diebold, Francis X & Rudebusch, Glenn D, 1989. "Scoring the Leading Indicators," The Journal of Business, University of Chicago Press, vol. 62(3), pages 369-391, July.
    16. Hamilton, James D, 1989. "A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle," Econometrica, Econometric Society, vol. 57(2), pages 357-384, March.
    17. Casillas-Olvera, Gabriel & Bessler, David A., 2006. "Probability forecasting and central bank accountability," Journal of Policy Modeling, Elsevier, vol. 28(2), pages 223-234, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. John W. Galbraith & Simon van Norden, 2009. "Calibration and Resolution Diagnostics for Bank of England Density Forecasts," CIRANO Working Papers 2009s-36, CIRANO.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Galbraith, John W. & van Norden, Simon, 2011. "Kernel-based calibration diagnostics for recession and inflation probability forecasts," International Journal of Forecasting, Elsevier, vol. 27(4), pages 1041-1057, October.
    2. John W. Galbraith & Simon van Norden, 2009. "Calibration and Resolution Diagnostics for Bank of England Density Forecasts," CIRANO Working Papers 2009s-36, CIRANO.
    3. Lahiri, Kajal & Yang, Liu, 2013. "Forecasting Binary Outcomes," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1025-1106, Elsevier.
    4. Guérin, Pierre & Maurin, Laurent & Mohr, Matthias, 2015. "Trend-Cycle Decomposition Of Output And Euro Area Inflation Forecasts: A Real-Time Approach Based On Model Combination," Macroeconomic Dynamics, Cambridge University Press, vol. 19(2), pages 363-393, March.
    5. Clements Michael P., 2012. "Forecasting U.S. Output Growth with Non-Linear Models in the Presence of Data Uncertainty," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 16(1), pages 1-27, January.
    6. repec:spo:wpecon:info:hdl:2441/f4rshpf3v1umfa09lat09b1bg is not listed on IDEAS
    7. Marc-Oliver Pohle, 2020. "The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation," Papers 2005.01835, arXiv.org.
    8. repec:hal:wpspec:info:hdl:2441/f4rshpf3v1umfa09lat09b1bg is not listed on IDEAS
    9. repec:hal:spmain:info:hdl:2441/f4rshpf3v1umfa09lat09b1bg is not listed on IDEAS
    10. Paul Hubert, 2010. "Monetary policy, imperfect information and the expectations channel [Politique monétaire,information imparfaite et canal des anticipations]," SciencePo Working papers Main tel-04095385, HAL.
    11. Paul Hubert, 2010. "Monetary Policy, Imperfect Information and the Expectations Channel," Sciences Po publications info:hdl:2441/f4rshpf3v1u, Sciences Po.
    12. Carlos Medel, 2017. "Forecasting Chilean inflation with the hybrid new keynesian Phillips curve: globalisation, combination, and accuracy," Journal Economía Chilena (The Chilean Economy), Central Bank of Chile, vol. 20(3), pages 004-050, December.
    13. Galvão, Ana Beatriz, 2013. "Changes in predictive ability with mixed frequency data," International Journal of Forecasting, Elsevier, vol. 29(3), pages 395-410.
    14. Richard H. Clarida & Mark P. Taylor, 2003. "Nonlinear Permanent - Temporary Decompositions in Macroeconomics and Finance," Economic Journal, Royal Economic Society, vol. 113(486), pages 125-139, March.
    15. Aastveit, Knut Are & Trovik, Tørres, 2014. "Estimating the output gap in real time: A factor model approach," The Quarterly Review of Economics and Finance, Elsevier, vol. 54(2), pages 180-193.
    16. Antonin Aviat & Frédérique Bec & Claude Diebolt & Catherine Doz & Denis Ferrand & Laurent Ferrara & Eric Heyer & Valérie Mignon & Pierre-Alain Pionnier, 2021. "Dating business cycles in France: a reference chronology," SciencePo Working papers Main hal-03373425, HAL.
    17. Moolman, Elna, 2004. "A Markov switching regime model of the South African business cycle," Economic Modelling, Elsevier, vol. 21(4), pages 631-646, July.
    18. Dr. James Mitchell, 2009. "Measuring Output Gap Uncertainty," National Institute of Economic and Social Research (NIESR) Discussion Papers 342, National Institute of Economic and Social Research.
    19. Barbara Rossi, 2019. "Forecasting in the presence of instabilities: How do we know whether models predict well and how to improve them," Economics Working Papers 1711, Department of Economics and Business, Universitat Pompeu Fabra, revised Jul 2021.
    20. Carlos A. Medel, 2018. "Forecasting Inflation with the Hybrid New Keynesian Phillips Curve: A Compact-Scale Global VAR Approach," International Economic Journal, Taylor & Francis Journals, vol. 32(3), pages 331-371, July.
    21. Massimiliano Marcellino, 2008. "A linear benchmark for forecasting GDP growth and inflation?," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 27(4), pages 305-340.
    22. Ahsan ul Haq Satti & Wasim Shahid Malik, 2017. "The Unreliability of Output-Gap Estimates in Real Time," The Pakistan Development Review, Pakistan Institute of Development Economics, vol. 56(3), pages 193-219.
    23. Mandler, Martin, 2007. "Decomposing Federal Funds Rate forecast uncertainty using real-time data," MPRA Paper 13498, University Library of Munich, Germany, revised Jan 2009.

    More about this item

    Keywords

    calibration; probability forecast; real-time data; inflation; recession; calibration; probabilités de prévisions; données « en temps réel »; inflation; récession;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cir:cirwor:2008s-28. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Webmaster (email available below). General contact details of provider: https://edirc.repec.org/data/ciranca.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.