Probabilistic forecasts, calibration and sharpness
AbstractProbabilistic forecasts of continuous variables take the form of predictive densities or predictive cumulative distribution functions. We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of "maximizing the sharpness of the predictive distributions subject to calibration". Calibration refers to the statistical consistency between the distributional forecasts and the observations and is a joint property of the predictions and the events that materialize. Sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only. A simple theoretical framework allows us to distinguish between probabilistic calibration, exceedance calibration and marginal calibration. We propose and study tools for checking calibration and sharpness, among them the probability integral transform histogram, marginal calibration plots, the sharpness diagram and proper scoring rules. The diagnostic approach is illustrated by an assessment and ranking of probabilistic forecasts of wind speed at the Stateline wind energy centre in the US Pacific Northwest. In combination with cross-validation or in the time series context, our proposal provides very general, nonparametric alternatives to the use of information criteria for model diagnostics and model selection. Copyright 2007 Royal Statistical Society.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
As the access to this document is restricted, you may want to look for a different version under "Related research" (further below) or search for a different version of it.
Bibliographic InfoArticle provided by Royal Statistical Society in its journal Journal of the Royal Statistical Society: Series B (Statistical Methodology).
Volume (Year): 69 (2007)
Issue (Month): 2 ()
Contact details of provider:
Postal: 12 Errol Street, London EC1Y 8LX, United Kingdom
Web page: http://www.blackwellpublishing.com/journal.asp?ref=1369-7412
More information through EDIRC
You can help add them by filling out this form.
CitEc Project, subscribe to its RSS feed for this item.
- John Galbraith & Simon van Norden, 2008.
"The Calibration Of Probabilistic Economic Forecasts,"
Departmental Working Papers
2008-05, McGill University, Department of Economics.
- John Galbraith & Simon van Norden, 2008. "The Calibration of Probabilistic Economic Forecasts," CIRANO Working Papers 2008s-28, CIRANO.
- Jason Ng & Catherine S. Forbes & Gael M. Martin & Brendan P.M. McCabe, 2011.
"Non-Parametric Estimation of Forecast Distributions in Non-Gaussian, Non-linear State Space Models,"
Monash Econometrics and Business Statistics Working Papers
11/11, Monash University, Department of Econometrics and Business Statistics.
- Ng, Jason & Forbes, Catherine S. & Martin, Gael M. & McCabe, Brendan P.M., 2013. "Non-parametric estimation of forecast distributions in non-Gaussian, non-linear state space models," International Journal of Forecasting, Elsevier, vol. 29(3), pages 411-430.
- Pierre-Julien Trombe & Pierre Pinson & Henrik Madsen, 2012. "A General Probabilistic Forecasting Framework for Offshore Wind Power Fluctuations," Energies, MDPI, Open Access Journal, vol. 5(3), pages 621-657, March.
- Tsyplakov, Alexander, 2012. "Assessment of probabilistic forecasts: Proper scoring rules and moments," Applied Econometrics, Publishing House "SINERGIA PRESS", vol. 27(3), pages 115-132.
- Daniela Cocchi & Antonio Fabio Di Narzo, 2008. "A Bayesian Hierarchical Approach to Ensemble Weather Forecasting," Quaderni di Dipartimento 5, Department of Statistics, University of Bologna.
- Herbst, Edward & Schorfheide, Frank, 2012.
"Evaluating DSGE model forecasts of comovements,"
Journal of Econometrics,
Elsevier, vol. 171(2), pages 152-166.
- Edward Herbst & Frank Schorfheide, 2011. "Evaluating DSGE model forecasts of comovements," Working Papers 11-5, Federal Reserve Bank of Philadelphia.
- Edward Herbst & Frank Schorfheide, 2012. "Evaluating DSGE model forecasts of comovements," Finance and Economics Discussion Series 2012-11, Board of Governors of the Federal Reserve System (U.S.).
- Reason Lesego Machete, 2011. "Early Warning with Calibrated and Sharper Probabilistic Forecasts," Papers 1112.6390, arXiv.org, revised Jan 2012.
- Christoffel, Kai & Warne, Anders & Coenen, Günter, 2010. "Forecasting with DSGE models," Working Paper Series 1185, European Central Bank.
- Carlos Velasco, 2013. "Comments on: Model-free model-fitting and predictive distributions," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer, vol. 22(2), pages 237-239, June.
- Adrian Raftery & Jennifer Chunn & Patrick Gerland & Hana Ševčíková, 2013. "Bayesian Probabilistic Projections of Life Expectancy for All Countries," Demography, Springer, vol. 50(3), pages 777-801, June.
- Tsyplakov, Alexander, 2011. "Evaluating density forecasts: a comment," MPRA Paper 31184, University Library of Munich, Germany.
- Tsyplakov, Alexander, 2010. "The links between inﬂation and inﬂation uncertainty at the longer horizon," MPRA Paper 26908, University Library of Munich, Germany.
- Gneiting, Tilmann, 2011. "Quantiles as optimal point forecasts," International Journal of Forecasting, Elsevier, vol. 27(2), pages 197-207, April.
- Bjørnland, Hilde C. & Gerdrup, Karsten & Jore, Anne Sofie & Smith, Christie & Thorsrud, Leif Anders, 2011.
"Weights and pools for a Norwegian density combination,"
The North American Journal of Economics and Finance,
Elsevier, vol. 22(1), pages 61-76, January.
- Hilde Bjørnland & Karsten Gerdrup & Christie Smith & Anne Sofie Jore & Leif Anders Thorsrud, 2010. "Weights and pools for a Norwegian density combination," Working Paper 2010/06, Norges Bank.
- Worapree Maneesoonthorn & Gael M. Martin & Catherine S. Forbes & Simone Grose, 2010.
"Probabilistic Forecasts of Volatility and its Risk Premia,"
Monash Econometrics and Business Statistics Working Papers
22/10, Monash University, Department of Econometrics and Business Statistics.
- Maneesoonthorn, Worapree & Martin, Gael M. & Forbes, Catherine S. & Grose, Simone D., 2012. "Probabilistic forecasts of volatility and its risk premia," Journal of Econometrics, Elsevier, vol. 171(2), pages 217-236.
- Brentnall, Adam R. & Crowder, Martin J. & Hand, David J., 2010. "Predictive-sequential forecasting system development for cash machine stocking," International Journal of Forecasting, Elsevier, vol. 26(4), pages 764-776, October.
- Tsyplakov, Alexander, 2013. "Evaluation of Probabilistic Forecasts: Proper Scoring Rules and Moments," MPRA Paper 45186, University Library of Munich, Germany.
- Galbraith, John W. & van Norden, Simon, 2011. "Kernel-based calibration diagnostics for recession and inflation probability forecasts," International Journal of Forecasting, Elsevier, vol. 27(4), pages 1041-1057, October.
- John Galbraith & Simon van Norden, 2009. "Calibration and Resolution Diagnostics for Bank of England Density Forecasts," CIRANO Working Papers 2009s-36, CIRANO.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Wiley-Blackwell Digital Licensing) or (Christopher F. Baum).
If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.
If references are entirely missing, you can add them using this form.
If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.
If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.
Please note that corrections may take a couple of weeks to filter through the various RePEc services.