IDEAS home Printed from https://ideas.repec.org/a/bla/jorssb/v69y2007i2p243-268.html

Probabilistic forecasts, calibration and sharpness

Author

Listed:
  • Tilmann Gneiting
  • Fadoua Balabdaoui
  • Adrian E. Raftery

Abstract

Summary. Probabilistic forecasts of continuous variables take the form of predictive densities or predictive cumulative distribution functions. We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibration. Calibration refers to the statistical consistency between the distributional forecasts and the observations and is a joint property of the predictions and the events that materialize. Sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only. A simple theoretical framework allows us to distinguish between probabilistic calibration, exceedance calibration and marginal calibration. We propose and study tools for checking calibration and sharpness, among them the probability integral transform histogram, marginal calibration plots, the sharpness diagram and proper scoring rules. The diagnostic approach is illustrated by an assessment and ranking of probabilistic forecasts of wind speed at the Stateline wind energy centre in the US Pacific Northwest. In combination with cross‐validation or in the time series context, our proposal provides very general, nonparametric alternatives to the use of information criteria for model diagnostics and model selection.

Suggested Citation

  • Tilmann Gneiting & Fadoua Balabdaoui & Adrian E. Raftery, 2007. "Probabilistic forecasts, calibration and sharpness," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 69(2), pages 243-268, April.
    • Handle: RePEc:bla:jorssb:v:69:y:2007:i:2:p:243-268
    DOI: 10.1111/j.1467-9868.2007.00587.x
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/j.1467-9868.2007.00587.x
    Download Restriction: no
    File URL: https://libkey.io/10.1111/j.1467-9868.2007.00587.x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Gneiting, Tilmann & Larson, Kristin & Westrick, Kenneth & Genton, Marc G. & Aldrich, Eric, 2006. "Calibrated Probabilistic Forecasting at the Stateline Wind Energy Center: The Regime-Switching SpaceTime Method," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 968-979, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tascikaraoglu, Akin & Sanandaji, Borhan M. & Poolla, Kameshwar & Varaiya, Pravin, 2016. "Exploiting sparsity of interconnections in spatio-temporal wind speed forecasting using Wavelet Transform," Applied Energy, Elsevier, vol. 165(C), pages 735-747.
    2. Tansu Filik, 2016. "Improved Spatio-Temporal Linear Models for Very Short-Term Wind Speed Forecasting," Energies, MDPI, vol. 9(3), pages 1-15, March.
    3. Josselin Le Gal La Salle & Mathieu David & Philippe Lauret, 2025. "A Set of New Tools to Measure the Effective Value of Probabilistic Forecasts of Continuous Variables," Forecasting, MDPI, vol. 7(2), pages 1-18, June.
    4. Xinxin Zhu & Marc Genton & Yingzhong Gu & Le Xie, 2014. "Space-time wind speed forecasting for improved power system dispatch," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 23(1), pages 1-25, March.
    5. Pinson, P. & Reikard, G. & Bidlot, J.-R., 2012. "Probabilistic forecasting of the wave energy flux," Applied Energy, Elsevier, vol. 93(C), pages 364-370.
    6. Daniela Castro-Camilo & Raphaël Huser & Håvard Rue, 2019. "A Spliced Gamma-Generalized Pareto Model for Short-Term Extreme Wind Speed Probabilistic Forecasting," Journal of Agricultural, Biological and Environmental Statistics, Springer;The International Biometric Society;American Statistical Association, vol. 24(3), pages 517-534, September.
    7. Shah, Kamran Ali & Meng, Fantai & Li, Ye & Nagamune, Ryozo & Zhou, Yarong & Ren, Zhengru & Jiang, Zhiyu, 2021. "A synthesis of feasible control methods for floating offshore wind turbine system dynamics," Renewable and Sustainable Energy Reviews, Elsevier, vol. 151(C).
    8. Yang, Dazhi & van der Meer, Dennis, 2021. "Post-processing in solar forecasting: Ten overarching thinking tools," Renewable and Sustainable Energy Reviews, Elsevier, vol. 140(C).
    9. Flavie Didier & Yong-Chao Liu & Salah Laghrouche & Daniel Depernet, 2024. "A Comprehensive Review on Advanced Control Methods for Floating Offshore Wind Turbine Systems above the Rated Wind Speed," Energies, MDPI, vol. 17(10), pages 1-33, May.
    10. Amanda Hering, 2014. "Comments on: Space-time wind speed forecasting for improved power system dispatch," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 23(1), pages 34-44, March.
    11. Baggio, Roberta & Muzy, Jean-François, 2024. "Improving probabilistic wind speed forecasting using M-Rice distribution and spatial data integration," Applied Energy, Elsevier, vol. 360(C).
    12. Wang, Yun & Xu, Houhua & Song, Mengmeng & Zhang, Fan & Li, Yifen & Zhou, Shengchao & Zhang, Lingjun, 2023. "A convolutional Transformer-based truncated Gaussian density network with data denoising for wind speed forecasting," Applied Energy, Elsevier, vol. 333(C).
    13. Gallego, C. & Pinson, P. & Madsen, H. & Costa, A. & Cuerva, A., 2011. "Influence of local wind speed and direction on wind power dynamics – Application to offshore very short-term forecasting," Applied Energy, Elsevier, vol. 88(11), pages 4087-4096.
    14. Tilmann Gneiting & Larissa Stanberry & Eric Grimit & Leonhard Held & Nicholas Johnson, 2008. "Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 17(2), pages 211-235, August.
    15. Daniel Ambach & Robert Garthoff, 2016. "Vorhersagen der Windgeschwindigkeit und Windenergie in Deutschland," AStA Wirtschafts- und Sozialstatistisches Archiv, Springer;Deutsche Statistische Gesellschaft - German Statistical Society, vol. 10(1), pages 15-36, February.
    16. Chen, Xin & Ye, Xiaoling & Xiong, Xiong & Zhang, Yingchao & Li, Yuanlu, 2024. "Improving the accuracy of wind speed spatial interpolation: A pre-processing algorithm for wind speed dynamic time warping interpolation," Energy, Elsevier, vol. 295(C).
    17. Amanda S. Hering & Karen Kazor & William Kleiber, 2015. "A Markov-Switching Vector Autoregressive Stochastic Wind Generator for Multiple Spatial and Temporal Scales," Resources, MDPI, vol. 4(1), pages 1-23, February.
    18. Taylor, James W., 2017. "Probabilistic forecasting of wind power ramp events using autoregressive logit models," European Journal of Operational Research, Elsevier, vol. 259(2), pages 703-712.
    19. Men, Zhongxian & Yee, Eugene & Lien, Fue-Sang & Wen, Deyong & Chen, Yongsheng, 2016. "Short-term wind speed and power forecasting using an ensemble of mixture density neural networks," Renewable Energy, Elsevier, vol. 87(P1), pages 203-211.
    20. Branko Kosovic & Sue Ellen Haupt & Daniel Adriaansen & Stefano Alessandrini & Gerry Wiener & Luca Delle Monache & Yubao Liu & Seth Linden & Tara Jensen & William Cheng & Marcia Politovich & Paul Prest, 2020. "A Comprehensive Wind Power Forecasting System Integrating Artificial Intelligence and Numerical Weather Prediction," Energies, MDPI, vol. 13(6), pages 1-16, March.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jorssb:v:69:y:2007:i:2:p:243-268. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/rssssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.