IDEAS home Printed from https://ideas.repec.org/a/bla/jorssb/v69y2007i2p243-268.html
   My bibliography  Save this article

Probabilistic forecasts, calibration and sharpness

Author

Listed:
  • Tilmann Gneiting
  • Fadoua Balabdaoui
  • Adrian E. Raftery

Abstract

Summary. Probabilistic forecasts of continuous variables take the form of predictive densities or predictive cumulative distribution functions. We propose a diagnostic approach to the evaluation of predictive performance that is based on the paradigm of maximizing the sharpness of the predictive distributions subject to calibration. Calibration refers to the statistical consistency between the distributional forecasts and the observations and is a joint property of the predictions and the events that materialize. Sharpness refers to the concentration of the predictive distributions and is a property of the forecasts only. A simple theoretical framework allows us to distinguish between probabilistic calibration, exceedance calibration and marginal calibration. We propose and study tools for checking calibration and sharpness, among them the probability integral transform histogram, marginal calibration plots, the sharpness diagram and proper scoring rules. The diagnostic approach is illustrated by an assessment and ranking of probabilistic forecasts of wind speed at the Stateline wind energy centre in the US Pacific Northwest. In combination with cross‐validation or in the time series context, our proposal provides very general, nonparametric alternatives to the use of information criteria for model diagnostics and model selection.

Suggested Citation

  • Tilmann Gneiting & Fadoua Balabdaoui & Adrian E. Raftery, 2007. "Probabilistic forecasts, calibration and sharpness," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 69(2), pages 243-268, April.
    • Handle: RePEc:bla:jorssb:v:69:y:2007:i:2:p:243-268
    DOI: 10.1111/j.1467-9868.2007.00587.x
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/j.1467-9868.2007.00587.x
    Download Restriction: no
    File URL: https://libkey.io/10.1111/j.1467-9868.2007.00587.x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Gneiting, Tilmann & Larson, Kristin & Westrick, Kenneth & Genton, Marc G. & Aldrich, Eric, 2006. "Calibrated Probabilistic Forecasting at the Stateline Wind Energy Center: The Regime-Switching SpaceTime Method," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 968-979, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tascikaraoglu, Akin & Sanandaji, Borhan M. & Poolla, Kameshwar & Varaiya, Pravin, 2016. "Exploiting sparsity of interconnections in spatio-temporal wind speed forecasting using Wavelet Transform," Applied Energy, Elsevier, vol. 165(C), pages 735-747.
    2. Daniela Castro-Camilo & Raphaël Huser & Håvard Rue, 2019. "A Spliced Gamma-Generalized Pareto Model for Short-Term Extreme Wind Speed Probabilistic Forecasting," Journal of Agricultural, Biological and Environmental Statistics, Springer;The International Biometric Society;American Statistical Association, vol. 24(3), pages 517-534, September.
    3. Yang, Dazhi & van der Meer, Dennis, 2021. "Post-processing in solar forecasting: Ten overarching thinking tools," Renewable and Sustainable Energy Reviews, Elsevier, vol. 140(C).
    4. Amanda Hering, 2014. "Comments on: Space-time wind speed forecasting for improved power system dispatch," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 23(1), pages 34-44, March.
    5. Gallego, C. & Pinson, P. & Madsen, H. & Costa, A. & Cuerva, A., 2011. "Influence of local wind speed and direction on wind power dynamics – Application to offshore very short-term forecasting," Applied Energy, Elsevier, vol. 88(11), pages 4087-4096.
    6. Tilmann Gneiting & Larissa Stanberry & Eric Grimit & Leonhard Held & Nicholas Johnson, 2008. "Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 17(2), pages 211-235, August.
    7. Daniel Ambach & Robert Garthoff, 2016. "Vorhersagen der Windgeschwindigkeit und Windenergie in Deutschland," AStA Wirtschafts- und Sozialstatistisches Archiv, Springer;Deutsche Statistische Gesellschaft - German Statistical Society, vol. 10(1), pages 15-36, February.
    8. Amanda S. Hering & Karen Kazor & William Kleiber, 2015. "A Markov-Switching Vector Autoregressive Stochastic Wind Generator for Multiple Spatial and Temporal Scales," Resources, MDPI, vol. 4(1), pages 1-23, February.
    9. Taylor, James W., 2017. "Probabilistic forecasting of wind power ramp events using autoregressive logit models," European Journal of Operational Research, Elsevier, vol. 259(2), pages 703-712.
    10. Men, Zhongxian & Yee, Eugene & Lien, Fue-Sang & Wen, Deyong & Chen, Yongsheng, 2016. "Short-term wind speed and power forecasting using an ensemble of mixture density neural networks," Renewable Energy, Elsevier, vol. 87(P1), pages 203-211.
    11. Jiang, Yu & Song, Zhe & Kusiak, Andrew, 2013. "Very short-term wind speed forecasting with Bayesian structural break model," Renewable Energy, Elsevier, vol. 50(C), pages 637-647.
    12. Duca, Victor E.L.A. & Fonseca, Thaís C.O. & Cyrino Oliveira, Fernando Luiz, 2023. "An overview of non-Gaussian state-space models for wind speed data," Energy, Elsevier, vol. 266(C).
    13. Xydas, Erotokritos & Qadrdan, Meysam & Marmaras, Charalampos & Cipcigan, Liana & Jenkins, Nick & Ameli, Hossein, 2017. "Probabilistic wind power forecasting and its application in the scheduling of gas-fired generators," Applied Energy, Elsevier, vol. 192(C), pages 382-394.
    14. Cao, Qing & Ewing, Bradley T. & Thompson, Mark A., 2012. "Forecasting wind speed with recurrent neural networks," European Journal of Operational Research, Elsevier, vol. 221(1), pages 148-154.
    15. Xinxin Zhu & Marc Genton & Yingzhong Gu & Le Xie, 2014. "Rejoinder on: Space-time wind speed forecasting for improved power system dispatch," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 23(1), pages 45-50, March.
    16. He, Qingqing & Wang, Jianzhou & Lu, Haiyan, 2018. "A hybrid system for short-term wind speed forecasting," Applied Energy, Elsevier, vol. 226(C), pages 756-771.
    17. Casciaro, Gabriele & Ferrari, Francesco & Lagomarsino-Oneto, Daniele & Lira-Loarca, Andrea & Mazzino, Andrea, 2022. "Increasing the skill of short-term wind speed ensemble forecasts combining forecasts and observations via a new dynamic calibration," Energy, Elsevier, vol. 251(C).
    18. Croonenbroeck, Carsten & Ambach, Daniel, 2015. "Censored spatial wind power prediction with random effects," Renewable and Sustainable Energy Reviews, Elsevier, vol. 51(C), pages 613-622.
    19. Thordis L. Thorarinsdottir & Tilmann Gneiting, 2010. "Probabilistic forecasts of wind speed: ensemble model output statistics by using heteroscedastic censored regression," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 173(2), pages 371-388, April.
    20. Huang Huang & Stefano Castruccio & Marc G. Genton, 2022. "Forecasting high‐frequency spatio‐temporal wind power with dimensionally reduced echo state networks," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 71(2), pages 449-466, March.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jorssb:v:69:y:2007:i:2:p:243-268. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/rssssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.