IDEAS home Printed from https://ideas.repec.org/a/gam/jforec/v3y2021i4p47-773d666151.html
   My bibliography  Save this article

Assessing Goodness of Fit for Verifying Probabilistic Forecasts

Author

Listed:
  • Tae-Ho Kang

    (School of Civil and Environmental Engineering, University of New South Wales, Sydney, NSW 2052, Australia
    Water Resources Management Research Center, K-Water Institute, Deajeon 34350, Korea)

  • Ashish Sharma

    (School of Civil and Environmental Engineering, University of New South Wales, Sydney, NSW 2052, Australia)

  • Lucy Marshall

    (School of Civil and Environmental Engineering, University of New South Wales, Sydney, NSW 2052, Australia)

Abstract

The verification of probabilistic forecasts in hydro-climatology is integral to their development, use, and adoption. We propose here a means of utilizing goodness of fit measures for verifying the reliability of probabilistic forecasts. The difficulty in measuring the goodness of fit for a probabilistic prediction or forecast is that predicted probability distributions for a target variable are not stationary in time, meaning one observation alone exists to quantify goodness of fit for each prediction issued. Therefore, we suggest an additional dissociation that can dissociate target information from the other time variant part—the target to be verified in this study is the alignment of observations to the predicted probability distribution. For this dissociation, the probability integral transformation is used. To measure the goodness of fit for the predicted probability distributions, this study uses the root mean squared deviation metric. If the observations after the dissociation can be assumed to be independent, the mean square deviation metric becomes a chi-square test statistic, which enables statistically testing the hypothesis regarding whether the observations are from the same population as the predicted probability distributions. An illustration of our proposed rationale is provided using the multi-model ensemble prediction for El Niño–Southern Oscillation.

Suggested Citation

  • Tae-Ho Kang & Ashish Sharma & Lucy Marshall, 2021. "Assessing Goodness of Fit for Verifying Probabilistic Forecasts," Forecasting, MDPI, vol. 3(4), pages 1-11, October.
    • Handle: RePEc:gam:jforec:v:3:y:2021:i:4:p:47-773:d:666151
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2571-9394/3/4/47/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2571-9394/3/4/47/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. James E. Matheson & Robert L. Winkler, 1976. "Scoring Rules for Continuous Probability Distributions," Management Science, INFORMS, vol. 22(10), pages 1087-1096, June.
    2. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Diego Zappa & Gian Paolo Clemente & Francesco Della Corte & Nino Savelli, 2023. "Editorial on the Special Issue on Insurance: complexity, risks and its connection with social sciences," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(2), pages 125-130, December.
    2. Emanuela Raffinetti, 2023. "A Rank Graduation Accuracy measure to mitigate Artificial Intelligence risks," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(2), pages 131-150, December.
    3. Paolo Giudici & Emanuela Raffinetti, 2025. "RGA: a unified measure of predictive accuracy," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 19(1), pages 67-93, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ricardo Crisóstomo, 2021. "Estimating real‐world probabilities: A forward‐looking behavioral framework," Journal of Futures Markets, John Wiley & Sons, Ltd., vol. 41(11), pages 1797-1823, November.
    2. Magnus Reif, 2020. "Macroeconomics, Nonlinearities, and the Business Cycle," ifo Beiträge zur Wirtschaftsforschung, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, number 87, September.
    3. Nico Keilman, 2020. "Evaluating Probabilistic Population Forecasts," Economie et Statistique / Economics and Statistics, Institut National de la Statistique et des Etudes Economiques (INSEE), issue 520-521, pages 49-64.
    4. Nowotarski, Jakub & Weron, Rafał, 2018. "Recent advances in electricity price forecasting: A review of probabilistic forecasting," Renewable and Sustainable Energy Reviews, Elsevier, vol. 81(P1), pages 1548-1568.
    5. Bansal, Prateek & Krueger, Rico & Graham, Daniel J., 2021. "Fast Bayesian estimation of spatial count data models," Computational Statistics & Data Analysis, Elsevier, vol. 157(C).
    6. Malte Knüppel & Fabian Krüger, 2022. "Forecast uncertainty, disagreement, and the linear pool," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(1), pages 23-41, January.
    7. Song, Haiyan & Wen, Long & Liu, Chang, 2019. "Density tourism demand forecasting revisited," Annals of Tourism Research, Elsevier, vol. 75(C), pages 379-392.
    8. Tomás Marinozzi, 2023. "Forecasting Inflation in Argentina: A Probabilistic Approach," Ensayos Económicos, Central Bank of Argentina, Economic Research Department, vol. 1(81), pages 81-110, May.
    9. Marczak, Martyna & Proietti, Tommaso & Grassi, Stefano, 2018. "A data-cleaning augmented Kalman filter for robust estimation of state space models," Econometrics and Statistics, Elsevier, vol. 5(C), pages 107-123.
    10. Fabian Kruger & Hendrik Plett, 2022. "Prediction intervals for economic fixed-event forecasts," Papers 2210.13562, arXiv.org, revised Mar 2024.
    11. Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose & Robert L. Winkler, 2017. "Quantile Evaluation, Sensitivity to Bracketing, and Sharing Business Payoffs," Operations Research, INFORMS, vol. 65(3), pages 712-728, June.
    12. Krüger, Fabian & Nolte, Ingmar, 2016. "Disagreement versus uncertainty: Evidence from distribution forecasts," Journal of Banking & Finance, Elsevier, vol. 72(S), pages 172-186.
    13. André Victor Ribeiro Amaral & Elias Teixeira Krainski & Ruiman Zhong & Paula Moraga, 2024. "Model-Based Geostatistics Under Spatially Varying Preferential Sampling," Journal of Agricultural, Biological and Environmental Statistics, Springer;The International Biometric Society;American Statistical Association, vol. 29(4), pages 766-792, December.
    14. Reif Magnus, 2021. "Macroeconomic uncertainty and forecasting macroeconomic aggregates," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 25(2), pages 1-20, April.
    15. Fabian Krüger & Sebastian Lerch & Thordis Thorarinsdottir & Tilmann Gneiting, 2021. "Predictive Inference Based on Markov Chain Monte Carlo Output," International Statistical Review, International Statistical Institute, vol. 89(2), pages 274-301, August.
    16. Tommaso Proietti & Martyna Marczak & Gianluigi Mazzi, 2017. "Euromind‐ D : A Density Estimate of Monthly Gross Domestic Product for the Euro Area," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(3), pages 683-703, April.
    17. Bjørnland, Hilde C. & Gerdrup, Karsten & Jore, Anne Sofie & Smith, Christie & Thorsrud, Leif Anders, 2011. "Weights and pools for a Norwegian density combination," The North American Journal of Economics and Finance, Elsevier, vol. 22(1), pages 61-76, January.
    18. Krüger, Fabian & Pavlova, Lora, 2019. "Quantifying subjective oncertainty in survey expectations," Working Papers 0664, University of Heidelberg, Department of Economics.
    19. Stephen Hora & Erim Kardeş, 2015. "Calibration, sharpness and the weighting of experts in a linear opinion pool," Annals of Operations Research, Springer, vol. 229(1), pages 429-450, June.
    20. Thordis L. Thorarinsdottir & Tilmann Gneiting, 2010. "Probabilistic forecasts of wind speed: ensemble model output statistics by using heteroscedastic censored regression," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 173(2), pages 371-388, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jforec:v:3:y:2021:i:4:p:47-773:d:666151. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.