IDEAS home Printed from https://ideas.repec.org/a/wly/fufsci/v6y2024i4ne189.html

Scoring rules and performance, new analysis of expert judgment data

Author

Listed:
  • Gabriela F. Nane
  • Roger M. Cooke

Abstract

A review of scoring rules highlights the distinction between rewarding honesty and rewarding quality. This motivates the introduction of a scale‐invariant version of the Continuous Ranked Probability Score (CRPS) which enables statistical accuracy (SA) testing based on an exact rather than an asymptotic distribution of the density of convolutions. A recent data set of 6761 expert probabilistic forecasts for questions for which the actual values are known is used to compare performance. New insights include that (a) variance due to assessed variables dominates variance due to experts, (b) performance on mean absolute percentage error (MAPE) is weakly related to SA (c) scale‐invariant CRPS combinations compete with the Classical Model (CM) on SA and MAPE, and (d) CRPS is more forgiving with regard to SA than the CM as CRPS is insensitive to location bias.

Suggested Citation

  • Gabriela F. Nane & Roger M. Cooke, 2024. "Scoring rules and performance, new analysis of expert judgment data," Futures & Foresight Science, John Wiley & Sons, vol. 6(4), December.
    • Handle: RePEc:wly:fufsci:v:6:y:2024:i:4:n:e189
    DOI: 10.1002/ffo2.189
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/ffo2.189
    Download Restriction: no
    File URL: https://libkey.io/10.1002/ffo2.189?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Weissman, Ishay, 2017. "Sum of squares of uniform random variables," Statistics & Probability Letters, Elsevier, vol. 129(C), pages 147-154.
    2. Cooke, Roger M. & Marti, Deniz & Mazzuchi, Thomas, 2021. "Expert forecasting with and without uncertainty quantification and weighting: What do the data say?," International Journal of Forecasting, Elsevier, vol. 37(1), pages 378-387.
    3. Alexander Dawid & Monica Musio, 2014. "Theory and applications of proper scoring rules," METRON, Springer;Sapienza Università di Roma, vol. 72(2), pages 169-183, August.
    4. Colson, Abigail R. & Cooke, Roger M., 2017. "Cross validation for the classical model of structured expert judgment," Reliability Engineering and System Safety, Elsevier, vol. 163(C), pages 109-120.
    5. Peter A. Morris, 1974. "Decision Analysis Expert Use," Management Science, INFORMS, vol. 20(9), pages 1233-1241, May.
    6. Emir Shuford & Arthur Albert & H. Edward Massengill, 1966. "Admissible probability measurement procedures," Psychometrika, Springer;The Psychometric Society, vol. 31(2), pages 125-145, June.
    7. Arthur Carvalho, 2016. "An Overview of Applications of Proper Scoring Rules," Decision Analysis, INFORMS, vol. 13(4), pages 223-242, December.
    8. Tilmann Gneiting & Fadoua Balabdaoui & Adrian E. Raftery, 2007. "Probabilistic forecasts, calibration and sharpness," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 69(2), pages 243-268, April.
    9. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    10. Peter A. Morris, 1977. "Combining Expert Judgments: A Bayesian Approach," Management Science, INFORMS, vol. 23(7), pages 679-693, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Guus Rongen & Gabriela F. Nane & Oswaldo Morales‐Napoles & Roger M. Cooke, 2025. "Continuous Distributions and Measures of Statistical Accuracy for Structured Expert Judgment," Futures & Foresight Science, John Wiley & Sons, vol. 7(2), August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    2. Eric Neyman & Tim Roughgarden, 2023. "From Proper Scoring Rules to Max-Min Optimal Forecast Aggregation," Operations Research, INFORMS, vol. 71(6), pages 2175-2195, November.
    3. Kenneth C. Lichtendahl & Yael Grushka-Cockayne & Robert L. Winkler, 2013. "Is It Better to Average Probabilities or Quantiles?," Management Science, INFORMS, vol. 59(7), pages 1594-1611, July.
    4. Anne Sofie Jore & James Mitchell & Shaun P. Vahey, 2010. "Combining forecast densities from VARs with uncertain instabilities," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 621-634.
    5. HÃ¥vard Hegre & Paola Vesco & Michael Colaresi & Jonas Vestby & Alexa Timlick & Noorain Syed Kazmi & Angelica Lindqvist-McGowan & Friederike Becker & Marco Binetti & Tobias Bodentien & Tobias Bohne & , 2025. "The 2023/24 VIEWS Prediction challenge: Predicting the number of fatalities in armed conflict, with uncertainty," Journal of Peace Research, Peace Research Institute Oslo, vol. 62(6), pages 2070-2087, November.
    6. Enrique Moral-Benito, 2015. "Model Averaging In Economics: An Overview," Journal of Economic Surveys, Wiley Blackwell, vol. 29(1), pages 46-75, February.
    7. Borgonovo, Emanuele & Jose, Victor Richmond R. & Knowlton, Morgan & Shachter, Ross & Siebert, Johannes Ulrich & Ulu, Canan, 2026. "Fifty years of decision analysis in operational research: A review," European Journal of Operational Research, Elsevier, vol. 329(2), pages 355-377.
    8. Werner Ehm & Tilmann Gneiting & Alexander Jordan & Fabian Krüger, 2016. "Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 505-562, June.
    9. Geweke, John & Amisano, Gianni, 2011. "Optimal prediction pools," Journal of Econometrics, Elsevier, vol. 164(1), pages 130-141, September.
    10. Tino Werner, 2022. "Elicitability of Instance and Object Ranking," Decision Analysis, INFORMS, vol. 19(2), pages 123-140, June.
    11. Yo Joong Choe & Aaditya Ramdas, 2024. "Comparing Sequential Forecasters," Operations Research, INFORMS, vol. 72(4), pages 1368-1387, July.
    12. Robert L. Winkler & Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose, 2019. "Probability Forecasts and Their Combination: A Research Perspective," Decision Analysis, INFORMS, vol. 16(4), pages 239-260, December.
    13. Edgar C. Merkle & Robert Hartman, 2018. "Weighted Brier score decompositions for topically heterogenous forecasting tournaments," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(2), pages 185-201, March.
    14. Warne, Anders, 2023. "DSGE model forecasting: rational expectations vs. adaptive learning," Working Paper Series 2768, European Central Bank.
    15. Merkle, Edgar C. & Steyvers, Mark & Mellers, Barbara & Tetlock, Philip E., 2017. "A neglected dimension of good forecasting judgment: The questions we choose also matter," International Journal of Forecasting, Elsevier, vol. 33(4), pages 817-832.
    16. Lahiri, Kajal & Yang, Liu, 2013. "Forecasting Binary Outcomes," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1025-1106, Elsevier.
    17. Jenny Brynjarsdottir & Jonathan Hobbs & Amy Braverman & Lukas Mandrake, 2018. "Optimal Estimation Versus MCMC for $$\mathrm{{CO}}_{2}$$ CO 2 Retrievals," Journal of Agricultural, Biological and Environmental Statistics, Springer;The International Biometric Society;American Statistical Association, vol. 23(2), pages 297-316, June.
    18. Ellina, Polina & Mascarenhas, Briance & Theodossiou, Panayiotis, 2020. "Clarifying managerial biases using a probabilistic framework," Journal of Behavioral and Experimental Finance, Elsevier, vol. 27(C).
    19. Hall, Stephen G. & Mitchell, James, 2007. "Combining density forecasts," International Journal of Forecasting, Elsevier, vol. 23(1), pages 1-13.
    20. Simon Hirsch, 2025. "Online Multivariate Regularized Distributional Regression for High-dimensional Probabilistic Electricity Price Forecasting," Papers 2504.02518, arXiv.org, revised Apr 2026.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:fufsci:v:6:y:2024:i:4:n:e189. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1002/(ISSN)2573-5152 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.