IDEAS home Printed from https://ideas.repec.org/a/wly/fufsci/v7y2025i2ne70009.html

Continuous Distributions and Measures of Statistical Accuracy for Structured Expert Judgment

Author

Listed:
  • Guus Rongen
  • Gabriela F. Nane
  • Oswaldo Morales‐Napoles
  • Roger M. Cooke

Abstract

This study evaluates five scoring rules, or measures of statistical accuracy, for assessing uncertainty estimates from expert judgment studies and model forecasts. These rules — the Continuously Ranked Probability Score (CRPS ${CRPS}$), Kolmogorov‐Smirnov (KS ${KS}$), Cramer‐von Mises (CvM ${CvM}$), Anderson Darling (AD ${AD}$), and chi‐square test — were applied to 6864 expert uncertainty estimates from 49 Classical Model (CM) studies. We compared their sensitivity to various biases and their ability to serve as performance‐based weight for expert estimates. Additionally, the piecewise uniform and Metalog distribution were evaluated for their representation of expert estimates because four of the five rules require interpolating the experts' estimates. Simulating biased estimates reveals varying sensitivity of the considered test statistics to these biases. Expert weights derived using one measure of statistical accuracy were evaluated with other measures to assess their performance. The main conclusions are (1) CRPS ${CRPS}$ overlooks important biases, while chi‐square and AD ${AD}$ behave similarly, as do KS ${KS}$ and CvM ${CvM}$. (2) All measures except CRPS ${CRPS}$ agree that performance weighting is superior to equal weighting with respect to statistical accuracy. (3) Neither distributions can effectively predict the position of a removed quantile estimate. These insights show the behavior of different scoring rules for combining uncertainty estimates from expert or models, and extent the knowledge for best‐practices.

Suggested Citation

  • Guus Rongen & Gabriela F. Nane & Oswaldo Morales‐Napoles & Roger M. Cooke, 2025. "Continuous Distributions and Measures of Statistical Accuracy for Structured Expert Judgment," Futures & Foresight Science, John Wiley & Sons, vol. 7(2), August.
    • Handle: RePEc:wly:fufsci:v:7:y:2025:i:2:n:e70009
    DOI: 10.1002/ffo2.70009
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/ffo2.70009
    Download Restriction: no
    File URL: https://libkey.io/10.1002/ffo2.70009?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Gabriela F. Nane & Roger M. Cooke, 2024. "Scoring rules and performance, new analysis of expert judgment data," Futures & Foresight Science, John Wiley & Sons, vol. 6(4), December.
    2. Cooke, Roger M. & Marti, Deniz & Mazzuchi, Thomas, 2021. "Expert forecasting with and without uncertainty quantification and weighting: What do the data say?," International Journal of Forecasting, Elsevier, vol. 37(1), pages 378-387.
    3. Colson, Abigail R. & Cooke, Roger M., 2017. "Cross validation for the classical model of structured expert judgment," Reliability Engineering and System Safety, Elsevier, vol. 163(C), pages 109-120.
    4. Marsaglia, George & Marsaglia, John, 2004. "Evaluating the Anderson-Darling Distribution," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 9(i02).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Roger M. Cooke, 2023. "Averaging quantiles, variance shrinkage, and overconfidence," Futures & Foresight Science, John Wiley & Sons, vol. 5(1), March.
    2. Ren, Xin & Nane, Gabriela F. & Terwel, Karel C. & van Gelder, Pieter H.A.J.M., 2024. "Measuring the impacts of human and organizational factors on human errors in the Dutch construction industry using structured expert judgement," Reliability Engineering and System Safety, Elsevier, vol. 244(C).
    3. Fathy, Mohammad & Kazemzadeh Haghighi, Foojan & Ahmadi, Mohammad, 2024. "Uncertainty quantification of reservoir performance using machine learning algorithms and structured expert judgment," Energy, Elsevier, vol. 288(C).
    4. Gabriela F. Nane & Roger M. Cooke, 2024. "Scoring rules and performance, new analysis of expert judgment data," Futures & Foresight Science, John Wiley & Sons, vol. 6(4), December.
    5. BenSaïda, Ahmed & Slim, Skander, 2016. "Highly flexible distributions to fit multiple frequency financial returns," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 442(C), pages 203-213.
    6. Fernández de Marcos Giménez de los Galanes, Alberto & García Portugués, Eduardo, 2022. "Data-driven stabilizations of goodness-of-fit tests," DES - Working Papers. Statistics and Econometrics. WS 35324, Universidad Carlos III de Madrid. Departamento de Estadística.
    7. Funk, Patrick & Davis, Alex & Vaishnav, Parth & Dewitt, Barry & Fuchs, Erica, 2020. "Individual inconsistency and aggregate rationality: Overcoming inconsistencies in expert judgment at the technical frontier," Technological Forecasting and Social Change, Elsevier, vol. 155(C).
    8. Patrick Afflerbach & Christopher Dun & Henner Gimpel & Dominik Parak & Johannes Seyfried, 2021. "A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(4), pages 329-348, August.
    9. Shibin Zhang & Xin M. Tu, 2022. "Tests for comparing time‐invariant and time‐varying spectra based on the Anderson–Darling statistic," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 76(3), pages 254-282, August.
    10. Jiwoong Kim, 2026. "Application of some $$L_{2}$$ L 2 optimization to a discrete distribution," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 78(1), pages 43-67, February.
    11. Mario P. Brito & Ian G. J. Dawson, 2020. "Predicting the Validity of Expert Judgments in Assessing the Impact of Risk Mitigation Through Failure Prevention and Correction," Risk Analysis, John Wiley & Sons, vol. 40(10), pages 1928-1943, October.
    12. Zinoviy Landsman & Udi Makov & Tomer Shushi, 2017. "Extended Generalized Skew-Elliptical Distributions and their Moments," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 79(1), pages 76-100, February.
    13. Nogal, Maria & Morales Nápoles, Oswaldo & O’Connor, Alan, 2019. "Structured expert judgement to understand the intrinsic vulnerability of traffic networks," Transportation Research Part A: Policy and Practice, Elsevier, vol. 127(C), pages 136-152.
    14. Kevin Rennert & Brian C. Prest & William A. Pizer & Richard G. Newell & David Anthoff & Cora Kingdon & Lisa Rennels & Roger Cooke & Adrian E. Raftery & Hana Sevcikova & Frank Errickson, 2021. "The Social Cost of Carbon: Advances in Long-Term Probabilistic Projections of Population, GDP, Emissions, and Discount Rates," Brookings Papers on Economic Activity, Economic Studies Program, The Brookings Institution, vol. 52(2 (Fall)), pages 223-305.
    15. James W. Taylor & Xiaochun Meng, 2026. "Angular Combining of Forecasts of Probability Distributions," Management Science, INFORMS, vol. 72(3), pages 2111-2133, March.
    16. Hocine Khelifa & Eric Vagnon & Abderrahmane Beroual, 2023. "Effect of Fullerene and Graphene Nanoparticles on the AC Dielectric Strength of Natural Ester," Energies, MDPI, vol. 16(4), pages 1-11, February.
    17. Jeremy Rohmer & Eric Chojnacki, 2021. "Forecast of environment systems using expert judgements: performance comparison between the possibilistic and the classical model," Environment Systems and Decisions, Springer, vol. 41(1), pages 131-146, March.
    18. Fernández-de-Marcos, Alberto & García-Portugués, Eduardo, 2023. "Data-driven stabilizations of goodness-of-fit tests," Computational Statistics & Data Analysis, Elsevier, vol. 179(C).
    19. Dobric, Jadran & Schmid, Friedrich, 2007. "A goodness of fit test for copulas based on Rosenblatt's transformation," Computational Statistics & Data Analysis, Elsevier, vol. 51(9), pages 4633-4642, May.
    20. Sung Ik Kim, 2022. "ARMA–GARCH model with fractional generalized hyperbolic innovations," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 8(1), pages 1-25, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:fufsci:v:7:y:2025:i:2:n:e70009. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1002/(ISSN)2573-5152 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.