IDEAS home Printed from https://ideas.repec.org/a/wly/riskan/v19y1999i2p187-203.html
   My bibliography  Save this article

Combining Probability Distributions From Experts in Risk Analysis

Author

Listed:
  • Robert T. Clemen
  • Robert L. Winkler

Abstract

This paper concerns the combination of experts' probability distributions in risk analysis, discussing a variety of combination methods and attempting to highlight the important conceptual and practical issues to be considered in designing a combination process in practice. The role of experts is important because their judgments can provide valuable information, particularly in view of the limited availability of “hard data” regarding many important uncertainties in risk analysis. Because uncertainties are represented in terms of probability distributions in probabilistic risk analysis (PRA), we consider expert information in terms of probability distributions. The motivation for the use of multiple experts is simply the desire to obtain as much information as possible. Combining experts' probability distributions summarizes the accumulated information for risk analysts and decision‐makers. Procedures for combining probability distributions are often compartmentalized as mathematical aggregation methods or behavioral approaches, and we discuss both categories. However, an overall aggregation process could involve both mathematical and behavioral aspects, and no single process is best in all circumstances. An understanding of the pros and cons of different methods and the key issues to consider is valuable in the design of acombination process for a specific PRA. The output, a ”combined probabilitydistribution,” can ideally be viewed as representing a summary of the current state of expert opinion regarding the uncertainty of interest.

Suggested Citation

  • Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    • Handle: RePEc:wly:riskan:v:19:y:1999:i:2:p:187-203
    DOI: 10.1111/j.1539-6924.1999.tb00399.x
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/j.1539-6924.1999.tb00399.x
    Download Restriction: no
    File URL: https://libkey.io/10.1111/j.1539-6924.1999.tb00399.x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Davis, James H., 1992. "Some compelling intuitions about group consensus decisions, theoretical and empirical research, and interpersonal aggregation phenomena: Selected examples 1950-1990," Organizational Behavior and Human Decision Processes, Elsevier, vol. 52(1), pages 3-38, June.
    2. Mohamed N. Jouini & Robert T. Clemen, 1996. "Copula Models for Aggregating Expert Opinions," Operations Research, INFORMS, vol. 44(3), pages 444-457, June.
    3. H. V. Ravinder & Don N. Kleinmuntz & James S. Dyer, 1988. "The Reliability of Subjective Probabilities Obtained Through Decomposition," Management Science, INFORMS, vol. 34(2), pages 186-199, February.
    4. Sniezek, Janet A. & Henry, Rebecca A., 1990. "Revision, Weighting, and commitment in consensus group judgment," Organizational Behavior and Human Decision Processes, Elsevier, vol. 45(1), pages 66-84, February.
    5. MacGregor, Donald & Lichtenstein, Sarah & Slovic, Paul, 1988. "Structuring knowledge retrieval: An analysis of decomposed quantitative judgments," Organizational Behavior and Human Decision Processes, Elsevier, vol. 42(3), pages 303-323, December.
    6. Argote, Linda & Seabright, Mark A. & Dyer, Linda, 1986. "Individual versus group use of base-rate and individuating information," Organizational Behavior and Human Decision Processes, Elsevier, vol. 38(1), pages 65-75, August.
    7. French, Simon, 1981. "Consensus of opinion," European Journal of Operational Research, Elsevier, vol. 7(4), pages 332-340, August.
    8. Peter A. Morris, 1983. "An Axiomatic Approach to Expert Resolution," Management Science, INFORMS, vol. 29(1), pages 24-32, January.
    9. Shlyakhter, Alexander I. & Kammen, Daniel M. & Broido, Claire L. & Wilson, Richard, 1994. "Quantifying the credibility of energy projections from trends in past data : The US energy sector," Energy Policy, Elsevier, vol. 22(2), pages 119-130, February.
    10. Peter A. Morris, 1974. "Decision Analysis Expert Use," Management Science, INFORMS, vol. 20(9), pages 1233-1241, May.
    11. Uecker, Wc, 1982. "The Quality Of Group-Performance In Simplified Information Evaluation," Journal of Accounting Research, Wiley Blackwell, vol. 20(2), pages 388-402.
    12. James K. Hammitt & Alexander I. Shlyakhter, 1999. "The Expected Value of Information and the Probability of Surprise," Risk Analysis, John Wiley & Sons, vol. 19(1), pages 135-152, February.
    13. Sniezek, Janet A., 1989. "An examination of group process in judgmental forecasting," International Journal of Forecasting, Elsevier, vol. 5(2), pages 171-178.
    14. Innami, Ichiro, 1994. "The Quality of Group Decisions, Group Verbal Behavior, and Intervention," Organizational Behavior and Human Decision Processes, Elsevier, vol. 60(3), pages 409-430, December.
    15. Robert T. Clemen & Robert L. Winkler, 1985. "Limits for the Precision and Value of Information from Dependent Sources," Operations Research, INFORMS, vol. 33(2), pages 427-442, April.
    16. Joseph Lipscomb & Giovanni Parmigiani & Vic Hasselblad, 1998. "Combining Expert Judgment by Hierarchical Modeling: An Application to Physician Staffing," Management Science, INFORMS, vol. 44(2), pages 149-161, February.
    17. Robert T. Clemen & Robert L. Winkler, 1990. "Unanimity and Compromise Among Probability Forecasters," Management Science, INFORMS, vol. 36(7), pages 767-779, July.
    18. Dennis Lindley, 1983. "Reconciliation of Probability Distributions," Operations Research, INFORMS, vol. 31(5), pages 866-880, October.
    19. Robert L. Winkler & Thomas S. Wallsten & Ronald G. Whitfield & Harvey M. Richmond & Stanley R. Hayes & Arlene S. Rosenbaum, 1995. "An Assessment of the Risk of Chronic Lung Injury Attributable to Long-Term Ozone Exposure," Operations Research, INFORMS, vol. 43(1), pages 19-28, February.
    20. Robert L. Winkler, 1981. "Combining Probability Distributions from Dependent Information Sources," Management Science, INFORMS, vol. 27(4), pages 479-488, April.
    21. Clemen, Robert T., 1989. "Combining forecasts: A review and annotated bibliography," International Journal of Forecasting, Elsevier, vol. 5(4), pages 559-583.
    22. Robert L. Winkler & Roy M. Poses, 1993. "Evaluating and Combining Physicians' Probabilities of Survival in an Intensive Care Unit," Management Science, INFORMS, vol. 39(12), pages 1526-1543, December.
    23. Spyros Makridakis & Robert L. Winkler, 1983. "Averages of Forecasts: Some Empirical Results," Management Science, INFORMS, vol. 29(9), pages 987-996, September.
    24. Robert L. Winkler & Robert T. Clemen, 1992. "Sensitivity of Weights in Combining Forecasts," Operations Research, INFORMS, vol. 40(3), pages 609-614, June.
    25. Peter A. Morris, 1977. "Combining Expert Judgments: A Bayesian Approach," Management Science, INFORMS, vol. 23(7), pages 679-693, March.
    26. Robert T. Clemen & Robert L. Winkler, 1993. "Aggregating Point Estimates: A Flexible Modeling Approach," Management Science, INFORMS, vol. 39(4), pages 501-515, April.
    27. M. J. Lawrence & R. H. Edmundson & M. J. O'Connor, 1986. "The Accuracy of Combining Judgemental and Statistical Forecasts," Management Science, INFORMS, vol. 32(12), pages 1521-1532, December.
    28. Robert T. Clemen, 1986. "Calibration and the Aggregation of Probabilities," Management Science, INFORMS, vol. 32(3), pages 312-314, March.
    29. Robert T. Clemen & Terence Reilly, 1999. "Correlations and Copulas for Decision and Risk Analysis," Management Science, INFORMS, vol. 45(2), pages 208-224, February.
    30. Harry Otway & Detlof von Winterfeldt, 1992. "Expert Judgment in Risk Analysis and Management: Process, Context, and Pitfalls," Risk Analysis, John Wiley & Sons, vol. 12(1), pages 83-93, March.
    31. Sniezek, Janet A., 1992. "Groups under uncertainty: An examination of confidence in group decision making," Organizational Behavior and Human Decision Processes, Elsevier, vol. 52(1), pages 124-155, June.
    32. Sniezek, Janet A. & Henry, Rebecca A., 1989. "Accuracy and confidence in group judgment," Organizational Behavior and Human Decision Processes, Elsevier, vol. 43(1), pages 1-28, February.
    33. David C. Schmittlein & Jinho Kim & Donald G. Morrison, 1990. "Combining Forecasts: Operational Adjustments to Theoretically Optimal Rules," Management Science, INFORMS, vol. 36(9), pages 1044-1056, September.
    34. Robert L. Winkler, 1968. "The Consensus of Subjective Probability Distributions," Management Science, INFORMS, vol. 15(2), pages 61-75, October.
    35. Alexander I. Shlyakhter, 1994. "An Improved Framework for Uncertainty Analysis: Accounting for Unsuspected Errors," Risk Analysis, John Wiley & Sons, vol. 14(4), pages 441-447, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. James E. Smith & Detlof von Winterfeldt, 2004. "Anniversary Article: Decision Analysis in Management Science," Management Science, INFORMS, vol. 50(5), pages 561-574, May.
    2. Jason R. W. Merrick & J. Rene van Dorp & Amita Singh, 2005. "Analysis of Correlated Expert Judgments from Extended Pairwise Comparisons," Decision Analysis, INFORMS, vol. 2(1), pages 17-29, March.
    3. Paola Monari & Patrizia Agati, 2001. "Fiducial inference in combining expert judgements," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 10(1), pages 81-97, January.
    4. Jason R. W. Merrick, 2008. "Getting the Right Mix of Experts," Decision Analysis, INFORMS, vol. 5(1), pages 43-52, March.
    5. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    6. Weyant John, 2014. "Integrated assessment of climate change: state of the literature," Journal of Benefit-Cost Analysis, De Gruyter, vol. 5(3), pages 377-409, December.
    7. James K. Hammitt & Alexander I. Shlyakhter, 1999. "The Expected Value of Information and the Probability of Surprise," Risk Analysis, John Wiley & Sons, vol. 19(1), pages 135-152, February.
    8. Ali E. Abbas, 2009. "A Kullback-Leibler View of Linear and Log-Linear Pools," Decision Analysis, INFORMS, vol. 6(1), pages 25-37, March.
    9. Robert T. Clemen & Gregory W. Fischer & Robert L. Winkler, 2000. "Assessing Dependence: Some Experimental Results," Management Science, INFORMS, vol. 46(8), pages 1100-1115, August.
    10. Knut Are Aastveit & James Mitchell & Francesco Ravazzolo & Herman van Dijk, 2018. "The Evolution of Forecast Density Combinations in Economics," Tinbergen Institute Discussion Papers 18-069/III, Tinbergen Institute.
    11. Joseph Lipscomb & Giovanni Parmigiani & Vic Hasselblad, 1998. "Combining Expert Judgment by Hierarchical Modeling: An Application to Physician Staffing," Management Science, INFORMS, vol. 44(2), pages 149-161, February.
    12. Robert J. Budnitz & George Apostolakis & David M. Boore & Lloyd S. Cluff & Kevin J. Coppersmith & C. Allin Cornell & Peter A. Morris, 1998. "Use of Technical Expert Panels: Applications to Probabilistic Seismic Hazard Analysis," Risk Analysis, John Wiley & Sons, vol. 18(4), pages 463-469, August.
    13. Robert L. Winkler & Robert T. Clemen, 2004. "Multiple Experts vs. Multiple Methods: Combining Correlation Assessments," Decision Analysis, INFORMS, vol. 1(3), pages 167-176, September.
    14. Pennings, Clint L.P. & van Dalen, Jan & Rook, Laurens, 2019. "Coordinating judgmental forecasting: Coping with intentional biases," Omega, Elsevier, vol. 87(C), pages 46-56.
    15. David M. Pennock & Michael P. Wellman, 2005. "Graphical Models for Groups: Belief Aggregation and Risk Sharing," Decision Analysis, INFORMS, vol. 2(3), pages 148-164, September.
    16. Hurley, W. J. & Lior, D. U., 2002. "Combining expert judgment: On the performance of trimmed mean vote aggregation procedures in the presence of strategic voting," European Journal of Operational Research, Elsevier, vol. 140(1), pages 142-147, July.
    17. Budescu, David V. & Rantilla, Adrian K. & Yu, Hsiu-Ting & Karelitz, Tzur M., 2003. "The effects of asymmetry among advisors on the aggregation of their opinions," Organizational Behavior and Human Decision Processes, Elsevier, vol. 90(1), pages 178-194, January.
    18. Li, Yongquan & Zhu, Kaijie, 2009. "Information acquisition in new product introduction," European Journal of Operational Research, Elsevier, vol. 198(2), pages 618-625, October.
    19. Fu, Qi & Zhu, Kaijie, 2010. "Endogenous information acquisition in supply chain management," European Journal of Operational Research, Elsevier, vol. 201(2), pages 454-462, March.
    20. Wang, W., 1997. "Subjective estimation of the delay time distribution in maintenance modelling," European Journal of Operational Research, Elsevier, vol. 99(3), pages 516-529, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:riskan:v:19:y:1999:i:2:p:187-203. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1111/(ISSN)1539-6924 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.