IDEAS home Printed from https://ideas.repec.org/a/inm/ordeca/v4y2007i2p91-109.html

Elicitation from Large, Heterogeneous Expert Panels: Using Multiple Uncertainty Measures to Characterize Information Quality for Decision Analysis

Author

Listed:
  • Sandra Hoffmann

    (Resources for the Future, 1616 P St. NW, Washington, D.C. 20036)

  • Paul Fischbeck

    (Department of Social and Decision Sciences, Carnegie Mellon University, Pittsburgh, Pennsylvania 15213)

  • Alan Krupnick

    (Resources for the Future, 1616 P St. NW, Washington, D.C. 20036)

  • Michael McWilliams

    (Department of Economics, University of Michigan, Ann Arbor, Michigan 48109)

Abstract

Decision analysts are frequently called on to help inform decision makers in cases involving considerable uncertainty. In such situations, expert elicitation of parameter values is frequently used to supplement more conventional research. Expert elicitations typically rely on small panels of experts. However, in cases where the information needed for risk management must draw on a broad range of disciplines or types of professional backgrounds and experience, a larger, more heterogeneous expert panel is needed. In this paper we develop a formal protocol and a suite of uncertainty measures for this work. The protocol uses formal survey methods to take advantage of variation in individual expert uncertainty and heterogeneity among experts as a means of quantifying and comparing sources of uncertainty about parameters of interest. We illustrate the use of this protocol with an expert elicitation on the distribution of foodborne illness in the United States across foods. In the survey, experts are asked to attribute illnesses associated with one of eleven major foodborne pathogens to the consumption of one of eleven categories of food. Results show how the distributions of multiple measures of uncertainty (e.g., agreement of experts and uncertainty in knowledge), made feasible by use of a large panel of experts, can help identify which of several types of risk management actions may be most appropriate.

Suggested Citation

  • Sandra Hoffmann & Paul Fischbeck & Alan Krupnick & Michael McWilliams, 2007. "Elicitation from Large, Heterogeneous Expert Panels: Using Multiple Uncertainty Measures to Characterize Information Quality for Decision Analysis," Decision Analysis, INFORMS, vol. 4(2), pages 91-109, June.
    • Handle: RePEc:inm:ordeca:v:4:y:2007:i:2:p:91-109
    DOI: 10.1287/deca.1070.0090
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/deca.1070.0090
    Download Restriction: no
    File URL: https://libkey.io/10.1287/deca.1070.0090?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    2. F. Owen Hoffman & Stan Kaplan, 1999. "Beyond the Domain of Direct Observation: How to Specify a Probability Distribution that Represents the “State of Knowledge” About Uncertain Inputs," Risk Analysis, John Wiley & Sons, vol. 19(1), pages 131-134, February.
    3. Jason R. W. Merrick & J. Rene van Dorp & Amita Singh, 2005. "Analysis of Correlated Expert Judgments from Extended Pairwise Comparisons," Decision Analysis, INFORMS, vol. 2(1), pages 17-29, March.
    4. Hoffmann, Sandra & Fischbeck, Paul & Krupnick, Alan & McWilliams, Michael, 2008. "Informing risk-mitigation priorities using uncertainty measures derived from heterogeneous expert panels: A demonstration using foodborne pathogens," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 687-698.
    5. Richard P. Barke & Hank C. Jenkins‐Smith, 1993. "Politics and Scientific Expertise: Scientists, Risk Perception, and Nuclear Waste Policy," Risk Analysis, John Wiley & Sons, vol. 13(4), pages 425-439, August.
    6. M. Granger Morgan & H. Keith Florig & Michael L. DeKay & Paul Fischbeck, 2000. "Categorizing Risks for Risk Ranking," Risk Analysis, John Wiley & Sons, vol. 20(1), pages 49-58, February.
    7. Francis Adams, 2006. "Expert elicitation and Bayesian analysis of construction contract risks: an investigation," Construction Management and Economics, Taylor & Francis Journals, vol. 24(1), pages 81-96.
    8. Ine H. J. Van Der Fels‐Klerx & Louis H. J. Goossens & Helmut W. Saatkamp & Suzan H. S. Horst, 2002. "Elicitation of Quantitative Data from a Heterogeneous Expert Panel: Formal Process and Application in Animal Health," Risk Analysis, John Wiley & Sons, vol. 22(1), pages 67-81, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. L. Robin Keller, 2009. "From the Editor..," Decision Analysis, INFORMS, vol. 6(1), pages 1-3, March.
    2. Robert F. Bordley, 2009. "Combining the Opinions of Experts Who Partition Events Differently," Decision Analysis, INFORMS, vol. 6(1), pages 38-46, March.
    3. Laura M. Keating & Lea Randall & Rebecca Stanton & Casey McCormack & Michael Lucid & Travis Seaborn & Sarah J. Converse & Stefano Canessa & Axel Moehrenschlager, 2023. "Using Decision Analysis to Determine the Feasibility of a Conservation Translocation," Decision Analysis, INFORMS, vol. 20(4), pages 295-310, December.
    4. Rakesh K. Sarin, 2013. "From the Editor —Optimal Betting, Reducing Unnecessary Mammography in Breast Cancer Diagnosis, Product Line Design, and Value of Information," Decision Analysis, INFORMS, vol. 10(3), pages 187-188, September.
    5. Kesten Green & J. Scott Armstrong & Andreas Graefe, 2007. "Methods to Elicit Forecasts from Groups: Delphi and Prediction Markets Compared," Foresight: The International Journal of Applied Forecasting, International Institute of Forecasters, issue 8, pages 17-20, Fall.
    6. L. Robin Keller & Kelly M. Kophazi, 2008. "From the Editors..," Decision Analysis, INFORMS, vol. 5(2), pages 57-59, June.
    7. Alison Peel & Michelle Jenks & Moni Choudhury & Rosemary Lovett & Juan Carlos Rejon-Parrilla & Andrew Sims & Joyce Craig, 2018. "Use of Expert Judgement Across NICE Guidance-Making Programmes: A Review of Current Processes and Suitability of Existing Tools to Support the Use of Expert Elicitation," Applied Health Economics and Health Policy, Springer, vol. 16(6), pages 819-836, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hoffmann, Sandra A. & Fischbeck, Paul S. & Krupnick, Alan J. & McWilliams, Michael, 2006. "Eliciting Information on Uncertainty from Heterogeneous Expert Panels: Attributing U.S. Foodborne Pathogen Illness to Food Consumption," Discussion Papers 10444, Resources for the Future.
    2. Erin Baker & Olaitan Olaleye, 2013. "Combining Experts: Decomposition and Aggregation Order," Risk Analysis, John Wiley & Sons, vol. 33(6), pages 1116-1127, June.
    3. Ibsen Chivatá Cárdenas & Saad S.H. Al‐jibouri & Johannes I.M. Halman & Frits A. van Tol, 2013. "Capturing and Integrating Knowledge for Managing Risks in Tunnel Works," Risk Analysis, John Wiley & Sons, vol. 33(1), pages 92-108, January.
    4. H. J. Van der Fels‐Klerx & Roger M. Cooke & Maarten N. Nauta & Louis H. Goossens & Arie H. Havelaar, 2005. "A Structured Expert Judgment Study for a Model of Campylobacter Transmission During Broiler‐Chicken Processing," Risk Analysis, John Wiley & Sons, vol. 25(1), pages 109-124, February.
    5. Mitchell J. Small, 2008. "Methods for Assessing Uncertainty in Fundamental Assumptions and Associated Models for Cancer Risk Assessment," Risk Analysis, John Wiley & Sons, vol. 28(5), pages 1289-1308, October.
    6. L. Robin Keller, 2008. "From the Editor..," Decision Analysis, INFORMS, vol. 5(1), pages 1-4, March.
    7. Amanda P. Rehr & Mitchell J. Small & Paul S. Fischbeck & Patricia Bradley & William S. Fisher, 2014. "The role of scientific studies in building consensus in environmental decision making: a coral reef example," Environment Systems and Decisions, Springer, vol. 34(1), pages 60-87, March.
    8. Denys Yemshanov & Frank H. Koch & Yakov Ben‐Haim & Marla Downing & Frank Sapio & Marty Siltanen, 2013. "A New Multicriteria Risk Mapping Approach Based on a Multiattribute Frontier Concept," Risk Analysis, John Wiley & Sons, vol. 33(9), pages 1694-1709, September.
    9. Ibsen Chivatá Cárdenas & Saad S. H. Al‐Jibouri & Johannes I. M. Halman & Wim van de Linde & Frank Kaalberg, 2014. "Using Prior Risk‐Related Knowledge to Support Risk Management Decisions: Lessons Learnt from a Tunneling Project," Risk Analysis, John Wiley & Sons, vol. 34(10), pages 1923-1943, October.
    10. Avner Engel & Shalom Shachar, 2006. "Measuring and optimizing systems' quality costs and project duration," Systems Engineering, John Wiley & Sons, vol. 9(3), pages 259-280, September.
    11. Gillingham, Kenneth & Nordhaus, William & Anthoff, David & Bosetti, Valentina & McJeon, Haewon & Blanford, Geoffrey & Christensen, Peter & Reilly, John & Sztorc, Paul, "undated". "Modeling Uncertainty in Climate Change: A Multi-Model Comparison," MITP: Mitigation, Innovation and Transformation Pathways 232219, Fondazione Eni Enrico Mattei (FEEM).
    12. Atanasov, Pavel & Witkowski, Jens & Ungar, Lyle & Mellers, Barbara & Tetlock, Philip, 2020. "Small steps to accuracy: Incremental belief updaters are better forecasters," Organizational Behavior and Human Decision Processes, Elsevier, vol. 160(C), pages 19-35.
    13. Yawson, Robert M. & Kuzma, Jennifer, 2010. "Evidence review and experts’ opinion on consumer acceptance of agrifood nanotechnology," MPRA Paper 40807, University Library of Munich, Germany.
    14. Timothy L. McDaniels & Stephanie E. Chang & David Hawkins & Gerard Chew & Holly Longstaff, 2015. "Towards disaster-resilient cities: an approach for setting priorities in infrastructure mitigation efforts," Environment Systems and Decisions, Springer, vol. 35(2), pages 252-263, June.
    15. Carl Danner & Paul Schulman, 2019. "Rethinking Risk Assessment for Public Utility Safety Regulation," Risk Analysis, John Wiley & Sons, vol. 39(5), pages 1044-1059, May.
    16. Rob Goble, 2021. "Through a Glass Darkly: How Natural Science and Technical Communities Looked at Social Science Advances in Understanding Risk," Risk Analysis, John Wiley & Sons, vol. 41(3), pages 414-428, March.
    17. Kit S. Hagemann & Joachim Scholderer, 2009. "Hot Potato: Expert‐Consumer Differences in the Perception of a Second‐Generation Novel Food," Risk Analysis, John Wiley & Sons, vol. 29(7), pages 1041-1055, July.
    18. Naomi Aoki, 2018. "Who Would Be Willing to Accept Disaster Debris in Their Backyard? Investigating the Determinants of Public Attitudes in Post‐Fukushima Japan," Risk Analysis, John Wiley & Sons, vol. 38(3), pages 535-547, March.
    19. Patrick Afflerbach & Christopher Dun & Henner Gimpel & Dominik Parak & Johannes Seyfried, 2021. "A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(4), pages 329-348, August.
    20. George E. Apostolakis & Douglas M. Lemon, 2005. "A Screening Methodology for the Identification and Ranking of Infrastructure Vulnerabilities Due to Terrorism," Risk Analysis, John Wiley & Sons, vol. 25(2), pages 361-376, April.

    More about this item

    Keywords

    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ordeca:v:4:y:2007:i:2:p:91-109. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.