IDEAS home Printed from https://ideas.repec.org/a/wly/riskan/v22y2002i1p67-81.html
   My bibliography  Save this article

Elicitation of Quantitative Data from a Heterogeneous Expert Panel: Formal Process and Application in Animal Health

Author

Listed:
  • Ine H. J. Van Der Fels‐Klerx
  • Louis H. J. Goossens
  • Helmut W. Saatkamp
  • Suzan H. S. Horst

Abstract

This paper presents a protocol for a formal expert judgment process using a heterogeneous expert panel aimed at the quantification of continuous variables. The emphasis is on the process's requirements related to the nature of expertise within the panel, in particular the heterogeneity of both substantive and normative expertise. The process provides the opportunity for interaction among the experts so that they fully understand and agree upon the problem at hand, including qualitative aspects relevant to the variables of interest, prior to the actual quantification task. Individual experts' assessments on the variables of interest, cast in the form of subjective probability density functions, are elicited with a minimal demand for normative expertise. The individual experts' assessments are aggregated into a single probability density function per variable, thereby weighting the experts according to their expertise. Elicitation techniques proposed include the Delphi technique for the qualitative assessment task and the ELI method for the actual quantitative assessment task. Appropriately, the Classical model was used to weight the experts' assessments in order to construct a single distribution per variable. Applying this model, the experts' quality typically was based on their performance on seed variables. An application of the proposed protocol in the broad and multidisciplinary field of animal health is presented. Results of this expert judgment process showed that the proposed protocol in combination with the proposed elicitation and analysis techniques resulted in valid data on the (continuous) variables of interest. In conclusion, the proposed protocol for a formal expert judgment process aimed at the elicitation of quantitative data from a heterogeneous expert panel provided satisfactory results. Hence, this protocol might be useful for expert judgment studies in other broad and/or multidisciplinary fields of interest.

Suggested Citation

  • Ine H. J. Van Der Fels‐Klerx & Louis H. J. Goossens & Helmut W. Saatkamp & Suzan H. S. Horst, 2002. "Elicitation of Quantitative Data from a Heterogeneous Expert Panel: Formal Process and Application in Animal Health," Risk Analysis, John Wiley & Sons, vol. 22(1), pages 67-81, February.
    • Handle: RePEc:wly:riskan:v:22:y:2002:i:1:p:67-81
    DOI: 10.1111/0272-4332.t01-1-00007
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/0272-4332.t01-1-00007
    Download Restriction: no
    File URL: https://libkey.io/10.1111/0272-4332.t01-1-00007?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. L.H.J. Goossens & R.M. Cooke & F Woudenberg & P Van Der Torn, 1998. "Expert judgement and lethal toxicity of inhaled chemicals," Journal of Risk Research, Taylor & Francis Journals, vol. 1(2), pages 117-133, April.
    2. Robert J. Budnitz & George Apostolakis & David M. Boore & Lloyd S. Cluff & Kevin J. Coppersmith & C. Allin Cornell & Peter A. Morris, 1998. "Use of Technical Expert Panels: Applications to Probabilistic Seismic Hazard Analysis," Risk Analysis, John Wiley & Sons, vol. 18(4), pages 463-469, August.
    3. Van Lenthe, Jelle, 1993. "ELI: An Interactive Elicitation Technique for Subjective Probability Distributions," Organizational Behavior and Human Decision Processes, Elsevier, vol. 55(3), pages 379-413, August.
    4. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    5. Harry Otway & Detlof von Winterfeldt, 1992. "Expert Judgment in Risk Analysis and Management: Process, Context, and Pitfalls," Risk Analysis, John Wiley & Sons, vol. 12(1), pages 83-93, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. H. J. Van der Fels‐Klerx & Roger M. Cooke & Maarten N. Nauta & Louis H. Goossens & Arie H. Havelaar, 2005. "A Structured Expert Judgment Study for a Model of Campylobacter Transmission During Broiler‐Chicken Processing," Risk Analysis, John Wiley & Sons, vol. 25(1), pages 109-124, February.
    2. Mitchell J. Small, 2008. "Methods for Assessing Uncertainty in Fundamental Assumptions and Associated Models for Cancer Risk Assessment," Risk Analysis, John Wiley & Sons, vol. 28(5), pages 1289-1308, October.
    3. João Delgado & Simon Pollard & Emma Snary & Edgar Black & George Prpich & Phil Longhurst, 2013. "A Systems Approach to the Policy‐Level Risk Assessment of Exotic Animal Diseases: Network Model and Application to Classical Swine Fever," Risk Analysis, John Wiley & Sons, vol. 33(8), pages 1454-1472, August.
    4. Crawford, Megan M., 2019. "A comprehensive scenario intervention typology," Technological Forecasting and Social Change, Elsevier, vol. 149(C).
    5. Sandra Hoffmann & Paul Fischbeck & Alan Krupnick & Michael McWilliams, 2007. "Elicitation from Large, Heterogeneous Expert Panels: Using Multiple Uncertainty Measures to Characterize Information Quality for Decision Analysis," Decision Analysis, INFORMS, vol. 4(2), pages 91-109, June.
    6. Hoffmann, Sandra A. & Fischbeck, Paul S. & Krupnick, Alan J. & McWilliams, Michael, 2006. "Eliciting Information on Uncertainty from Heterogeneous Expert Panels: Attributing U.S. Foodborne Pathogen Illness to Food Consumption," Discussion Papers 10444, Resources for the Future.
    7. Cameron J. Williams & Kevin J. Wilson & Nina Wilson, 2021. "A comparison of prior elicitation aggregation using the classical method and SHELF," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(3), pages 920-940, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. H. J. Van der Fels‐Klerx & Roger M. Cooke & Maarten N. Nauta & Louis H. Goossens & Arie H. Havelaar, 2005. "A Structured Expert Judgment Study for a Model of Campylobacter Transmission During Broiler‐Chicken Processing," Risk Analysis, John Wiley & Sons, vol. 25(1), pages 109-124, February.
    2. Mario Paulo Brito & Gwyn Griffiths & Peter Challenor, 2010. "Risk Analysis for Autonomous Underwater Vehicle Operations in Extreme Environments," Risk Analysis, John Wiley & Sons, vol. 30(12), pages 1771-1788, December.
    3. Yuyu Fan & David V. Budescu & David Mandel & Mark Himmelstein, 2019. "Improving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments," Decision Analysis, INFORMS, vol. 16(3), pages 197-217, September.
    4. A. Procter & T. McDaniels & R. Vignola, 2017. "Using expert judgments to inform economic evaluation of ecosystem-based adaptation decisions: watershed management for enhancing water supply for Tegucigalpa, Honduras," Environment Systems and Decisions, Springer, vol. 37(4), pages 410-422, December.
    5. Robert L. Winkler*, 2015. "Equal Versus Differential Weighting in Combining Forecasts," Risk Analysis, John Wiley & Sons, vol. 35(1), pages 16-18, January.
    6. Avner Engel & Shalom Shachar, 2006. "Measuring and optimizing systems' quality costs and project duration," Systems Engineering, John Wiley & Sons, vol. 9(3), pages 259-280, September.
    7. J. C. Helton & F. J. Davis, 2002. "Illustration of Sampling‐Based Methods for Uncertainty and Sensitivity Analysis," Risk Analysis, John Wiley & Sons, vol. 22(3), pages 591-622, June.
    8. D. Warner North, 2010. "Probability Theory and Consistent Reasoning," Risk Analysis, John Wiley & Sons, vol. 30(3), pages 377-380, March.
    9. Franz Dietrich & Christian List, 2017. "Probabilistic opinion pooling generalized. Part one: general agendas," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 48(4), pages 747-786, April.
    10. repec:cup:judgdm:v:13:y:2018:i:6:p:607-621 is not listed on IDEAS
    11. Robert L. Winkler & Robert T. Clemen, 2004. "Multiple Experts vs. Multiple Methods: Combining Correlation Assessments," Decision Analysis, INFORMS, vol. 1(3), pages 167-176, September.
    12. Minh Ha-Duong, 2008. "Hierarchical fusion of expert opinion in the Transferable Belief Model, application on climate sensitivity," Post-Print halshs-00112129, HAL.
    13. Pennings, Clint L.P. & van Dalen, Jan & Rook, Laurens, 2019. "Coordinating judgmental forecasting: Coping with intentional biases," Omega, Elsevier, vol. 87(C), pages 46-56.
    14. Jeffrey M. Keisler, 2005. "Additivity of Information Value in Two‐Act Linear Loss Decisions with Normal Priors," Risk Analysis, John Wiley & Sons, vol. 25(2), pages 351-359, April.
    15. Lin, Shi-Woei & Bier, Vicki M., 2008. "A study of expert overconfidence," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 711-721.
    16. Chunchang Zhang & Hu Sun & Yuanyuan Zhang & Gen Li & Shibo Li & Junyu Chang & Gongqian Shi, 2023. "Fire Accident Risk Analysis of Lithium Battery Energy Storage Systems during Maritime Transportation," Sustainability, MDPI, vol. 15(19), pages 1-12, September.
    17. Jason R. W. Merrick, 2008. "Getting the Right Mix of Experts," Decision Analysis, INFORMS, vol. 5(1), pages 43-52, March.
    18. Gregory F. Nemet & Erin Baker, 2009. "Demand Subsidies Versus R&D: Comparing the Uncertain Impacts of Policy on a Pre-commercial Low-carbon Energy Technology," The Energy Journal, International Association for Energy Economics, vol. 0(Number 4), pages 49-80.
    19. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    20. Gillingham, Kenneth & Nordhaus, William & Anthoff, David & Bosetti, Valentina & McJeon, Haewon & Blanford, Geoffrey & Christensen, Peter & Reilly, John & Sztorc, Paul, "undated". "Modeling Uncertainty in Climate Change: A Multi-Model Comparison," MITP: Mitigation, Innovation and Transformation Pathways 232219, Fondazione Eni Enrico Mattei (FEEM).
    21. Richard Volkert & Jerrell Stracener & Junfang Yu, 2014. "Incorporating a Measure of Uncertainty into Systems of Systems Development Performance Measures," Systems Engineering, John Wiley & Sons, vol. 17(3), pages 297-312, September.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:riskan:v:22:y:2002:i:1:p:67-81. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1111/(ISSN)1539-6924 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.