IDEAS home Printed from https://ideas.repec.org/a/wly/riskan/v22y2002i1p67-81.html
   My bibliography  Save this article

Elicitation of Quantitative Data from a Heterogeneous Expert Panel: Formal Process and Application in Animal Health

Author

Listed:
  • Ine H. J. Van Der Fels‐Klerx
  • Louis H. J. Goossens
  • Helmut W. Saatkamp
  • Suzan H. S. Horst

Abstract

This paper presents a protocol for a formal expert judgment process using a heterogeneous expert panel aimed at the quantification of continuous variables. The emphasis is on the process's requirements related to the nature of expertise within the panel, in particular the heterogeneity of both substantive and normative expertise. The process provides the opportunity for interaction among the experts so that they fully understand and agree upon the problem at hand, including qualitative aspects relevant to the variables of interest, prior to the actual quantification task. Individual experts' assessments on the variables of interest, cast in the form of subjective probability density functions, are elicited with a minimal demand for normative expertise. The individual experts' assessments are aggregated into a single probability density function per variable, thereby weighting the experts according to their expertise. Elicitation techniques proposed include the Delphi technique for the qualitative assessment task and the ELI method for the actual quantitative assessment task. Appropriately, the Classical model was used to weight the experts' assessments in order to construct a single distribution per variable. Applying this model, the experts' quality typically was based on their performance on seed variables. An application of the proposed protocol in the broad and multidisciplinary field of animal health is presented. Results of this expert judgment process showed that the proposed protocol in combination with the proposed elicitation and analysis techniques resulted in valid data on the (continuous) variables of interest. In conclusion, the proposed protocol for a formal expert judgment process aimed at the elicitation of quantitative data from a heterogeneous expert panel provided satisfactory results. Hence, this protocol might be useful for expert judgment studies in other broad and/or multidisciplinary fields of interest.

Suggested Citation

  • Ine H. J. Van Der Fels‐Klerx & Louis H. J. Goossens & Helmut W. Saatkamp & Suzan H. S. Horst, 2002. "Elicitation of Quantitative Data from a Heterogeneous Expert Panel: Formal Process and Application in Animal Health," Risk Analysis, John Wiley & Sons, vol. 22(1), pages 67-81, February.
    • Handle: RePEc:wly:riskan:v:22:y:2002:i:1:p:67-81
    DOI: 10.1111/0272-4332.t01-1-00007
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/0272-4332.t01-1-00007
    Download Restriction: no
    File URL: https://libkey.io/10.1111/0272-4332.t01-1-00007?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. L.H.J. Goossens & R.M. Cooke & F Woudenberg & P Van Der Torn, 1998. "Expert judgement and lethal toxicity of inhaled chemicals," Journal of Risk Research, Taylor & Francis Journals, vol. 1(2), pages 117-133, April.
    2. Robert J. Budnitz & George Apostolakis & David M. Boore & Lloyd S. Cluff & Kevin J. Coppersmith & C. Allin Cornell & Peter A. Morris, 1998. "Use of Technical Expert Panels: Applications to Probabilistic Seismic Hazard Analysis," Risk Analysis, John Wiley & Sons, vol. 18(4), pages 463-469, August.
    3. Van Lenthe, Jelle, 1993. "ELI: An Interactive Elicitation Technique for Subjective Probability Distributions," Organizational Behavior and Human Decision Processes, Elsevier, vol. 55(3), pages 379-413, August.
    4. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    5. Harry Otway & Detlof von Winterfeldt, 1992. "Expert Judgment in Risk Analysis and Management: Process, Context, and Pitfalls," Risk Analysis, John Wiley & Sons, vol. 12(1), pages 83-93, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. H. J. Van der Fels‐Klerx & Roger M. Cooke & Maarten N. Nauta & Louis H. Goossens & Arie H. Havelaar, 2005. "A Structured Expert Judgment Study for a Model of Campylobacter Transmission During Broiler‐Chicken Processing," Risk Analysis, John Wiley & Sons, vol. 25(1), pages 109-124, February.
    2. Mitchell J. Small, 2008. "Methods for Assessing Uncertainty in Fundamental Assumptions and Associated Models for Cancer Risk Assessment," Risk Analysis, John Wiley & Sons, vol. 28(5), pages 1289-1308, October.
    3. João Delgado & Simon Pollard & Emma Snary & Edgar Black & George Prpich & Phil Longhurst, 2013. "A Systems Approach to the Policy‐Level Risk Assessment of Exotic Animal Diseases: Network Model and Application to Classical Swine Fever," Risk Analysis, John Wiley & Sons, vol. 33(8), pages 1454-1472, August.
    4. Crawford, Megan M., 2019. "A comprehensive scenario intervention typology," Technological Forecasting and Social Change, Elsevier, vol. 149(C).
    5. Sandra Hoffmann & Paul Fischbeck & Alan Krupnick & Michael McWilliams, 2007. "Elicitation from Large, Heterogeneous Expert Panels: Using Multiple Uncertainty Measures to Characterize Information Quality for Decision Analysis," Decision Analysis, INFORMS, vol. 4(2), pages 91-109, June.
    6. Hoffmann, Sandra A. & Fischbeck, Paul S. & Krupnick, Alan J. & McWilliams, Michael, 2006. "Eliciting Information on Uncertainty from Heterogeneous Expert Panels: Attributing U.S. Foodborne Pathogen Illness to Food Consumption," Discussion Papers 10444, Resources for the Future.
    7. Cameron J. Williams & Kevin J. Wilson & Nina Wilson, 2021. "A comparison of prior elicitation aggregation using the classical method and SHELF," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(3), pages 920-940, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yuyu Fan & David V. Budescu & David Mandel & Mark Himmelstein, 2019. "Improving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments," Decision Analysis, INFORMS, vol. 16(3), pages 197-217, September.
    2. A. Procter & T. McDaniels & R. Vignola, 2017. "Using expert judgments to inform economic evaluation of ecosystem-based adaptation decisions: watershed management for enhancing water supply for Tegucigalpa, Honduras," Environment Systems and Decisions, Springer, vol. 37(4), pages 410-422, December.
    3. H. J. Van der Fels‐Klerx & Roger M. Cooke & Maarten N. Nauta & Louis H. Goossens & Arie H. Havelaar, 2005. "A Structured Expert Judgment Study for a Model of Campylobacter Transmission During Broiler‐Chicken Processing," Risk Analysis, John Wiley & Sons, vol. 25(1), pages 109-124, February.
    4. Robert L. Winkler*, 2015. "Equal Versus Differential Weighting in Combining Forecasts," Risk Analysis, John Wiley & Sons, vol. 35(1), pages 16-18, January.
    5. Mario Paulo Brito & Gwyn Griffiths & Peter Challenor, 2010. "Risk Analysis for Autonomous Underwater Vehicle Operations in Extreme Environments," Risk Analysis, John Wiley & Sons, vol. 30(12), pages 1771-1788, December.
    6. Kenneth Gillingham & William D. Nordhaus & David Anthoff & Geoffrey Blanford & Valentina Bosetti & Peter Christensen & Haewon McJeon & John Reilly & Paul Sztorc, 2015. "Modeling Uncertainty in Climate Change: A Multi-Model Comparison," NBER Working Papers 21637, National Bureau of Economic Research, Inc.
    7. Avner Engel & Shalom Shachar, 2006. "Measuring and optimizing systems' quality costs and project duration," Systems Engineering, John Wiley & Sons, vol. 9(3), pages 259-280, September.
    8. Roland W. Scholz & Ralf Hansmann, 2007. "Combining Experts' Risk Judgments on Technology Performance of Phytoremediation: Self‐Confidence Ratings, Averaging Procedures, and Formative Consensus Building," Risk Analysis, John Wiley & Sons, vol. 27(1), pages 225-240, February.
    9. Atanasov, Pavel & Witkowski, Jens & Ungar, Lyle & Mellers, Barbara & Tetlock, Philip, 2020. "Small steps to accuracy: Incremental belief updaters are better forecasters," Organizational Behavior and Human Decision Processes, Elsevier, vol. 160(C), pages 19-35.
    10. J. C. Helton & F. J. Davis, 2002. "Illustration of Sampling‐Based Methods for Uncertainty and Sensitivity Analysis," Risk Analysis, John Wiley & Sons, vol. 22(3), pages 591-622, June.
    11. D. Warner North, 2010. "Probability Theory and Consistent Reasoning," Risk Analysis, John Wiley & Sons, vol. 30(3), pages 377-380, March.
    12. Franz Dietrich & Christian List, 2017. "Probabilistic opinion pooling generalized. Part one: general agendas," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 48(4), pages 747-786, April.
    13. repec:cup:judgdm:v:13:y:2018:i:6:p:607-621 is not listed on IDEAS
    14. Patrick Afflerbach & Christopher Dun & Henner Gimpel & Dominik Parak & Johannes Seyfried, 2021. "A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(4), pages 329-348, August.
    15. George E. Apostolakis & Douglas M. Lemon, 2005. "A Screening Methodology for the Identification and Ranking of Infrastructure Vulnerabilities Due to Terrorism," Risk Analysis, John Wiley & Sons, vol. 25(2), pages 361-376, April.
    16. Chiara Franzoni & Paula Stephan & Reinhilde Veugelers, 2022. "Funding Risky Research," Entrepreneurship and Innovation Policy and the Economy, University of Chicago Press, vol. 1(1), pages 103-133.
    17. Robert F. Bordley, 2009. "Combining the Opinions of Experts Who Partition Events Differently," Decision Analysis, INFORMS, vol. 6(1), pages 38-46, March.
    18. Jason R. W. Merrick & J. Rene van Dorp & Jack Harrald & Thomas Mazzuchi & John E. Spahn & Martha Grabowski, 2000. "A systems approach to managing oil transportation risk in Prince William Sound," Systems Engineering, John Wiley & Sons, vol. 3(3), pages 128-142.
    19. Donnacha Bolger & Brett Houlding, 2016. "Reliability updating in linear opinion pooling for multiple decision makers," Journal of Risk and Reliability, , vol. 230(3), pages 309-322, June.
    20. Robert L. Winkler & Robert T. Clemen, 2004. "Multiple Experts vs. Multiple Methods: Combining Correlation Assessments," Decision Analysis, INFORMS, vol. 1(3), pages 167-176, September.
    21. Kenichiro McAlinn & Knut Are Aastveit & Jouchi Nakajima & Mike West, 2020. "Multivariate Bayesian Predictive Synthesis in Macroeconomic Forecasting," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 115(531), pages 1092-1110, July.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:riskan:v:22:y:2002:i:1:p:67-81. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1111/(ISSN)1539-6924 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.