IDEAS home Printed from https://ideas.repec.org/a/inm/ordeca/v2y2005i1p17-29.html
   My bibliography  Save this article

Analysis of Correlated Expert Judgments from Extended Pairwise Comparisons

Author

Listed:
  • Jason R. W. Merrick

    (Department of Statistical Sciences and Operations Research, Virginia Commonwealth University, P.O. Box 843083, 1001 West Main Street, Richmond, Virginia 23284)

  • J. Rene van Dorp

    (George Washington University, 1776 G. Street, NW, Washington, DC 20052)

  • Amita Singh

    (George Washington University, 1776 G. Street, NW, Washington, DC 20052)

Abstract

We develop a Bayesian multivariate analysis of expert judgment elicited using an extended form of pairwise comparisons. The method can be used to estimate the effect of multiple factors on the probability of an event and can be applied in risk analysis and other decision problems. The model, which parallels Bayesian models for combining expert judgments, provides predictions of the quantity of interest that incorporate dependencies among the various experts. In this form we may learn about the dependencies between the experts from their responses. The analysis is applied to a real data set of expert judgments elicited during the Washington State Ferries Risk Assessment. The effect of the statistical dependence among experts is compared to an analysis assuming independence among them.

Suggested Citation

  • Jason R. W. Merrick & J. Rene van Dorp & Amita Singh, 2005. "Analysis of Correlated Expert Judgments from Extended Pairwise Comparisons," Decision Analysis, INFORMS, vol. 2(1), pages 17-29, March.
    • Handle: RePEc:inm:ordeca:v:2:y:2005:i:1:p:17-29
    DOI: 10.1287/deca.1050.0031
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/deca.1050.0031
    Download Restriction: no
    File URL: https://libkey.io/10.1287/deca.1050.0031?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Johan R. Van Dorp & Jason R. W. Merrick & John R. Harrald & Thomas A. Mazzuchi & Martha Grabowski, 2001. "A Risk Management Procedure for the Washington State Ferries," Risk Analysis, John Wiley & Sons, vol. 21(1), pages 127-142, February.
    2. Mohamed N. Jouini & Robert T. Clemen, 1996. "Copula Models for Aggregating Expert Opinions," Operations Research, INFORMS, vol. 44(3), pages 444-457, June.
    3. French, Simon, 1981. "Consensus of opinion," European Journal of Operational Research, Elsevier, vol. 7(4), pages 332-340, August.
    4. Peter A. Morris, 1983. "An Axiomatic Approach to Expert Resolution," Management Science, INFORMS, vol. 29(1), pages 24-32, January.
    5. Peter A. Morris, 1974. "Decision Analysis Expert Use," Management Science, INFORMS, vol. 20(9), pages 1233-1241, May.
    6. Dewispelare, Aaron R. & Herren, L. Tandy & Clemen, Robert T., 1995. "The use of probability elicitation in the high-level nuclear waste regulation program," International Journal of Forecasting, Elsevier, vol. 11(1), pages 5-24, March.
    7. Robert T. Clemen, 1987. "Combining Overlapping Information," Management Science, INFORMS, vol. 33(3), pages 373-380, March.
    8. Robert T. Clemen & Robert L. Winkler, 1985. "Limits for the Precision and Value of Information from Dependent Sources," Operations Research, INFORMS, vol. 33(2), pages 427-442, April.
    9. Dennis Lindley, 1983. "Reconciliation of Probability Distributions," Operations Research, INFORMS, vol. 31(5), pages 866-880, October.
    10. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    11. Robert L. Winkler, 1981. "Combining Probability Distributions from Dependent Information Sources," Management Science, INFORMS, vol. 27(4), pages 479-488, April.
    12. Peter A. Morris, 1977. "Combining Expert Judgments: A Bayesian Approach," Management Science, INFORMS, vol. 23(7), pages 679-693, March.
    13. Robert T. Clemen, 1986. "Calibration and the Aggregation of Probabilities," Management Science, INFORMS, vol. 32(3), pages 312-314, March.
    14. Robert T. Clemen & Terence Reilly, 1999. "Correlations and Copulas for Decision and Risk Analysis," Management Science, INFORMS, vol. 45(2), pages 208-224, February.
    15. Jason R. W. Merrick & J. René van Dorp & Thomas Mazzuchi & John R. Harrald & John E. Spahn & Martha Grabowski, 2002. "The Prince William Sound Risk Assessment," Interfaces, INFORMS, vol. 32(6), pages 25-40, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. L. Robin Keller, 2007. "From the Editor..," Decision Analysis, INFORMS, vol. 4(3), pages 111-113, September.
    2. Jason R. W. Merrick & Fabrizio Ruggeri & Refik Soyer & L. Robin Keller, 2012. "From the Editors---Games and Decisions in Reliability and Risk," Decision Analysis, INFORMS, vol. 9(2), pages 81-85, June.
    3. Erin Baker & Olaitan Olaleye, 2013. "Combining Experts: Decomposition and Aggregation Order," Risk Analysis, John Wiley & Sons, vol. 33(6), pages 1116-1127, June.
    4. L. Robin Keller & Ali Abbas & Manel Baucells & Vicki M. Bier & David Budescu & John C. Butler & Philippe Delquié & Jason R. W. Merrick & Ahti Salo & George Wu, 2010. "From the Editors..," Decision Analysis, INFORMS, vol. 7(4), pages 327-330, December.
      • L. Robin Keller & Manel Baucells & Kevin F. McCardle & Gregory S. Parnell & Ahti Salo, 2007. "From the Editors..," Decision Analysis, INFORMS, vol. 4(4), pages 173-175, December.
      • L. Robin Keller & Manel Baucells & John C. Butler & Philippe Delquié & Jason R. W. Merrick & Gregory S. Parnell & Ahti Salo, 2008. "From the Editors..," Decision Analysis, INFORMS, vol. 5(4), pages 173-176, December.
      • L. Robin Keller & Manel Baucells & John C. Butler & Philippe Delquié & Jason R. W. Merrick & Gregory S. Parnell & Ahti Salo, 2009. "From the Editors ..," Decision Analysis, INFORMS, vol. 6(4), pages 199-201, December.
    5. L. Robin Keller, 2008. "From the Editor..," Decision Analysis, INFORMS, vol. 5(1), pages 1-4, March.
    6. Sandra Hoffmann & Paul Fischbeck & Alan Krupnick & Michael McWilliams, 2007. "Elicitation from Large, Heterogeneous Expert Panels: Using Multiple Uncertainty Measures to Characterize Information Quality for Decision Analysis," Decision Analysis, INFORMS, vol. 4(2), pages 91-109, June.
    7. Suyi Li & Qiang Meng & Xiaobo Qu, 2012. "An Overview of Maritime Waterway Quantitative Risk Assessment Models," Risk Analysis, John Wiley & Sons, vol. 32(3), pages 496-512, March.
    8. Jason R. W. Merrick & Rene Van Dorp, 2006. "Speaking the Truth in Maritime Risk Assessment," Risk Analysis, John Wiley & Sons, vol. 26(1), pages 223-237, February.
    9. L. Robin Keller, 2007. "From the Editor..," Decision Analysis, INFORMS, vol. 4(2), pages 47-48, June.
    10. Gino J. Lim & Jaeyoung Cho & Selim Bora & Taofeek Biobaku & Hamid Parsaei, 2018. "Models and computational algorithms for maritime risk analysis: a review," Annals of Operations Research, Springer, vol. 271(2), pages 765-786, December.
    11. Hoffmann, Sandra A. & Fischbeck, Paul S. & Krupnick, Alan J. & McWilliams, Michael, 2006. "Eliciting Information on Uncertainty from Heterogeneous Expert Panels: Attributing U.S. Foodborne Pathogen Illness to Food Consumption," Discussion Papers 10444, Resources for the Future.
    12. Saurabh Bansal & Genaro J. Gutierrez, 2020. "Estimating Uncertainties Using Judgmental Forecasts with Expert Heterogeneity," Operations Research, INFORMS, vol. 68(2), pages 363-380, March.
    13. L. Robin Keller & Kelly M. Kophazi, 2010. "From the Editors..," Decision Analysis, INFORMS, vol. 7(2), pages 151-154, June.
    14. J. Dorp & Jason Merrick, 2011. "On a risk management analysis of oil spill risk using maritime transportation system simulation," Annals of Operations Research, Springer, vol. 187(1), pages 249-277, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jason R. W. Merrick, 2008. "Getting the Right Mix of Experts," Decision Analysis, INFORMS, vol. 5(1), pages 43-52, March.
    2. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    3. Ali E. Abbas, 2009. "A Kullback-Leibler View of Linear and Log-Linear Pools," Decision Analysis, INFORMS, vol. 6(1), pages 25-37, March.
    4. Paola Monari & Patrizia Agati, 2001. "Fiducial inference in combining expert judgements," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 10(1), pages 81-97, January.
    5. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    6. Erin Baker & Olaitan Olaleye, 2013. "Combining Experts: Decomposition and Aggregation Order," Risk Analysis, John Wiley & Sons, vol. 33(6), pages 1116-1127, June.
    7. Knut Are Aastveit & James Mitchell & Francesco Ravazzolo & Herman van Dijk, 2018. "The Evolution of Forecast Density Combinations in Economics," Tinbergen Institute Discussion Papers 18-069/III, Tinbergen Institute.
    8. James E. Smith & Detlof von Winterfeldt, 2004. "Anniversary Article: Decision Analysis in Management Science," Management Science, INFORMS, vol. 50(5), pages 561-574, May.
    9. Weyant John, 2014. "Integrated assessment of climate change: state of the literature," Journal of Benefit-Cost Analysis, De Gruyter, vol. 5(3), pages 377-409, December.
    10. Anne Sofie Jore & James Mitchell & Shaun P. Vahey, 2010. "Combining forecast densities from VARs with uncertain instabilities," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 621-634.
    11. Robert L. Winkler & Robert T. Clemen, 2004. "Multiple Experts vs. Multiple Methods: Combining Correlation Assessments," Decision Analysis, INFORMS, vol. 1(3), pages 167-176, September.
    12. Dr. James Mitchell, 2009. "Measuring Output Gap Uncertainty," National Institute of Economic and Social Research (NIESR) Discussion Papers 342, National Institute of Economic and Social Research.
    13. Wilson, Kevin J., 2017. "An investigation of dependence in expert judgement studies with multiple experts," International Journal of Forecasting, Elsevier, vol. 33(1), pages 325-336.
    14. Garratt, Anthony & Mise, Emi, 2014. "Forecasting exchange rates using panel model and model averaging," Economic Modelling, Elsevier, vol. 37(C), pages 32-40.
    15. Li, Yongquan & Zhu, Kaijie, 2009. "Information acquisition in new product introduction," European Journal of Operational Research, Elsevier, vol. 198(2), pages 618-625, October.
    16. Fu, Qi & Zhu, Kaijie, 2010. "Endogenous information acquisition in supply chain management," European Journal of Operational Research, Elsevier, vol. 201(2), pages 454-462, March.
    17. Wang, W., 1997. "Subjective estimation of the delay time distribution in maintenance modelling," European Journal of Operational Research, Elsevier, vol. 99(3), pages 516-529, June.
    18. Neil A. Stiber & Mitchell J. Small & Marina Pantazidou, 2004. "Site‐Specific Updating and Aggregation of Bayesian Belief Network Models for Multiple Experts," Risk Analysis, John Wiley & Sons, vol. 24(6), pages 1529-1538, December.
    19. Jason R. W. Merrick & Rene Van Dorp, 2006. "Speaking the Truth in Maritime Risk Assessment," Risk Analysis, John Wiley & Sons, vol. 26(1), pages 223-237, February.
    20. Wolden Bache, Ida & Sofie Jore, Anne & Mitchell, James & Vahey, Shaun P., 2011. "Combining VAR and DSGE forecast densities," Journal of Economic Dynamics and Control, Elsevier, vol. 35(10), pages 1659-1670, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ordeca:v:2:y:2005:i:1:p:17-29. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.