IDEAS home Printed from
   My bibliography  Save this article

Expert judgement for dependence in probabilistic modelling: A systematic literature review and future research directions


  • Werner, Christoph
  • Bedford, Tim
  • Cooke, Roger M.
  • Hanea, Anca M.
  • Morales-Nápoles, Oswaldo


Many applications in decision making under uncertainty and probabilistic risk assessment require the assessment of multiple, dependent uncertain quantities, so that in addition to marginal distributions, interdependence needs to be modelled in order to properly understand the overall risk. Nevertheless, relevant historical data on dependence information are often not available or simply too costly to obtain. In this case, the only sensible option is to elicit this uncertainty through the use of expert judgements. In expert judgement studies, a structured approach to eliciting variables of interest is desirable so that their assessment is methodologically robust. One of the key decisions during the elicitation process is the form in which the uncertainties are elicited. This choice is subject to various, potentially conflicting, desiderata related to e.g. modelling convenience, coherence between elicitation parameters and the model, combining judgements, and the assessment burden for the experts. While extensive and systematic guidance to address these considerations exists for single variable uncertainty elicitation, for higher dimensions very little such guidance is available. Therefore, this paper offers a systematic review of the current literature on eliciting dependence. The literature on the elicitation of dependence parameters such as correlations is presented alongside commonly used dependence models and experience from case studies. From this, guidance about the strategy for dependence assessment is given and gaps in the existing research are identified to determine future directions for structured methods to elicit dependence.

Suggested Citation

  • Werner, Christoph & Bedford, Tim & Cooke, Roger M. & Hanea, Anca M. & Morales-Nápoles, Oswaldo, 2017. "Expert judgement for dependence in probabilistic modelling: A systematic literature review and future research directions," European Journal of Operational Research, Elsevier, vol. 258(3), pages 801-819.
  • Handle: RePEc:eee:ejores:v:258:y:2017:i:3:p:801-819
    DOI: 10.1016/j.ejor.2016.10.018

    Download full text from publisher

    File URL:
    Download Restriction: Full text for ScienceDirect subscribers only

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    1. Roger Cooke, 2013. "Uncertainty analysis comes to integrated assessment models for climate change…and conversely," Climatic Change, Springer, vol. 117(3), pages 467-479, April.
    2. Ehrgott, Matthias & Klamroth, Kathrin & Schwehm, Christian, 2004. "An MCDM approach to portfolio optimization," European Journal of Operational Research, Elsevier, vol. 155(3), pages 752-770, June.
    3. Bradley, R. & Dietrich, F.K. & List, C., 2007. "Aggregating causal judgements," Research Memorandum 001, Maastricht University, Maastricht Research School of Economics of Technology and Organization (METEOR).
    4. Percy, David F., 2002. "Bayesian enhanced strategic decision making for reliability," European Journal of Operational Research, Elsevier, vol. 139(1), pages 133-145, May.
    5. Greco, Salvatore & Matarazzo, Benedetto & Slowinski, Roman, 2001. "Rough sets theory for multicriteria decision analysis," European Journal of Operational Research, Elsevier, vol. 129(1), pages 1-47, February.
    6. Cooke, Roger M. & Kelly, G.N., 2010. "Climate Change Uncertainty Quantification: Lessons Learned from the Joint EU-USNRC Project on Uncertainty Analysis of Probabilistic Accident Consequence Codes," Discussion Papers dp-10-29, Resources For the Future.
    7. Ali E. Abbas & David V. Budescu & Yuhong (Rola) Gu, 2010. "Assessing Joint Distributions with Isoprobability Contours," Management Science, INFORMS, vol. 56(6), pages 997-1011, June.
    8. Greco, Salvatore & Mousseau, Vincent & Słowiński, Roman, 2014. "Robust ordinal regression for value functions handling interacting criteria," European Journal of Operational Research, Elsevier, vol. 239(3), pages 711-730.
    9. Stephen Hora & Erim Kardeş, 2015. "Calibration, sharpness and the weighting of experts in a linear opinion pool," Annals of Operations Research, Springer, vol. 229(1), pages 429-450, June.
    10. Victor Richmond R. Jose & Robert F. Nau & Robert L. Winkler, 2009. "Sensitivity to Distance and Baseline Distributions in Forecast Evaluation," Management Science, INFORMS, vol. 55(4), pages 582-590, April.
    11. Morales, O. & Kurowicka, D. & Roelen, A., 2008. "Eliciting conditional and unconditional rank correlations from conditional probabilities," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 699-710.
    12. von Winterfeldt, Detlof & Fasolo, Barbara, 2009. "Structuring decision problems: A case study and reflections for practitioners," European Journal of Operational Research, Elsevier, vol. 199(3), pages 857-866, December.
    13. Shafiqah Al-Awadhi & Paul Garthwaite, 2006. "Quantifying expert opinion for modelling fauna habitat distributions," Computational Statistics, Springer, vol. 21(1), pages 121-140, March.
    14. Moskowitz, Herbert & Bunn, Derek, 1987. "Decision and risk analysis," European Journal of Operational Research, Elsevier, vol. 28(3), pages 247-260, March.
    15. Arbenz, Philipp & Canestraro, Davide, 2012. "Estimating Copulas for Insurance from Scarce Observations, Expert Opinion and Prior Information: A Bayesian Approach," ASTIN Bulletin, Cambridge University Press, vol. 42(1), pages 271-290, May.
    16. Samuel Kotz & Johan René van Dorp, 2010. "Generalized Diagonal Band Copulas with Two-Sided Generating Densities," Decision Analysis, INFORMS, vol. 7(2), pages 196-214, June.
    17. Robert L. Winkler & Robert T. Clemen, 2004. "Multiple Experts vs. Multiple Methods: Combining Correlation Assessments," Decision Analysis, INFORMS, vol. 1(3), pages 167-176, September.
    18. Christian Genest & Michel Gendron & Michaël Bourdeau-Brien, 2009. "The Advent of Copulas in Finance," The European Journal of Finance, Taylor & Francis Journals, vol. 15(7-8), pages 609-618.
    19. Dietrich, Franz & List, Christian, 2014. "Probabilistic Opinion Pooling," MPRA Paper 54806, University Library of Munich, Germany.
    20. Philip M. Lurie & Matthew S. Goldberg, 1998. "An Approximate Method for Sampling Correlated Random Variables from Partially-Specified Distributions," Management Science, INFORMS, vol. 44(2), pages 203-218, February.
    21. Hanea, Anca & Morales Napoles, Oswaldo & Ababei, Dan, 2015. "Non-parametric Bayesian networks: Improving theory and reviewing applications," Reliability Engineering and System Safety, Elsevier, vol. 144(C), pages 265-284.
    22. Ouchi, Fumika, 2004. "A literature review on the use of expert opinion in probabilistic risk analysis," Policy Research Working Paper Series 3201, The World Bank.
    23. Greco, Salvatore & Matarazzo, Benedetto & Slowinski, Roman, 2004. "Axiomatic characterization of a general utility function and its particular cases in terms of conjoint measurement and rough-set decision rules," European Journal of Operational Research, Elsevier, vol. 158(2), pages 271-292, October.
    24. Paul H. Garthwaite & Shafeeqah A. Al‐Awadhi, 2001. "Non‐conjugate prior distribution assessment for multivariate normal sampling," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 63(1), pages 95-110.
    25. Cooke, Roger M. & Goossens, Louis L.H.J., 2008. "TU Delft expert judgment data base," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 657-674.
    26. Paul H. Garthwaite & Shafeeqah A. Al-Awadhi & Fadlalla G. Elfadaly & David J. Jenkinson, 2013. "Prior distribution elicitation for generalized linear and piecewise-linear models," Journal of Applied Statistics, Taylor & Francis Journals, vol. 40(1), pages 59-75, January.
    27. Garthwaite, Paul H. & Kadane, Joseph B. & O'Hagan, Anthony, 2005. "Statistical Methods for Eliciting Probability Distributions," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 680-701, June.
    28. Paul L. Fackler, 1991. "Modeling Interdependence: An Approach to Simulation and Elicitation," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 73(4), pages 1091-1097.
    29. Robert T. Clemen & Terence Reilly, 1999. "Correlations and Copulas for Decision and Risk Analysis," Management Science, INFORMS, vol. 45(2), pages 208-224, February.
    30. Fadlalla Elfadaly & Paul Garthwaite, 2013. "Eliciting Dirichlet and Connor–Mosimann prior distributions for multinomial models," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 22(4), pages 628-646, November.
    31. van Dorp, J. Rene, 2005. "Statistical dependence through common risk factors: With applications in uncertainty analysis," European Journal of Operational Research, Elsevier, vol. 161(1), pages 240-255, February.
    32. Dubois, Didier & Prade, Henri & Sabbadin, Regis, 2001. "Decision-theoretic foundations of qualitative possibility theory," European Journal of Operational Research, Elsevier, vol. 128(3), pages 459-478, February.
    33. Bernd Kraan & Tim Bedford, 2005. "Probabilistic Inversion of Expert Judgments in the Quantification of Model Uncertainty," Management Science, INFORMS, vol. 51(6), pages 995-1006, June.
    34. Mkrtchyan, L. & Podofillini, L. & Dang, V.N., 2015. "Bayesian belief networks for human reliability analysis: A review of applications and gaps," Reliability Engineering and System Safety, Elsevier, vol. 139(C), pages 1-16.
    35. Donald L. Keefer & Samuel E. Bodily, 1983. "Three-Point Approximations for Continuous Random Variables," Management Science, INFORMS, vol. 29(5), pages 595-609, May.
    36. Robert T. Clemen & Gregory W. Fischer & Robert L. Winkler, 2000. "Assessing Dependence: Some Experimental Results," Management Science, INFORMS, vol. 46(8), pages 1100-1115, August.
    37. Rita Esther Zapata-V�zquez & Anthony O'Hagan & Leonardo Soares Bastos, 2014. "Eliciting expert judgements about a set of proportions," Journal of Applied Statistics, Taylor & Francis Journals, vol. 41(9), pages 1919-1933, September.
    38. Marichal, Jean-Luc, 2004. "Tolerant or intolerant character of interacting criteria in aggregation by the Choquet integral," European Journal of Operational Research, Elsevier, vol. 155(3), pages 771-791, June.
    39. Herbert Moskowitz & Rakesh K. Sarin, 1983. "Improving the Consistency of Conditional Probability Assessments for Forecasting and Decision Making," Management Science, INFORMS, vol. 29(6), pages 735-749, June.
    40. Luca Regis, 2011. "A Bayesian copula model for stochastic claims reserving," Carlo Alberto Notebooks 227, Collegio Carlo Alberto.
    41. James E. Smith & Detlof von Winterfeldt, 2004. "Anniversary Article: Decision Analysis in Management Science," Management Science, INFORMS, vol. 50(5), pages 561-574, May.
    42. Jyrki Wallenius & James S. Dyer & Peter C. Fishburn & Ralph E. Steuer & Stanley Zionts & Kalyanmoy Deb, 2008. "Multiple Criteria Decision Making, Multiattribute Utility Theory: Recent Accomplishments and What Lies Ahead," Management Science, INFORMS, vol. 54(7), pages 1336-1349, July.
    43. Angilella, Silvia & Greco, Salvatore & Lamantia, Fabio & Matarazzo, Benedetto, 2004. "Assessing non-additive utility for multicriteria decision aid," European Journal of Operational Research, Elsevier, vol. 158(3), pages 734-744, November.
    44. Figueira, José Rui & Greco, Salvatore & Slowinski, Roman, 2009. "Building a set of additive value functions representing a reference preorder and intensities of preference: GRIP method," European Journal of Operational Research, Elsevier, vol. 195(2), pages 460-486, June.
    45. Carl S. Spetzler & Carl-Axel S. Staël Von Holstein, 1975. "Exceptional Paper--Probability Encoding in Decision Analysis," Management Science, INFORMS, vol. 22(3), pages 340-358, November.
    46. Grabisch, Michel, 1996. "The application of fuzzy integrals in multicriteria decision making," European Journal of Operational Research, Elsevier, vol. 89(3), pages 445-456, March.
    47. Blaszczynski, Jerzy & Greco, Salvatore & Slowinski, Roman, 2007. "Multi-criteria classification - A new scheme for application of dominance-based decision rules," European Journal of Operational Research, Elsevier, vol. 181(3), pages 1030-1044, September.
    Full references (including those not matched with items on IDEAS)


    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.

    Cited by:

    1. Nogal, Maria & Morales Nápoles, Oswaldo & O’Connor, Alan, 2019. "Structured expert judgement to understand the intrinsic vulnerability of traffic networks," Transportation Research Part A: Policy and Practice, Elsevier, vol. 127(C), pages 136-152.
    2. Christoph Werner & Tim Bedford & John Quigley, 2018. "Sequential Refined Partitioning for Probabilistic Dependence Assessment," Risk Analysis, John Wiley & Sons, vol. 38(12), pages 2683-2702, December.
    3. Wang, Fan & Li, Heng & Dong, Chao & Ding, Lieyun, 2019. "Knowledge representation using non-parametric Bayesian networks for tunneling risk analysis," Reliability Engineering and System Safety, Elsevier, vol. 191(C).
    4. Supper, Hendrik & Irresberger, Felix & Weiß, Gregor, 2020. "A comparison of tail dependence estimators," European Journal of Operational Research, Elsevier, vol. 284(2), pages 728-742.


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ejores:v:258:y:2017:i:3:p:801-819. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Haili He). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.