Combining Experts’ Judgments: Comparison of Algorithmic Methods using Synthetic Data
AbstractExpert judgment (or expert elicitation) is a formal process for eliciting judgments from subject-matter experts about the value of a decision-relevant quantity. Judgments in the form of subjective probability distributions are obtained from several experts, raising the question how best to combine information from multiple experts. A number of algorithmic approaches have been proposed, of which the most commonly employed is the equal-weight combination (the average of the experts’ distributions). We evaluate the properties of five combination methods (equal-weight, best-expert, performance, frequentist, and copula) using simulated expert-judgment data for which we know the process generating the experts’ distributions. We examine cases in which two well-calibrated experts are of equal or unequal quality and their judgments are independent, positively or negatively dependent. In this setting, the copula, frequentist, and best-expert approaches perform better and the equal-weight combination method performs worse than the alternative approaches.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
Bibliographic InfoPaper provided by Toulouse School of Economics (TSE) in its series TSE Working Papers with number 12-293.
Date of creation: Mar 2012
Date of revision:
Publication status: Published in Risk Analysis, vol.�33, n°1, janvier 2013, p.�109-120.
This paper has been announced in the following NEP Reports:
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Robert T. Clemen & Gregory W. Fischer & Robert L. Winkler, 2000. "Assessing Dependence: Some Experimental Results," Management Science, INFORMS, vol. 46(8), pages 1100-1115, August.
- Stephen C. Hora, 2004. "Probability Judgments for Continuous Quantities: Linear Combinations and Calibration," Management Science, INFORMS, vol. 50(5), pages 597-604, May.
- Robert T. Clemen & Terence Reilly, 1999. "Correlations and Copulas for Decision and Risk Analysis," Management Science, INFORMS, vol. 45(2), pages 208-224, February.
- Robert L. Winkler & Roy M. Poses, 1993. "Evaluating and Combining Physicians' Probabilities of Survival in an Intensive Care Unit," Management Science, INFORMS, vol. 39(12), pages 1526-1543, December.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ().
If references are entirely missing, you can add them using this form.