IDEAS home Printed from https://ideas.repec.org/a/wly/riskan/v33y2013i6p1116-1127.html

Combining Experts: Decomposition and Aggregation Order

Author

Listed:
  • Erin Baker
  • Olaitan Olaleye

Abstract

In this article we ask, if quantities in an elicitation have been decomposed, is it better to combine experts before or after recomposing the quantities? We find that combining experts earlier, before recomposition of the quantities, leads to smaller errors with less variance. A simulation shows that these differences may be quite small on average; while an application to actual data shows that the differences can be significant in individual decision problems.

Suggested Citation

  • Erin Baker & Olaitan Olaleye, 2013. "Combining Experts: Decomposition and Aggregation Order," Risk Analysis, John Wiley & Sons, vol. 33(6), pages 1116-1127, June.
    • Handle: RePEc:wly:riskan:v:33:y:2013:i:6:p:1116-1127
    DOI: 10.1111/j.1539-6924.2012.01937.x
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/j.1539-6924.2012.01937.x
    Download Restriction: no
    File URL: https://libkey.io/10.1111/j.1539-6924.2012.01937.x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Baker, Erin & Chon, Haewon & Keisler, Jeffrey, 2009. "Advanced solar R&D: Combining economic analysis with expert elicitations to inform climate policy," Energy Economics, Elsevier, vol. 31(Supplemen), pages 37-49.
    2. Mohamed N. Jouini & Robert T. Clemen, 1996. "Copula Models for Aggregating Expert Opinions," Operations Research, INFORMS, vol. 44(3), pages 444-457, June.
    3. Roger M. Cooke, 1986. "Problems with Empirical Bayes," Risk Analysis, John Wiley & Sons, vol. 6(3), pages 269-272, September.
    4. French, Simon, 1981. "Consensus of opinion," European Journal of Operational Research, Elsevier, vol. 7(4), pages 332-340, August.
    5. Cooke, Roger M. & ElSaadany, Susie & Huang, Xinzheng, 2008. "On the performance of social network and likelihood-based expert weighting schemes," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 745-756.
    6. Dennis Lindley, 1983. "Reconciliation of Probability Distributions," Operations Research, INFORMS, vol. 31(5), pages 866-880, October.
    7. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    8. Lin, Shi-Woei & Bier, Vicki M., 2008. "A study of expert overconfidence," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 711-721.
    9. Cooke, Roger M. & Goossens, Louis L.H.J., 2008. "TU Delft expert judgment data base," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 657-674.
    10. Wisse, Bram & Bedford, Tim & Quigley, John, 2008. "Expert judgement combination using moment methods," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 675-686.
    11. Joel B. Predd & Daniel N. Osherson & Sanjeev R. Kulkarni & H. Vincent Poor, 2008. "Aggregating Probabilistic Forecasts from Incoherent and Abstaining Experts," Decision Analysis, INFORMS, vol. 5(4), pages 177-189, December.
    12. Robert T. Clemen, 1987. "Combining Overlapping Information," Management Science, INFORMS, vol. 33(3), pages 373-380, March.
    13. J. Michael Harrison, 1977. "Independence and Calibration in Decision Analysis," Management Science, INFORMS, vol. 24(3), pages 320-328, November.
    14. Robert L. Winkler, 1981. "Combining Probability Distributions from Dependent Information Sources," Management Science, INFORMS, vol. 27(4), pages 479-488, April.
    15. Tuomisto, Jouni T. & Wilson, Andrew & Evans, John S. & Tainio, Marko, 2008. "Uncertainty in mortality response to airborne fine particulate matter: Combining European air pollution experts," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 732-744.
    16. Jason R. W. Merrick & J. Rene van Dorp & Amita Singh, 2005. "Analysis of Correlated Expert Judgments from Extended Pairwise Comparisons," Decision Analysis, INFORMS, vol. 2(1), pages 17-29, March.
    17. Thomas S. Wallsten & David V. Budescu, 1983. "State of the Art---Encoding Subjective Probabilities: A Psychological and Psychometric Review," Management Science, INFORMS, vol. 29(2), pages 151-173, February.
    18. Robert F. Bordley, 2009. "Combining the Opinions of Experts Who Partition Events Differently," Decision Analysis, INFORMS, vol. 6(1), pages 38-46, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jason R. W. Merrick, 2008. "Getting the Right Mix of Experts," Decision Analysis, INFORMS, vol. 5(1), pages 43-52, March.
    2. Jason R. W. Merrick & J. Rene van Dorp & Amita Singh, 2005. "Analysis of Correlated Expert Judgments from Extended Pairwise Comparisons," Decision Analysis, INFORMS, vol. 2(1), pages 17-29, March.
    3. Ali E. Abbas, 2009. "A Kullback-Leibler View of Linear and Log-Linear Pools," Decision Analysis, INFORMS, vol. 6(1), pages 25-37, March.
    4. James E. Smith & Detlof von Winterfeldt, 2004. "Anniversary Article: Decision Analysis in Management Science," Management Science, INFORMS, vol. 50(5), pages 561-574, May.
    5. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    6. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    7. Jason R. W. Merrick & Fabrizio Ruggeri & Refik Soyer & L. Robin Keller, 2012. "From the Editors---Games and Decisions in Reliability and Risk," Decision Analysis, INFORMS, vol. 9(2), pages 81-85, June.
    8. Cooke, Roger M., 2014. "Deep and Shallow Uncertainty in Messaging Climate Change," RFF Working Paper Series dp-14-11, Resources for the Future.
    9. David V. Budescu & Eva Chen, 2015. "Identifying Expertise to Extract the Wisdom of Crowds," Management Science, INFORMS, vol. 61(2), pages 267-280, February.
    10. Francesco Billari & Rebecca Graziani & Eugenio Melilli, 2014. "Stochastic Population Forecasting Based on Combinations of Expert Evaluations Within the Bayesian Paradigm," Demography, Springer;Population Association of America (PAA), vol. 51(5), pages 1933-1954, October.
    11. Patrick Afflerbach & Christopher Dun & Henner Gimpel & Dominik Parak & Johannes Seyfried, 2021. "A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(4), pages 329-348, August.
    12. Robert F. Bordley, 2009. "Combining the Opinions of Experts Who Partition Events Differently," Decision Analysis, INFORMS, vol. 6(1), pages 38-46, March.
    13. Donnacha Bolger & Brett Houlding, 2016. "Reliability updating in linear opinion pooling for multiple decision makers," Journal of Risk and Reliability, , vol. 230(3), pages 309-322, June.
    14. Ville A. Satopää, 2022. "Regularized Aggregation of One-Off Probability Predictions," Operations Research, INFORMS, vol. 70(6), pages 3558-3580, November.
    15. Yuyu Fan & David V. Budescu & David Mandel & Mark Himmelstein, 2019. "Improving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments," Decision Analysis, INFORMS, vol. 16(3), pages 197-217, September.
    16. Saurabh Bansal & Genaro J. Gutierrez, 2020. "Estimating Uncertainties Using Judgmental Forecasts with Expert Heterogeneity," Operations Research, INFORMS, vol. 68(2), pages 363-380, March.
    17. Christopher W. Karvetski & David R. Mandel & Daniel Irwin, 2020. "Improving Probability Judgment in Intelligence Analysis: From Structured Analysis to Statistical Aggregation," Risk Analysis, John Wiley & Sons, vol. 40(5), pages 1040-1057, May.
    18. Jason R. W. Merrick & Rene Van Dorp, 2006. "Speaking the Truth in Maritime Risk Assessment," Risk Analysis, John Wiley & Sons, vol. 26(1), pages 223-237, February.
    19. Knut Are Aastveit & James Mitchell & Francesco Ravazzolo & Herman van Dijk, 2018. "The Evolution of Forecast Density Combinations in Economics," Tinbergen Institute Discussion Papers 18-069/III, Tinbergen Institute.
    20. Christopher W. Karvetski & Kenneth C. Olson & David R. Mandel & Charles R. Twardy, 2013. "Probabilistic Coherence Weighting for Optimizing Expert Forecasts," Decision Analysis, INFORMS, vol. 10(4), pages 305-326, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:riskan:v:33:y:2013:i:6:p:1116-1127. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1111/(ISSN)1539-6924 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.