IDEAS home Printed from https://ideas.repec.org/a/wly/riskan/v33y2013i6p1116-1127.html
   My bibliography  Save this article

Combining Experts: Decomposition and Aggregation Order

Author

Listed:
  • Erin Baker
  • Olaitan Olaleye

Abstract

In this article we ask, if quantities in an elicitation have been decomposed, is it better to combine experts before or after recomposing the quantities? We find that combining experts earlier, before recomposition of the quantities, leads to smaller errors with less variance. A simulation shows that these differences may be quite small on average; while an application to actual data shows that the differences can be significant in individual decision problems.

Suggested Citation

  • Erin Baker & Olaitan Olaleye, 2013. "Combining Experts: Decomposition and Aggregation Order," Risk Analysis, John Wiley & Sons, vol. 33(6), pages 1116-1127, June.
    • Handle: RePEc:wly:riskan:v:33:y:2013:i:6:p:1116-1127
    DOI: 10.1111/j.1539-6924.2012.01937.x
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/j.1539-6924.2012.01937.x
    Download Restriction: no
    File URL: https://libkey.io/10.1111/j.1539-6924.2012.01937.x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Baker, Erin & Chon, Haewon & Keisler, Jeffrey, 2009. "Advanced solar R&D: Combining economic analysis with expert elicitations to inform climate policy," Energy Economics, Elsevier, vol. 31(Supplemen), pages 37-49.
    2. Mohamed N. Jouini & Robert T. Clemen, 1996. "Copula Models for Aggregating Expert Opinions," Operations Research, INFORMS, vol. 44(3), pages 444-457, June.
    3. Joel B. Predd & Daniel N. Osherson & Sanjeev R. Kulkarni & H. Vincent Poor, 2008. "Aggregating Probabilistic Forecasts from Incoherent and Abstaining Experts," Decision Analysis, INFORMS, vol. 5(4), pages 177-189, December.
    4. Roger M. Cooke, 1986. "Problems with Empirical Bayes," Risk Analysis, John Wiley & Sons, vol. 6(3), pages 269-272, September.
    5. French, Simon, 1981. "Consensus of opinion," European Journal of Operational Research, Elsevier, vol. 7(4), pages 332-340, August.
    6. Cooke, Roger M. & ElSaadany, Susie & Huang, Xinzheng, 2008. "On the performance of social network and likelihood-based expert weighting schemes," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 745-756.
    7. Robert T. Clemen, 1987. "Combining Overlapping Information," Management Science, INFORMS, vol. 33(3), pages 373-380, March.
    8. Dennis Lindley, 1983. "Reconciliation of Probability Distributions," Operations Research, INFORMS, vol. 31(5), pages 866-880, October.
    9. J. Michael Harrison, 1977. "Independence and Calibration in Decision Analysis," Management Science, INFORMS, vol. 24(3), pages 320-328, November.
    10. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    11. Robert L. Winkler, 1981. "Combining Probability Distributions from Dependent Information Sources," Management Science, INFORMS, vol. 27(4), pages 479-488, April.
    12. Tuomisto, Jouni T. & Wilson, Andrew & Evans, John S. & Tainio, Marko, 2008. "Uncertainty in mortality response to airborne fine particulate matter: Combining European air pollution experts," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 732-744.
    13. Jason R. W. Merrick & J. Rene van Dorp & Amita Singh, 2005. "Analysis of Correlated Expert Judgments from Extended Pairwise Comparisons," Decision Analysis, INFORMS, vol. 2(1), pages 17-29, March.
    14. Thomas S. Wallsten & David V. Budescu, 1983. "State of the Art---Encoding Subjective Probabilities: A Psychological and Psychometric Review," Management Science, INFORMS, vol. 29(2), pages 151-173, February.
    15. Lin, Shi-Woei & Bier, Vicki M., 2008. "A study of expert overconfidence," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 711-721.
    16. Cooke, Roger M. & Goossens, Louis L.H.J., 2008. "TU Delft expert judgment data base," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 657-674.
    17. Robert F. Bordley, 2009. "Combining the Opinions of Experts Who Partition Events Differently," Decision Analysis, INFORMS, vol. 6(1), pages 38-46, March.
    18. Wisse, Bram & Bedford, Tim & Quigley, John, 2008. "Expert judgement combination using moment methods," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 675-686.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jason R. W. Merrick, 2008. "Getting the Right Mix of Experts," Decision Analysis, INFORMS, vol. 5(1), pages 43-52, March.
    2. Jason R. W. Merrick & J. Rene van Dorp & Amita Singh, 2005. "Analysis of Correlated Expert Judgments from Extended Pairwise Comparisons," Decision Analysis, INFORMS, vol. 2(1), pages 17-29, March.
    3. Ali E. Abbas, 2009. "A Kullback-Leibler View of Linear and Log-Linear Pools," Decision Analysis, INFORMS, vol. 6(1), pages 25-37, March.
    4. James E. Smith & Detlof von Winterfeldt, 2004. "Anniversary Article: Decision Analysis in Management Science," Management Science, INFORMS, vol. 50(5), pages 561-574, May.
    5. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    6. Patrick Afflerbach & Christopher Dun & Henner Gimpel & Dominik Parak & Johannes Seyfried, 2021. "A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(4), pages 329-348, August.
    7. Robert F. Bordley, 2009. "Combining the Opinions of Experts Who Partition Events Differently," Decision Analysis, INFORMS, vol. 6(1), pages 38-46, March.
    8. Donnacha Bolger & Brett Houlding, 2016. "Reliability updating in linear opinion pooling for multiple decision makers," Journal of Risk and Reliability, , vol. 230(3), pages 309-322, June.
    9. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    10. Yuyu Fan & David V. Budescu & David Mandel & Mark Himmelstein, 2019. "Improving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments," Decision Analysis, INFORMS, vol. 16(3), pages 197-217, September.
    11. Jason R. W. Merrick & Fabrizio Ruggeri & Refik Soyer & L. Robin Keller, 2012. "From the Editors---Games and Decisions in Reliability and Risk," Decision Analysis, INFORMS, vol. 9(2), pages 81-85, June.
    12. Saurabh Bansal & Genaro J. Gutierrez, 2020. "Estimating Uncertainties Using Judgmental Forecasts with Expert Heterogeneity," Operations Research, INFORMS, vol. 68(2), pages 363-380, March.
    13. James K. Hammitt & Yifan Zhang, 2013. "Combining Experts’ Judgments: Comparison of Algorithmic Methods Using Synthetic Data," Risk Analysis, John Wiley & Sons, vol. 33(1), pages 109-120, January.
    14. Christopher W. Karvetski & David R. Mandel & Daniel Irwin, 2020. "Improving Probability Judgment in Intelligence Analysis: From Structured Analysis to Statistical Aggregation," Risk Analysis, John Wiley & Sons, vol. 40(5), pages 1040-1057, May.
    15. Cooke, Roger M., 2014. "Deep and Shallow Uncertainty in Messaging Climate Change," RFF Working Paper Series dp-14-11, Resources for the Future.
    16. Jason R. W. Merrick & Rene Van Dorp, 2006. "Speaking the Truth in Maritime Risk Assessment," Risk Analysis, John Wiley & Sons, vol. 26(1), pages 223-237, February.
    17. Knut Are Aastveit & James Mitchell & Francesco Ravazzolo & Herman van Dijk, 2018. "The Evolution of Forecast Density Combinations in Economics," Tinbergen Institute Discussion Papers 18-069/III, Tinbergen Institute.
    18. David V. Budescu & Eva Chen, 2015. "Identifying Expertise to Extract the Wisdom of Crowds," Management Science, INFORMS, vol. 61(2), pages 267-280, February.
    19. Christopher W. Karvetski & Kenneth C. Olson & David R. Mandel & Charles R. Twardy, 2013. "Probabilistic Coherence Weighting for Optimizing Expert Forecasts," Decision Analysis, INFORMS, vol. 10(4), pages 305-326, December.
    20. Patrizio Frederic & Mario Di Bacco & Frank Lad, 2012. "Combining expert probabilities using the product of odds," Theory and Decision, Springer, vol. 73(4), pages 605-619, October.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:riskan:v:33:y:2013:i:6:p:1116-1127. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1111/(ISSN)1539-6924 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.