IDEAS home Printed from https://ideas.repec.org/a/inm/ordeca/v5y2008i4p177-189.html
   My bibliography  Save this article

Aggregating Probabilistic Forecasts from Incoherent and Abstaining Experts

Author

Listed:
  • Joel B. Predd

    (RAND Corporation, Pittsburgh, Pennsylvania 15213)

  • Daniel N. Osherson

    (Department of Psychology, Princeton University, Princeton, New Jersey 08544)

  • Sanjeev R. Kulkarni

    (Department of Electrical Engineering, Princeton University, Princeton, New Jersey 08544)

  • H. Vincent Poor

    (Department of Electrical Engineering, Princeton University, Princeton, New Jersey 08544)

Abstract

Decision makers often rely on expert opinion when making forecasts under uncertainty. In doing so, they confront two methodological challenges: the elicitation problem, which requires them to extract meaningful information from experts; and the aggregation problem, which requires them to combine expert opinion by resolving disagreements. Linear averaging is a justifiably popular method for addressing aggregation, but its robust simplicity makes two requirements on elicitation. First, each expert must offer probabilistically coherent forecasts; second, each expert must respond to all our queries. In practice, human judges (even experts) may be incoherent, and may prefer to assess only the subset of events about which they are comfortable offering an opinion. In this paper, a new methodology is developed for combining expert assessment of chance. The method retains the conceptual and computational simplicity of linear averaging, but generalizes the standard approach by relaxing the requirements on expert elicitation. The method also enjoys provable performance guarantees, and in experiments with real-world forecasting data is shown to offer both computational efficiency and competitive forecasting gains as compared to rival aggregation methods. This paper is relevant to the practice of decision analysis, for it enables an elicitation methodology in which judges have freedom to choose the events they assess.

Suggested Citation

  • Joel B. Predd & Daniel N. Osherson & Sanjeev R. Kulkarni & H. Vincent Poor, 2008. "Aggregating Probabilistic Forecasts from Incoherent and Abstaining Experts," Decision Analysis, INFORMS, vol. 5(4), pages 177-189, December.
    • Handle: RePEc:inm:ordeca:v:5:y:2008:i:4:p:177-189
    DOI: 10.1287/deca.1080.0119
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/deca.1080.0119
    Download Restriction: no
    File URL: https://libkey.io/10.1287/deca.1080.0119?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    2. Clemen, Robert T., 1989. "Combining forecasts: A review and annotated bibliography," International Journal of Forecasting, Elsevier, vol. 5(4), pages 559-583.
    3. Osherson, Daniel & Vardi, Moshe Y., 2006. "Aggregating disparate estimates of chance," Games and Economic Behavior, Elsevier, vol. 56(1), pages 148-173, July.
    4. Sniezek, Janet A. & Henry, Rebecca A., 1989. "Accuracy and confidence in group judgment," Organizational Behavior and Human Decision Processes, Elsevier, vol. 43(1), pages 1-28, February.
    5. Norman Dalkey & Olaf Helmer, 1963. "An Experimental Application of the DELPHI Method to the Use of Experts," Management Science, INFORMS, vol. 9(3), pages 458-467, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jason R. W. Merrick & Fabrizio Ruggeri & Refik Soyer & L. Robin Keller, 2012. "From the Editors---Games and Decisions in Reliability and Risk," Decision Analysis, INFORMS, vol. 9(2), pages 81-85, June.
    2. Matthias Bogaert & Michel Ballings & Martijn Hosten & Dirk Van den Poel, 2017. "Identifying Soccer Players on Facebook Through Predictive Analytics," Decision Analysis, INFORMS, vol. 14(4), pages 274-297, December.
    3. David R. Mandel & Christopher W. Karvetski & Mandeep K. Dhami, 2018. "Boosting intelligence analysts’ judgment accuracy: What works, what fails?," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(6), pages 607-621, November.
    4. repec:cup:judgdm:v:13:y:2018:i:6:p:607-621 is not listed on IDEAS
    5. L. Robin Keller, 2009. "From the Editor..," Decision Analysis, INFORMS, vol. 6(1), pages 1-3, March.
    6. Christopher W. Karvetski & David R. Mandel & Daniel Irwin, 2020. "Improving Probability Judgment in Intelligence Analysis: From Structured Analysis to Statistical Aggregation," Risk Analysis, John Wiley & Sons, vol. 40(5), pages 1040-1057, May.
    7. L. Robin Keller & Ali Abbas & Manel Baucells & Vicki M. Bier & David Budescu & John C. Butler & Philippe Delquié & Jason R. W. Merrick & Ahti Salo & George Wu, 2010. "From the Editors..," Decision Analysis, INFORMS, vol. 7(4), pages 327-330, December.
      • L. Robin Keller & Manel Baucells & Kevin F. McCardle & Gregory S. Parnell & Ahti Salo, 2007. "From the Editors..," Decision Analysis, INFORMS, vol. 4(4), pages 173-175, December.
      • L. Robin Keller & Manel Baucells & John C. Butler & Philippe Delquié & Jason R. W. Merrick & Gregory S. Parnell & Ahti Salo, 2008. "From the Editors..," Decision Analysis, INFORMS, vol. 5(4), pages 173-176, December.
      • L. Robin Keller & Manel Baucells & John C. Butler & Philippe Delquié & Jason R. W. Merrick & Gregory S. Parnell & Ahti Salo, 2009. "From the Editors ..," Decision Analysis, INFORMS, vol. 6(4), pages 199-201, December.
    8. Yuyu Fan & David V. Budescu & David Mandel & Mark Himmelstein, 2019. "Improving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments," Decision Analysis, INFORMS, vol. 16(3), pages 197-217, September.
    9. Robert F. Bordley, 2009. "Combining the Opinions of Experts Who Partition Events Differently," Decision Analysis, INFORMS, vol. 6(1), pages 38-46, March.
    10. David R. Mandel & Robert N. Collins & Evan F. Risko & Jonathan A. Fugelsang, 2020. "Effect of confidence interval construction on judgment accuracy," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 15(5), pages 783-797, September.
    11. L. Robin Keller, 2011. "From the Editor ---Multiattribute and Intertemporal Preferences, Probability, and Stochastic Processes: Models and Assessment," Decision Analysis, INFORMS, vol. 8(3), pages 165-169, September.
    12. L. Robin Keller & Kelly M. Kophazi, 2012. "From the Editors ---Copulas, Group Preferences, Multilevel Defenders, Sharing Rewards, and Communicating Analytics," Decision Analysis, INFORMS, vol. 9(3), pages 213-218, September.
    13. L. Robin Keller & Kelly M. Kophazi, 2011. "From the Editors---Deterrence, Multiattribute Utility, and Probability and Bayes' Updating," Decision Analysis, INFORMS, vol. 8(2), pages 83-87, June.
    14. repec:cup:judgdm:v:15:y:2020:i:5:p:783-797 is not listed on IDEAS
    15. repec:cup:judgdm:v:12:y:2017:i:4:p:369-381 is not listed on IDEAS
    16. Rakesh K. Sarin, 2013. "From the Editor ---Median Aggregation, Scoring Rules, Expert Forecasts, Choices with Binary Attributes, Portfolio with Dependent Projects, and Information Security," Decision Analysis, INFORMS, vol. 10(4), pages 277-278, December.
    17. Charles Vlek, 2013. "What Can National Risk Assessors Learn from Decision Theorists and Psychologists?," Risk Analysis, John Wiley & Sons, vol. 33(8), pages 1389-1393, August.
    18. Barbara A. Mellers & Joshua D. Baker & Eva Chen & David R. Mandel & Philip E. Tetlock, 2017. "How generalizable is good judgment? A multi-task, multi-benchmark study," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 12(4), pages 369-381, July.
    19. Guanchun Wang & Sanjeev R. Kulkarni & H. Vincent Poor & Daniel N. Osherson, 2011. "Aggregating Large Sets of Probabilistic Forecasts by Weighted Coherent Adjustment," Decision Analysis, INFORMS, vol. 8(2), pages 128-144, June.
    20. Christopher W. Karvetski & Kenneth C. Olson & David R. Mandel & Charles R. Twardy, 2013. "Probabilistic Coherence Weighting for Optimizing Expert Forecasts," Decision Analysis, INFORMS, vol. 10(4), pages 305-326, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Guanchun Wang & Sanjeev R. Kulkarni & H. Vincent Poor & Daniel N. Osherson, 2011. "Aggregating Large Sets of Probabilistic Forecasts by Weighted Coherent Adjustment," Decision Analysis, INFORMS, vol. 8(2), pages 128-144, June.
    2. Franz Dietrich & Christian List, 2017. "Probabilistic opinion pooling generalized. Part one: general agendas," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 48(4), pages 747-786, April.
    3. repec:cup:judgdm:v:13:y:2018:i:6:p:607-621 is not listed on IDEAS
    4. Patrick Afflerbach & Christopher Dun & Henner Gimpel & Dominik Parak & Johannes Seyfried, 2021. "A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(4), pages 329-348, August.
    5. Julia A. Minson & Jennifer S. Mueller & Richard P. Larrick, 2018. "The Contingent Wisdom of Dyads: When Discussion Enhances vs. Undermines the Accuracy of Collaborative Judgments," Management Science, INFORMS, vol. 64(9), pages 4177-4192, September.
    6. Robert L. Winkler & Robert T. Clemen, 2004. "Multiple Experts vs. Multiple Methods: Combining Correlation Assessments," Decision Analysis, INFORMS, vol. 1(3), pages 167-176, September.
    7. Kenichiro McAlinn & Knut Are Aastveit & Jouchi Nakajima & Mike West, 2020. "Multivariate Bayesian Predictive Synthesis in Macroeconomic Forecasting," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 115(531), pages 1092-1110, July.
    8. Pennings, Clint L.P. & van Dalen, Jan & Rook, Laurens, 2019. "Coordinating judgmental forecasting: Coping with intentional biases," Omega, Elsevier, vol. 87(C), pages 46-56.
    9. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    10. Yuyu Fan & David V. Budescu & David Mandel & Mark Himmelstein, 2019. "Improving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments," Decision Analysis, INFORMS, vol. 16(3), pages 197-217, September.
    11. Johannes Müller-Trede & Shoham Choshen-Hillel & Meir Barneron & Ilan Yaniv, 2018. "The Wisdom of Crowds in Matters of Taste," Management Science, INFORMS, vol. 64(4), pages 1779-1803, April.
    12. Michael P. Clements & David I. Harvey, 2010. "Forecast encompassing tests and probability forecasts," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(6), pages 1028-1062.
    13. Tomás Lejarraga & Johannes Müller-Trede, 2017. "When Experience Meets Description: How Dyads Integrate Experiential and Descriptive Information in Risky Decisions," Management Science, INFORMS, vol. 63(6), pages 1953-1971, June.
    14. Budescu, David V. & Rantilla, Adrian K. & Yu, Hsiu-Ting & Karelitz, Tzur M., 2003. "The effects of asymmetry among advisors on the aggregation of their opinions," Organizational Behavior and Human Decision Processes, Elsevier, vol. 90(1), pages 178-194, January.
    15. Yaniv, Ilan, 2004. "Receiving other people's advice: Influence and benefit," Organizational Behavior and Human Decision Processes, Elsevier, vol. 93(1), pages 1-13, January.
    16. Christopher W. Karvetski & David R. Mandel & Daniel Irwin, 2020. "Improving Probability Judgment in Intelligence Analysis: From Structured Analysis to Statistical Aggregation," Risk Analysis, John Wiley & Sons, vol. 40(5), pages 1040-1057, May.
    17. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    18. K=osaku Takanashi & Kenichiro McAlinn, 2019. "Equivariant online predictions of non-stationary time series," Papers 1911.08662, arXiv.org, revised Jun 2023.
    19. Despoina Makariou & Pauline Barrieu & George Tzougas, 2021. "A Finite Mixture Modelling Perspective for Combining Experts’ Opinions with an Application to Quantile-Based Risk Measures," Risks, MDPI, vol. 9(6), pages 1-25, June.
    20. David V. Budescu & Eva Chen, 2015. "Identifying Expertise to Extract the Wisdom of Crowds," Management Science, INFORMS, vol. 61(2), pages 267-280, February.
    21. Satopää, Ville A. & Salikhov, Marat & Tetlock, Philip E. & Mellers, Barbara, 2023. "Decomposing the effects of crowd-wisdom aggregators: The bias–information–noise (BIN) model," International Journal of Forecasting, Elsevier, vol. 39(1), pages 470-485.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ordeca:v:5:y:2008:i:4:p:177-189. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.