IDEAS home Printed from https://ideas.repec.org/a/inm/ordeca/v10y2013i4p305-326.html
   My bibliography  Save this article

Probabilistic Coherence Weighting for Optimizing Expert Forecasts

Author

Listed:
  • Christopher W. Karvetski

    (Department of Applied Information Technology, George Mason University, Fairfax, Virginia 22030)

  • Kenneth C. Olson

    (Department of Applied Information Technology, George Mason University, Fairfax, Virginia 22030)

  • David R. Mandel

    (Socio-Cognitive Systems Section, DRDC Toronto; and Department of Psychology, York University, Toronto, Ontario M3J 1P3, Canada)

  • Charles R. Twardy

    (Command, Control, Communications, Computing, and Intelligence Center, George Mason University, Fairfax, Virginia 22030)

Abstract

Methods for eliciting and aggregating expert judgment are necessary when decision-relevant data are scarce. Such methods have been used for aggregating the judgments of a large, heterogeneous group of forecasters, as well as the multiple judgments produced from an individual forecaster. This paper addresses how multiple related individual forecasts can be used to improve aggregation of probabilities for a binary event across a set of forecasters. We extend previous efforts that use probabilistic incoherence of an individual forecaster's subjective probability judgments to weight and aggregate the judgments of multiple forecasters for the goal of increasing the accuracy of forecasts. With data from two studies, we describe an approach for eliciting extra probability judgments to (i) adjust the judgments of each individual forecaster, and (ii) assign weights to the judgments to aggregate over the entire set of forecasters. We show improvement of up to 30% over the established benchmark of a simple equal-weighted averaging of forecasts. We also describe how this method can be used to remedy the “fifty--fifty blip” that occurs when forecasters use the probability value of 0.5 to represent epistemic uncertainty.

Suggested Citation

  • Christopher W. Karvetski & Kenneth C. Olson & David R. Mandel & Charles R. Twardy, 2013. "Probabilistic Coherence Weighting for Optimizing Expert Forecasts," Decision Analysis, INFORMS, vol. 10(4), pages 305-326, December.
    • Handle: RePEc:inm:ordeca:v:10:y:2013:i:4:p:305-326
    DOI: 10.1287/deca.2013.0279
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/deca.2013.0279
    Download Restriction: no
    File URL: https://libkey.io/10.1287/deca.2013.0279?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Gneiting, Tilmann, 2011. "Making and Evaluating Point Forecasts," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 746-762.
    2. Joel B. Predd & Daniel N. Osherson & Sanjeev R. Kulkarni & H. Vincent Poor, 2008. "Aggregating Probabilistic Forecasts from Incoherent and Abstaining Experts," Decision Analysis, INFORMS, vol. 5(4), pages 177-189, December.
    3. Richard P. Larrick & Jack B. Soll, 2006. "Erratum--Intuitions About Combining Opinions: Misappreciation of the Averaging Principle," Management Science, INFORMS, vol. 52(2), pages 309-310, February.
    4. Guanchun Wang & Sanjeev R. Kulkarni & H. Vincent Poor & Daniel N. Osherson, 2011. "Aggregating Large Sets of Probabilistic Forecasts by Weighted Coherent Adjustment," Decision Analysis, INFORMS, vol. 8(2), pages 128-144, June.
    5. Hsee, Christopher K., 1996. "The Evaluability Hypothesis: An Explanation for Preference Reversals between Joint and Separate Evaluations of Alternatives," Organizational Behavior and Human Decision Processes, Elsevier, vol. 67(3), pages 247-257, September.
    6. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    7. Wändi Bruine de Bruin & Paul S. Fischbeck & Neil A. Stiber & Baruch Fischhoff, 2002. "What Number is “Fifty‐Fifty”?: Redistributing Excessive 50% Responses in Elicited Probabilities," Risk Analysis, John Wiley & Sons, vol. 22(4), pages 713-723, August.
    8. Robert L. Winkler & Robert T. Clemen, 2004. "Multiple Experts vs. Multiple Methods: Combining Correlation Assessments," Decision Analysis, INFORMS, vol. 1(3), pages 167-176, September.
    9. Sharp, Glen L. & Cutler, Brian L. & Penrod, Steven D., 1988. "Performance feedback improves the resolution of confidence judgments," Organizational Behavior and Human Decision Processes, Elsevier, vol. 42(3), pages 271-283, December.
    10. Cooke, Roger M. & Goossens, Louis L.H.J., 2008. "TU Delft expert judgment data base," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 657-674.
    11. Wright, George & Rowe, Gene & Bolger, Fergus & Gammack, John, 1994. "Coherence, Calibration, and Expertise in Judgmental Probability Forecasting," Organizational Behavior and Human Decision Processes, Elsevier, vol. 57(1), pages 1-25, January.
    12. Osherson, Daniel & Vardi, Moshe Y., 2006. "Aggregating disparate estimates of chance," Games and Economic Behavior, Elsevier, vol. 56(1), pages 148-173, July.
    13. Jason R. W. Merrick, 2008. "Getting the Right Mix of Experts," Decision Analysis, INFORMS, vol. 5(1), pages 43-52, March.
    14. Mark A Burgman & Marissa McBride & Raquel Ashton & Andrew Speirs-Bridge & Louisa Flander & Bonnie Wintle & Fiona Fidler & Libby Rumpff & Charles Twardy, 2011. "Expert Status and Performance," PLOS ONE, Public Library of Science, vol. 6(7), pages 1-7, July.
    15. Sieck, Winston R. & Merkle, Edgar C. & Van Zandt, Trisha, 2007. "Option fixation: A cognitive contributor to overconfidence," Organizational Behavior and Human Decision Processes, Elsevier, vol. 103(1), pages 68-83, May.
    16. Richard P. Larrick & Jack B. Soll, 2006. "Intuitions About Combining Opinions: Misappreciation of the Averaging Principle," Management Science, INFORMS, vol. 52(1), pages 111-127, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. World Health Organization, Foodborne Epidemiology Reference Group, Source Attribution Task Force, 2016. "Research Synthesis Methods in an Age of Globalized Risks: Lessons from the Global Burden of Foodborne Disease Expert Elicitation," Risk Analysis, John Wiley & Sons, vol. 36(2), pages 191-202, February.
    2. repec:cup:judgdm:v:16:y:2021:i:4:p:1072-1096 is not listed on IDEAS
    3. David R. Mandel & Daniel Irwin, 2021. "Tracking accuracy of strategic intelligence forecasts: Findings from a long‐term Canadian study," Futures & Foresight Science, John Wiley & Sons, vol. 3(3-4), September.
    4. David R. Mandel & Christopher W. Karvetski & Mandeep K. Dhami, 2018. "Boosting intelligence analysts’ judgment accuracy: What works, what fails?," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(6), pages 607-621, November.
    5. repec:cup:judgdm:v:13:y:2018:i:6:p:607-621 is not listed on IDEAS
    6. Christopher W. Karvetski & David R. Mandel & Daniel Irwin, 2020. "Improving Probability Judgment in Intelligence Analysis: From Structured Analysis to Statistical Aggregation," Risk Analysis, John Wiley & Sons, vol. 40(5), pages 1040-1057, May.
    7. Jared A. Beekman & Ronald F. A. Woodaman & Dennis M. Buede, 2020. "A Review of Probabilistic Opinion Pooling Algorithms with Application to Insider Threat Detection," Decision Analysis, INFORMS, vol. 17(1), pages 39-55, March.
    8. Yuyu Fan & David V. Budescu & David Mandel & Mark Himmelstein, 2019. "Improving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments," Decision Analysis, INFORMS, vol. 16(3), pages 197-217, September.
    9. David A. Comerford, 2021. "Apparent age and gender differences in survival optimism: To what extent are they a bias in the translation of beliefs onto a percentage scale?," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 16(4), pages 1072-1096, July.
    10. David R. Mandel & Robert N. Collins & Evan F. Risko & Jonathan A. Fugelsang, 2020. "Effect of confidence interval construction on judgment accuracy," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 15(5), pages 783-797, September.
    11. repec:cup:judgdm:v:15:y:2020:i:6:p:939-958 is not listed on IDEAS
    12. repec:cup:judgdm:v:15:y:2020:i:5:p:783-797 is not listed on IDEAS
    13. repec:cup:judgdm:v:8:y:2013:i:5:p:552-560 is not listed on IDEAS
    14. repec:cup:judgdm:v:12:y:2017:i:4:p:369-381 is not listed on IDEAS
    15. David R. Mandel, 2020. "Studies past and future of the past and future: Commentary on Schoemaker 2020," Futures & Foresight Science, John Wiley & Sons, vol. 2(3-4), September.
    16. Christopher W. Karvetski & David R. Mandel, 2020. "Coherence of probability judgments from uncertain evidence: Does ACH help?," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 15(6), pages 939-958, November.
    17. Mandeep K. Dhami & David R. Mandel, 2013. "How do defendants choose their trial court? Evidence for a heuristic processing account," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 8(5), pages 552-560, September.
    18. Barbara A. Mellers & Joshua D. Baker & Eva Chen & David R. Mandel & Philip E. Tetlock, 2017. "How generalizable is good judgment? A multi-task, multi-benchmark study," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 12(4), pages 369-381, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Christopher W. Karvetski & David R. Mandel & Daniel Irwin, 2020. "Improving Probability Judgment in Intelligence Analysis: From Structured Analysis to Statistical Aggregation," Risk Analysis, John Wiley & Sons, vol. 40(5), pages 1040-1057, May.
    2. repec:cup:judgdm:v:13:y:2018:i:6:p:607-621 is not listed on IDEAS
    3. David R. Mandel & Christopher W. Karvetski & Mandeep K. Dhami, 2018. "Boosting intelligence analysts’ judgment accuracy: What works, what fails?," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(6), pages 607-621, November.
    4. Yuyu Fan & David V. Budescu & David Mandel & Mark Himmelstein, 2019. "Improving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments," Decision Analysis, INFORMS, vol. 16(3), pages 197-217, September.
    5. Patrick Afflerbach & Christopher Dun & Henner Gimpel & Dominik Parak & Johannes Seyfried, 2021. "A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(4), pages 329-348, August.
    6. L. Robin Keller & Ali Abbas & Manel Baucells & Vicki M. Bier & David Budescu & John C. Butler & Philippe Delquié & Jason R. W. Merrick & Ahti Salo & George Wu, 2010. "From the Editors..," Decision Analysis, INFORMS, vol. 7(4), pages 327-330, December.
      • L. Robin Keller & Manel Baucells & Kevin F. McCardle & Gregory S. Parnell & Ahti Salo, 2007. "From the Editors..," Decision Analysis, INFORMS, vol. 4(4), pages 173-175, December.
      • L. Robin Keller & Manel Baucells & John C. Butler & Philippe Delquié & Jason R. W. Merrick & Gregory S. Parnell & Ahti Salo, 2008. "From the Editors..," Decision Analysis, INFORMS, vol. 5(4), pages 173-176, December.
      • L. Robin Keller & Manel Baucells & John C. Butler & Philippe Delquié & Jason R. W. Merrick & Gregory S. Parnell & Ahti Salo, 2009. "From the Editors ..," Decision Analysis, INFORMS, vol. 6(4), pages 199-201, December.
    7. Atanasov, Pavel & Witkowski, Jens & Ungar, Lyle & Mellers, Barbara & Tetlock, Philip, 2020. "Small steps to accuracy: Incremental belief updaters are better forecasters," Organizational Behavior and Human Decision Processes, Elsevier, vol. 160(C), pages 19-35.
    8. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    9. Anca M. Hanea & Marissa F. McBride & Mark A. Burgman & Bonnie C. Wintle, 2018. "The Value of Performance Weights and Discussion in Aggregated Expert Judgments," Risk Analysis, John Wiley & Sons, vol. 38(9), pages 1781-1794, September.
    10. Johannes Müller-Trede & Shoham Choshen-Hillel & Meir Barneron & Ilan Yaniv, 2018. "The Wisdom of Crowds in Matters of Taste," Management Science, INFORMS, vol. 64(4), pages 1779-1803, April.
    11. Graefe, Andreas & Armstrong, J. Scott & Jones, Randall J. & Cuzán, Alfred G., 2014. "Combining forecasts: An application to elections," International Journal of Forecasting, Elsevier, vol. 30(1), pages 43-54.
    12. Eric Libby & Leon Glass, 2010. "The Calculus of Committee Composition," PLOS ONE, Public Library of Science, vol. 5(9), pages 1-8, September.
    13. Tomás Lejarraga & Johannes Müller-Trede, 2017. "When Experience Meets Description: How Dyads Integrate Experiential and Descriptive Information in Risky Decisions," Management Science, INFORMS, vol. 63(6), pages 1953-1971, June.
    14. L. Robin Keller, 2011. "From the Editor ---Multiattribute and Intertemporal Preferences, Probability, and Stochastic Processes: Models and Assessment," Decision Analysis, INFORMS, vol. 8(3), pages 165-169, September.
    15. repec:cup:judgdm:v:12:y:2017:i:4:p:369-381 is not listed on IDEAS
    16. Ville A. Satopää & Marat Salikhov & Philip E. Tetlock & Barbara Mellers, 2021. "Bias, Information, Noise: The BIN Model of Forecasting," Management Science, INFORMS, vol. 67(12), pages 7599-7618, December.
    17. L. Robin Keller & Kelly M. Kophazi, 2012. "From the Editors ---Copulas, Group Preferences, Multilevel Defenders, Sharing Rewards, and Communicating Analytics," Decision Analysis, INFORMS, vol. 9(3), pages 213-218, September.
    18. L. Robin Keller & Kelly M. Kophazi, 2011. "From the Editors---Deterrence, Multiattribute Utility, and Probability and Bayes' Updating," Decision Analysis, INFORMS, vol. 8(2), pages 83-87, June.
    19. Lyon, Aidan & Wintle, Bonnie C. & Burgman, Mark, 2015. "Collective wisdom: Methods of confidence interval aggregation," Journal of Business Research, Elsevier, vol. 68(8), pages 1759-1767.
    20. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    21. Rakesh K. Sarin, 2013. "From the Editor ---Median Aggregation, Scoring Rules, Expert Forecasts, Choices with Binary Attributes, Portfolio with Dependent Projects, and Information Security," Decision Analysis, INFORMS, vol. 10(4), pages 277-278, December.
    22. Hathout, Michel & Vuillet, Marc & Carvajal, Claudio & Peyras, Laurent & Diab, Youssef, 2019. "Expert judgments calibration and combination for assessment of river levee failure probability," Reliability Engineering and System Safety, Elsevier, vol. 188(C), pages 377-392.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ordeca:v:10:y:2013:i:4:p:305-326. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.