IDEAS home Printed from https://ideas.repec.org/a/eee/ejores/v304y2023i2p661-675.html
   My bibliography  Save this article

Testing the effectiveness of debiasing techniques to reduce overprecision in the elicitation of subjective continuous probability distributions

Author

Listed:
  • Ferretti, Valentina
  • Montibeller, Gilberto
  • von Winterfeldt, Detlof

Abstract

Formal expert elicitation is a widely used method for quantifying uncertain variables in decision and risk analysis. When estimating uncertain variables, experts and laypeople exhibit overprecision, meaning that the ranges of their estimates are too narrow. Overprecision, a form of overconfidence, is pervasive and hard to correct, thus posing a challenge to expert elicitation. Following the increasing interest toward improving judgments in Behavioral Operational Research (OR), and the limited evidence about the effectiveness of debiasing tools, the aim of our research is to test the effectiveness of commonly employed practices for debiasing overprecision. We conducted two experiments, testing a set of debiasing techniques when eliciting points of a cumulative distribution functions for general knowledge questions. The debiasing procedures included hypothetical bets, counterfactual argumentation, and automatic stretching to increase the ranges of subjects’ initial estimates. We find that two debiasing strategies that require further reasoning after initial estimates (hypothetical bets and counterfactuals) were not very effective for reducing overprecision, while the use of multipliers that increase the initial range of distributions, coupled with a re-elicitation of the distribution with the new range, provided more positive results. We provide some recommendations for expert elicitation in OR practice, based on our findings, and suggest avenues for further research into debiasing overprecision.

Suggested Citation

  • Ferretti, Valentina & Montibeller, Gilberto & von Winterfeldt, Detlof, 2023. "Testing the effectiveness of debiasing techniques to reduce overprecision in the elicitation of subjective continuous probability distributions," European Journal of Operational Research, Elsevier, vol. 304(2), pages 661-675.
    • Handle: RePEc:eee:ejores:v:304:y:2023:i:2:p:661-675
    DOI: 10.1016/j.ejor.2022.04.008
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0377221722003046
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ejor.2022.04.008?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Terry Connolly & Doug Dean, 1997. "Decomposed Versus Holistic Estimates of Effort Required for Software Writing Tasks," Management Science, INFORMS, vol. 43(7), pages 1029-1045, July.
    2. James E. Matheson & Robert L. Winkler, 1976. "Scoring Rules for Continuous Probability Distributions," Management Science, INFORMS, vol. 22(10), pages 1087-1096, June.
    3. repec:cup:judgdm:v:11:y:2016:i:5:p:509-526 is not listed on IDEAS
    4. Moore, Don A. & Carter, Ashli B. & Yang, Heather H.J., 2015. "Wide of the mark: Evidence on the underlying causes of overprecision in judgment," Organizational Behavior and Human Decision Processes, Elsevier, vol. 131(C), pages 110-120.
    5. Matteo M. Galizzi, 2014. "What Is Really Behavioral in Behavioral Health Policy? And Does It Work?," Applied Economic Perspectives and Policy, Agricultural and Applied Economics Association, vol. 36(1), pages 25-60.
    6. Kriti Jain & Kanchan Mukherjee & J. Neil Bearden & Anil Gaba, 2013. "Unpacking the Future: A Nudge Toward Wider Subjective Confidence Intervals," Management Science, INFORMS, vol. 59(9), pages 1970-1987, September.
    7. Werner, Christoph & Bedford, Tim & Cooke, Roger M. & Hanea, Anca M. & Morales-Nápoles, Oswaldo, 2017. "Expert judgement for dependence in probabilistic modelling: A systematic literature review and future research directions," European Journal of Operational Research, Elsevier, vol. 258(3), pages 801-819.
    8. Aloysius, John A. & Davis, Fred D. & Wilson, Darryl D. & Ross Taylor, A. & Kottemann, Jeffrey E., 2006. "User acceptance of multi-criteria decision support systems: The impact of preference elicitation techniques," European Journal of Operational Research, Elsevier, vol. 169(1), pages 273-285, February.
    9. Langnickel, Ferdinand & Zeisberger, Stefan, 2016. "Do we measure overconfidence? A closer look at the interval production task," Journal of Economic Behavior & Organization, Elsevier, vol. 128(C), pages 121-133.
    10. repec:cup:judgdm:v:5:y:2010:i:7:p:467-476 is not listed on IDEAS
    11. Ferretti, Valentina & Guney, Sule & Montibeller, Gilberto & Winterfeldt, Detlof von, 2016. "Testing best practices to reduce the overconfidence bias in multi-criteria decision analysis," LSE Research Online Documents on Economics 67179, London School of Economics and Political Science, LSE Library.
    12. Matthew B. Welsh & Steve H. Begg, 2018. "More-or-less elicitation (MOLE): reducing bias in range estimation and forecasting," EURO Journal on Decision Processes, Springer;EURO - The Association of European Operational Research Societies, vol. 6(1), pages 171-212, June.
    13. Camilleri, Adrian R. & Newell, Ben R., 2019. "Better calibration when predicting from experience (rather than description)," Organizational Behavior and Human Decision Processes, Elsevier, vol. 150(C), pages 62-82.
    14. Block, Richard A. & Harper, David R., 1991. "Overconfidence in estimation: Testing the anchoring-and-adjustment hypothesis," Organizational Behavior and Human Decision Processes, Elsevier, vol. 49(2), pages 188-207, August.
    15. Daniel J. Walters & Philip M. Fernbach & Craig R. Fox & Steven A. Sloman, 2017. "Known Unknowns: A Critical Determinant of Confidence and Calibration," Management Science, INFORMS, vol. 63(12), pages 4298-4307, December.
    16. Yufei Ren & Rachel Croson, 2013. "Overconfidence in Newsvendor Orders: An Experimental Study," Management Science, INFORMS, vol. 59(11), pages 2502-2517, November.
    17. Kirshner, Samuel Nathan & Shao, Lusheng, 2019. "The overconfident and optimistic price-setting newsvendor," European Journal of Operational Research, Elsevier, vol. 277(1), pages 166-173.
    18. Lahtinen, Tuomas J. & Hämäläinen, Raimo P. & Jenytin, Cosmo, 2020. "On preference elicitation processes which mitigate the accumulation of biases in multi-criteria decision analysis," European Journal of Operational Research, Elsevier, vol. 282(1), pages 201-210.
    19. Keren, Gideon, 1987. "Facing uncertainty in the game of bridge: A calibration study," Organizational Behavior and Human Decision Processes, Elsevier, vol. 39(1), pages 98-114, February.
    20. Craig R. Fox & Amos Tversky, 1998. "A Belief-Based Account of Decision Under Uncertainty," Management Science, INFORMS, vol. 44(7), pages 879-895, July.
    21. David V. Budescu & Ning Du, 2007. "Coherence and Consistency of Investors' Probability Judgments," Management Science, INFORMS, vol. 53(11), pages 1731-1744, November.
    22. Franco, L. Alberto & Hämäläinen, Raimo P. & Rouwette, Etiënne A.J.A. & Leppänen, Ilkka, 2021. "Taking stock of behavioural OR: A review of behavioural studies with an intervention focus," European Journal of Operational Research, Elsevier, vol. 293(2), pages 401-418.
    23. Gilberto Montibeller & Detlof von Winterfeldt, 2015. "Cognitive and Motivational Biases in Decision and Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 35(7), pages 1230-1251, July.
    24. McKenzie, Craig R.M. & Liersch, Michael J. & Yaniv, Ilan, 2008. "Overconfidence in interval estimates: What does expertise buy you?," Organizational Behavior and Human Decision Processes, Elsevier, vol. 107(2), pages 179-191, November.
    25. Ali E. Abbas & David V. Budescu & Hsiu-Ting Yu & Ryan Haggerty, 2008. "A Comparison of Two Probability Encoding Methods: Fixed Probability vs. Fixed Variable Values," Decision Analysis, INFORMS, vol. 5(4), pages 190-202, December.
    26. Onkal, Dilek & Yates, J. Frank & Simga-Mugan, Can & Oztin, Sule, 2003. "Professional vs. amateur judgment accuracy: The case of foreign exchange rates," Organizational Behavior and Human Decision Processes, Elsevier, vol. 91(2), pages 169-185, July.
    27. Carl S. Spetzler & Carl-Axel S. Staël Von Holstein, 1975. "Exceptional Paper--Probability Encoding in Decision Analysis," Management Science, INFORMS, vol. 22(3), pages 340-358, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ferretti, Valentina & Montibeller, Gilberto & von Winterfeldt, Detlof, 2023. "Testing the effectiveness of debiasing techniques to reduce overprecision in the elicitation of subjective continuous probability distributions," LSE Research Online Documents on Economics 115333, London School of Economics and Political Science, LSE Library.
    2. repec:cup:judgdm:v:12:y:2017:i:1:p:29-41 is not listed on IDEAS
    3. Julia P. Prims & Don A. Moore, 2017. "Overconfidence over the lifespan," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 12(1), pages 29-41, January.
    4. Saemi Park & David V. Budescu, 2015. "Aggregating multiple probability intervals to improve calibration," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 10(2), pages 130-143, March.
    5. Yun Shin Lee & Enno Siemsen, 2017. "Task Decomposition and Newsvendor Decision Making," Management Science, INFORMS, vol. 63(10), pages 3226-3245, October.
    6. repec:cup:judgdm:v:10:y:2015:i:2:p:130-143 is not listed on IDEAS
    7. Sulian Wang & Chen Wang, 2021. "Quantile Judgments of Lognormal Losses: An Experimental Investigation," Decision Analysis, INFORMS, vol. 18(1), pages 78-99, March.
    8. repec:cup:judgdm:v:15:y:2020:i:5:p:783-797 is not listed on IDEAS
    9. David R. Mandel & Robert N. Collins & Evan F. Risko & Jonathan A. Fugelsang, 2020. "Effect of confidence interval construction on judgment accuracy," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 15(5), pages 783-797, September.
    10. Anil Gaba & Ilia Tsetlin & Robert L. Winkler, 2017. "Combining Interval Forecasts," Decision Analysis, INFORMS, vol. 14(1), pages 1-20, March.
    11. Ying Han & David V. Budescu, 2022. "Recalibrating probabilistic forecasts to improve their accuracy," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 17(1), pages 91-123, January.
    12. Hao, Zhongyuan & Li, Juan & Cai, Jinling, 2023. "Allocation of inventory responsibilities in overconfident supply chains," European Journal of Operational Research, Elsevier, vol. 305(1), pages 207-221.
    13. Matthew B. Welsh & Steve H. Begg, 2018. "More-or-less elicitation (MOLE): reducing bias in range estimation and forecasting," EURO Journal on Decision Processes, Springer;EURO - The Association of European Operational Research Societies, vol. 6(1), pages 171-212, June.
    14. repec:cup:judgdm:v:14:y:2019:i:4:p:395-411 is not listed on IDEAS
    15. repec:cup:judgdm:v:17:y:2022:i:1:p:91-123 is not listed on IDEAS
    16. James S. Dyer & James E. Smith, 2021. "Innovations in the Science and Practice of Decision Analysis: The Role of Management Science," Management Science, INFORMS, vol. 67(9), pages 5364-5378, September.
    17. Ying Han & David Budescu, 2019. "A universal method for evaluating the quality of aggregators," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 14(4), pages 395-411, July.
    18. Gilberto Montibeller & Detlof von Winterfeldt, 2015. "Cognitive and Motivational Biases in Decision and Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 35(7), pages 1230-1251, July.
    19. Bonaccorsi, Andrea & Apreda, Riccardo & Fantoni, Gualtiero, 2020. "Expert biases in technology foresight. Why they are a problem and how to mitigate them," Technological Forecasting and Social Change, Elsevier, vol. 151(C).
    20. Victor Richmond R. Jose & Robert L. Winkler, 2009. "Evaluating Quantile Assessments," Operations Research, INFORMS, vol. 57(5), pages 1287-1297, October.
    21. Kenneth C. Lichtendahl & Yael Grushka-Cockayne & Robert L. Winkler, 2013. "Is It Better to Average Probabilities or Quantiles?," Management Science, INFORMS, vol. 59(7), pages 1594-1611, July.
    22. Lawrence, Michael & Goodwin, Paul & O'Connor, Marcus & Onkal, Dilek, 2006. "Judgmental forecasting: A review of progress over the last 25 years," International Journal of Forecasting, Elsevier, vol. 22(3), pages 493-518.
    23. Thomas W. Keelin & Bradford W. Powley, 2011. "Quantile-Parameterized Distributions," Decision Analysis, INFORMS, vol. 8(3), pages 206-219, September.
    24. Brenner, Lyle & Griffin, Dale & Koehler, Derek J., 2005. "Modeling patterns of probability calibration with random support theory: Diagnosing case-based judgment," Organizational Behavior and Human Decision Processes, Elsevier, vol. 97(1), pages 64-81, May.
    25. McKenzie, Craig R.M. & Liersch, Michael J. & Yaniv, Ilan, 2008. "Overconfidence in interval estimates: What does expertise buy you?," Organizational Behavior and Human Decision Processes, Elsevier, vol. 107(2), pages 179-191, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ejores:v:304:y:2023:i:2:p:661-675. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/eor .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.