IDEAS home Printed from https://ideas.repec.org/p/ehl/lserod/115333.html
   My bibliography  Save this paper

Testing the effectiveness of debiasing techniques to reduce overprecision in the elicitation of subjective continuous probability distributions

Author

Listed:
  • Ferretti, Valentina
  • Montibeller, Gilberto
  • von Winterfeldt, Detlof

Abstract

Formal expert elicitation is a widely used method for quantifying uncertain variables in decision and risk analysis. When estimating uncertain variables, experts and laypeople exhibit overprecision, meaning that the ranges of their estimates are too narrow. Overprecision, a form of overconfidence, is pervasive and hard to correct, thus posing a challenge to expert elicitation. Following the increasing interest toward improving judgments in Behavioral Operational Research (OR), and the limited evidence about the effectiveness of debiasing tools, the aim of our research is to test the effectiveness of commonly employed practices for debiasing overprecision. We conducted two experiments, testing a set of debiasing techniques when eliciting points of a cumulative distribution functions for general knowledge questions. The debiasing procedures included hypothetical bets, counterfactual argumentation, and automatic stretching to increase the ranges of subjects’ initial estimates. We find that two debiasing strategies that require further reasoning after initial estimates (hypothetical bets and counterfactuals) were not very effective for reducing overprecision, while the use of multipliers that increase the initial range of distributions, coupled with a re-elicitation of the distribution with the new range, provided more positive results. We provide some recommendations for expert elicitation in OR practice, based on our findings, and suggest avenues for further research into debiasing overprecision.

Suggested Citation

  • Ferretti, Valentina & Montibeller, Gilberto & von Winterfeldt, Detlof, 2023. "Testing the effectiveness of debiasing techniques to reduce overprecision in the elicitation of subjective continuous probability distributions," LSE Research Online Documents on Economics 115333, London School of Economics and Political Science, LSE Library.
    • Handle: RePEc:ehl:lserod:115333
    as

    Download full text from publisher

    File URL: http://eprints.lse.ac.uk/115333/
    File Function: Open access version.
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Terry Connolly & Doug Dean, 1997. "Decomposed Versus Holistic Estimates of Effort Required for Software Writing Tasks," Management Science, INFORMS, vol. 43(7), pages 1029-1045, July.
    2. James E. Matheson & Robert L. Winkler, 1976. "Scoring Rules for Continuous Probability Distributions," Management Science, INFORMS, vol. 22(10), pages 1087-1096, June.
    3. repec:cup:judgdm:v:11:y:2016:i:5:p:509-526 is not listed on IDEAS
    4. Moore, Don A. & Carter, Ashli B. & Yang, Heather H.J., 2015. "Wide of the mark: Evidence on the underlying causes of overprecision in judgment," Organizational Behavior and Human Decision Processes, Elsevier, vol. 131(C), pages 110-120.
    5. Werner, Christoph & Bedford, Tim & Cooke, Roger M. & Hanea, Anca M. & Morales-Nápoles, Oswaldo, 2017. "Expert judgement for dependence in probabilistic modelling: A systematic literature review and future research directions," European Journal of Operational Research, Elsevier, vol. 258(3), pages 801-819.
    6. Camilleri, Adrian R. & Newell, Ben R., 2019. "Better calibration when predicting from experience (rather than description)," Organizational Behavior and Human Decision Processes, Elsevier, vol. 150(C), pages 62-82.
    7. David V. Budescu & Ning Du, 2007. "Coherence and Consistency of Investors' Probability Judgments," Management Science, INFORMS, vol. 53(11), pages 1731-1744, November.
    8. Carl S. Spetzler & Carl-Axel S. Staël Von Holstein, 1975. "Exceptional Paper--Probability Encoding in Decision Analysis," Management Science, INFORMS, vol. 22(3), pages 340-358, November.
    9. Matteo M. Galizzi, 2014. "What Is Really Behavioral in Behavioral Health Policy? And Does It Work?," Applied Economic Perspectives and Policy, Agricultural and Applied Economics Association, vol. 36(1), pages 25-60.
    10. Kriti Jain & Kanchan Mukherjee & J. Neil Bearden & Anil Gaba, 2013. "Unpacking the Future: A Nudge Toward Wider Subjective Confidence Intervals," Management Science, INFORMS, vol. 59(9), pages 1970-1987, September.
    11. Aloysius, John A. & Davis, Fred D. & Wilson, Darryl D. & Ross Taylor, A. & Kottemann, Jeffrey E., 2006. "User acceptance of multi-criteria decision support systems: The impact of preference elicitation techniques," European Journal of Operational Research, Elsevier, vol. 169(1), pages 273-285, February.
    12. Langnickel, Ferdinand & Zeisberger, Stefan, 2016. "Do we measure overconfidence? A closer look at the interval production task," Journal of Economic Behavior & Organization, Elsevier, vol. 128(C), pages 121-133.
    13. repec:cup:judgdm:v:5:y:2010:i:7:p:467-476 is not listed on IDEAS
    14. Matthew B. Welsh & Steve H. Begg, 2018. "More-or-less elicitation (MOLE): reducing bias in range estimation and forecasting," EURO Journal on Decision Processes, Springer;EURO - The Association of European Operational Research Societies, vol. 6(1), pages 171-212, June.
    15. Daniel J. Walters & Philip M. Fernbach & Craig R. Fox & Steven A. Sloman, 2017. "Known Unknowns: A Critical Determinant of Confidence and Calibration," Management Science, INFORMS, vol. 63(12), pages 4298-4307, December.
    16. Yufei Ren & Rachel Croson, 2013. "Overconfidence in Newsvendor Orders: An Experimental Study," Management Science, INFORMS, vol. 59(11), pages 2502-2517, November.
    17. Lahtinen, Tuomas J. & Hämäläinen, Raimo P. & Jenytin, Cosmo, 2020. "On preference elicitation processes which mitigate the accumulation of biases in multi-criteria decision analysis," European Journal of Operational Research, Elsevier, vol. 282(1), pages 201-210.
    18. Keren, Gideon, 1987. "Facing uncertainty in the game of bridge: A calibration study," Organizational Behavior and Human Decision Processes, Elsevier, vol. 39(1), pages 98-114, February.
    19. Craig R. Fox & Amos Tversky, 1998. "A Belief-Based Account of Decision Under Uncertainty," Management Science, INFORMS, vol. 44(7), pages 879-895, July.
    20. Franco, L. Alberto & Hämäläinen, Raimo P. & Rouwette, Etiënne A.J.A. & Leppänen, Ilkka, 2021. "Taking stock of behavioural OR: A review of behavioural studies with an intervention focus," European Journal of Operational Research, Elsevier, vol. 293(2), pages 401-418.
    21. Gilberto Montibeller & Detlof von Winterfeldt, 2015. "Cognitive and Motivational Biases in Decision and Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 35(7), pages 1230-1251, July.
    22. McKenzie, Craig R.M. & Liersch, Michael J. & Yaniv, Ilan, 2008. "Overconfidence in interval estimates: What does expertise buy you?," Organizational Behavior and Human Decision Processes, Elsevier, vol. 107(2), pages 179-191, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ferretti, Valentina & Montibeller, Gilberto & von Winterfeldt, Detlof, 2023. "Testing the effectiveness of debiasing techniques to reduce overprecision in the elicitation of subjective continuous probability distributions," European Journal of Operational Research, Elsevier, vol. 304(2), pages 661-675.
    2. repec:cup:judgdm:v:12:y:2017:i:1:p:29-41 is not listed on IDEAS
    3. Julia P. Prims & Don A. Moore, 2017. "Overconfidence over the lifespan," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 12(1), pages 29-41, January.
    4. Saemi Park & David V. Budescu, 2015. "Aggregating multiple probability intervals to improve calibration," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 10(2), pages 130-143, March.
    5. James S. Dyer & James E. Smith, 2021. "Innovations in the Science and Practice of Decision Analysis: The Role of Management Science," Management Science, INFORMS, vol. 67(9), pages 5364-5378, September.
    6. repec:cup:judgdm:v:10:y:2015:i:2:p:130-143 is not listed on IDEAS
    7. Yun Shin Lee & Enno Siemsen, 2017. "Task Decomposition and Newsvendor Decision Making," Management Science, INFORMS, vol. 63(10), pages 3226-3245, October.
    8. Sulian Wang & Chen Wang, 2021. "Quantile Judgments of Lognormal Losses: An Experimental Investigation," Decision Analysis, INFORMS, vol. 18(1), pages 78-99, March.
    9. Victor Richmond R. Jose & Robert L. Winkler, 2009. "Evaluating Quantile Assessments," Operations Research, INFORMS, vol. 57(5), pages 1287-1297, October.
    10. Thomas W. Keelin & Bradford W. Powley, 2011. "Quantile-Parameterized Distributions," Decision Analysis, INFORMS, vol. 8(3), pages 206-219, September.
    11. Junyi Chai & Zhiquan Weng & Wenbin Liu, 2021. "Behavioral Decision Making in Normative and Descriptive Views: A Critical Review of Literature," JRFM, MDPI, vol. 14(10), pages 1-14, October.
    12. Brenner, Lyle & Griffin, Dale & Koehler, Derek J., 2005. "Modeling patterns of probability calibration with random support theory: Diagnosing case-based judgment," Organizational Behavior and Human Decision Processes, Elsevier, vol. 97(1), pages 64-81, May.
    13. repec:cup:judgdm:v:14:y:2019:i:4:p:395-411 is not listed on IDEAS
    14. repec:cup:judgdm:v:17:y:2022:i:1:p:91-123 is not listed on IDEAS
    15. Ferretti, Valentina & Guney, Sule & Montibeller, Gilberto & Winterfeldt, Detlof von, 2016. "Testing best practices to reduce the overconfidence bias in multi-criteria decision analysis," LSE Research Online Documents on Economics 67179, London School of Economics and Political Science, LSE Library.
    16. Merkle, Christoph, 2018. "The curious case of negative volatility," Journal of Financial Markets, Elsevier, vol. 40(C), pages 92-108.
    17. Ying Han & David Budescu, 2019. "A universal method for evaluating the quality of aggregators," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 14(4), pages 395-411, July.
    18. Vicki Bier, 2020. "The Role of Decision Analysis in Risk Analysis: A Retrospective," Risk Analysis, John Wiley & Sons, vol. 40(S1), pages 2207-2217, November.
    19. Kuller, M. & Beutler, P. & Lienert, J., 2023. "Preference change in stakeholder group-decision processes in the public sector: Extent, causes and implications," European Journal of Operational Research, Elsevier, vol. 308(3), pages 1268-1285.
    20. Anil Gaba & Ilia Tsetlin & Robert L. Winkler, 2017. "Combining Interval Forecasts," Decision Analysis, INFORMS, vol. 14(1), pages 1-20, March.
    21. Ying Han & David V. Budescu, 2022. "Recalibrating probabilistic forecasts to improve their accuracy," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 17(1), pages 91-123, January.
    22. Aurélien Baillon, 2008. "Eliciting Subjective Probabilities Through Exchangeable Events: An Advantage and a Limitation," Decision Analysis, INFORMS, vol. 5(2), pages 76-87, June.
    23. Gilberto Montibeller & Detlof von Winterfeldt, 2015. "Cognitive and Motivational Biases in Decision and Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 35(7), pages 1230-1251, July.
    24. Hao, Zhongyuan & Li, Juan & Cai, Jinling, 2023. "Allocation of inventory responsibilities in overconfident supply chains," European Journal of Operational Research, Elsevier, vol. 305(1), pages 207-221.

    More about this item

    Keywords

    behavioral OR; debiasing; expert judgment; judgment calibration; overconfidence; overprecision;
    All these keywords.

    JEL classification:

    • J50 - Labor and Demographic Economics - - Labor-Management Relations, Trade Unions, and Collective Bargaining - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ehl:lserod:115333. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: LSERO Manager (email available below). General contact details of provider: https://edirc.repec.org/data/lsepsuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.