IDEAS home Printed from https://ideas.repec.org/a/wly/fufsci/v6y2024i2ne177.html
   My bibliography  Save this article

Calibration training for improving probabilistic judgments using an interactive app

Author

Listed:
  • Ross Gruetzemacher
  • Kang Bok Lee
  • David Paradice

Abstract

We describe an exploratory study examining the effectiveness of an interactive app and a novel training process for improving calibration and reducing overconfidence in probabilistic judgments. We evaluated the training used in the app by conducting an American college football forecasting tournament involving 153 business school students making 52 forecasts over 11 weeks. A coarsened exact matching analysis found statistical evidence that, in under 30 min, the more challenging training was able to modestly reduce overconfidence, improve calibration and improve the accuracy of probabilistic judgments (measured by the Brier score). The experimental results also suggest that the generic training can generalize across domains and that effective calibration training is possible without expert facilitators or pedagogical training materials. Although no previous studies have reported similar results, due to the modest effect, we conclude that these results should only be interpreted as a proof of concept and that further evaluation and validation of mechanisms of the app's effect is necessary.

Suggested Citation

  • Ross Gruetzemacher & Kang Bok Lee & David Paradice, 2024. "Calibration training for improving probabilistic judgments using an interactive app," Futures & Foresight Science, John Wiley & Sons, vol. 6(2), June.
  • Handle: RePEc:wly:fufsci:v:6:y:2024:i:2:n:e177
    DOI: 10.1002/ffo2.177
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/ffo2.177
    Download Restriction: no

    File URL: https://libkey.io/10.1002/ffo2.177?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Malmendier, Ulrike & Tate, Geoffrey, 2008. "Who makes acquisitions? CEO overconfidence and the market's reaction," Journal of Financial Economics, Elsevier, vol. 89(1), pages 20-43, July.
    2. repec:cup:judgdm:v:11:y:2016:i:5:p:509-526 is not listed on IDEAS
    3. Bolger, Fergus & Onkal-Atay, Dilek, 2004. "The effects of feedback on judgmental interval predictions," International Journal of Forecasting, Elsevier, vol. 20(1), pages 29-39.
    4. Benson, P. George & Onkal, Dilek, 1992. "The effects of feedback and training on the performance of probability forecasters," International Journal of Forecasting, Elsevier, vol. 8(4), pages 559-573, December.
    5. Gregory F. Nemet & Laura Diaz Anadon & Elena Verdolini, 2017. "Quantifying the Effects of Expert Selection and Elicitation Design on Experts’ Confidence in Their Judgments About Future Energy Technologies," Risk Analysis, John Wiley & Sons, vol. 37(2), pages 315-330, February.
    6. Antonio E. Bernardo & Ivo Welch, 2001. "On the Evolution of Overconfidence and Entrepreneurs," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 10(3), pages 301-330, September.
    7. Brad M. Barber & Terrance Odean, 2001. "Boys will be Boys: Gender, Overconfidence, and Common Stock Investment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 116(1), pages 261-292.
    8. Jose A. Scheinkman & Wei Xiong, 2003. "Overconfidence and Speculative Bubbles," Journal of Political Economy, University of Chicago Press, vol. 111(6), pages 1183-1219, December.
    9. Robin M. Hogarth & Natalia Karelaia, 2012. "Entrepreneurial Success and Failure: Confidence and Fallible Judgment," Organization Science, INFORMS, vol. 23(6), pages 1733-1747, December.
    10. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    11. Don A. Moore & Samuel A. Swift & Angela Minster & Barbara Mellers & Lyle Ungar & Philip Tetlock & Heather H. J. Yang & Elizabeth R. Tenney, 2017. "Confidence Calibration in a Multiyear Geopolitical Forecasting Competition," Management Science, INFORMS, vol. 63(11), pages 3552-3565, November.
    12. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    13. Garthwaite, Paul H. & Kadane, Joseph B. & O'Hagan, Anthony, 2005. "Statistical Methods for Eliciting Probability Distributions," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 680-701, June.
    14. Borgonovo, Emanuele & Hazen, Gordon B. & Jose, Victor Richmond R. & Plischke, Elmar, 2021. "Probabilistic sensitivity measures as information value," European Journal of Operational Research, Elsevier, vol. 289(2), pages 595-610.
    15. Iacus, Stefano M. & King, Gary & Porro, Giuseppe, 2012. "Causal Inference without Balance Checking: Coarsened Exact Matching," Political Analysis, Cambridge University Press, vol. 20(1), pages 1-24, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Prokudina, Elena & Renneboog, Luc & Tobler, Philippe, 2015. "Does Confidence Predict Out-of-Domain Effort?," Discussion Paper 2015-055, Tilburg University, Center for Economic Research.
    2. Jonathan F Schulz & Christian Thöni, 2016. "Overconfidence and Career Choice," PLOS ONE, Public Library of Science, vol. 11(1), pages 1-8, January.
    3. David Hirshleifer & Angie Low & Siew Hong Teoh, 2012. "Are Overconfident CEOs Better Innovators?," Journal of Finance, American Finance Association, vol. 67(4), pages 1457-1498, August.
    4. Benoît, Jean-Pierre & Dubra, Juan, 2007. "Overconfidence?," MPRA Paper 6017, University Library of Munich, Germany, revised Nov 2007.
    5. Guy Mayraz, 2011. "Wishful Thinking," CEP Discussion Papers dp1092, Centre for Economic Performance, LSE.
    6. Diego García & Francesco Sangiorgi & Branko Urošević, 2007. "Overconfidence and Market Efficiency with Heterogeneous Agents," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 30(2), pages 313-336, February.
    7. Michael Muthukrishna & Joseph Henrich & Wataru Toyokawa & Takeshi Hamamura & Tatsuya Kameda & Steven J Heine, 2018. "Overconfidence is universal? Elicitation of Genuine Overconfidence (EGO) procedure reveals systematic differences across domain, task knowledge, and incentives in four populations," PLOS ONE, Public Library of Science, vol. 13(8), pages 1-30, August.
    8. Caliendo, Frank & Huang, Kevin X.D., 2008. "Overconfidence and consumption over the life cycle," Journal of Macroeconomics, Elsevier, vol. 30(4), pages 1347-1369, December.
    9. Helen X. H. Bao & Steven Haotong Li, 2016. "Overconfidence And Real Estate Research: A Survey Of The Literature," The Singapore Economic Review (SER), World Scientific Publishing Co. Pte. Ltd., vol. 61(04), pages 1-24, September.
    10. Biljana N. Adebambo & Xuemin (Sterling) Yan, 2018. "Investor Overconfidence, Firm Valuation, and Corporate Decisions," Management Science, INFORMS, vol. 64(11), pages 5349-5369, November.
    11. Danz, David, 2020. "Never underestimate your opponent: Hindsight bias causes overplacement and overentry into competition," Games and Economic Behavior, Elsevier, vol. 124(C), pages 588-603.
    12. Schrand, Catherine M. & Zechman, Sarah L.C., 2012. "Executive overconfidence and the slippery slope to financial misreporting," Journal of Accounting and Economics, Elsevier, vol. 53(1), pages 311-329.
    13. Philipp Koellinger & Theresa Treffers, 2015. "Joy Leads to Overconfidence, and a Simple Countermeasure," PLOS ONE, Public Library of Science, vol. 10(12), pages 1-22, December.
    14. Ahrens, Steffen & Bosch-Rosa, Ciril & Roulund, Rasmus, 2019. "Price Dynamics and Trader Overconfidence," Rationality and Competition Discussion Paper Series 161, CRC TRR 190 Rationality and Competition.
    15. Banerjee, Suman & Dai, Lili & Humphery-Jenner, Mark & Nanda, Vikram, 2020. "Governance, board inattention, and the appointment of overconfident CEOs," Journal of Banking & Finance, Elsevier, vol. 113(C).
    16. Pikulina, Elena & Renneboog, Luc & Tobler, Philippe N., 2018. "Do confident individuals generally work harder?," Journal of Multinational Financial Management, Elsevier, vol. 44(C), pages 51-60.
    17. Cédric Gutierrez & Thomas Åstebro & Tomasz Obloj, 2020. "The Impact of Overconfidence and Ambiguity Attitude on Market Entry," Organization Science, INFORMS, vol. 31(2), pages 308-329, March.
    18. Brice Corgnet & Mark DeSantis & David Porter, 2015. "What Makes a Good Trader? On the Role of Quant Skills, Behavioral Biases and Intuition on Trader Performance," Working Papers 15-17, Chapman University, Economic Science Institute.
    19. Hanaki, Nobuyuki & Akiyama, Eizo & Ishikawa, Ryuichiro, 2018. "Behavioral uncertainty and the dynamics of traders’ confidence in their price forecasts," Journal of Economic Dynamics and Control, Elsevier, vol. 88(C), pages 121-136.
    20. Nguyen, Nhut H. & Truong, Cameron, 2013. "The information content of stock markets around the world: A cultural explanation," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 26(C), pages 1-29.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:fufsci:v:6:y:2024:i:2:n:e177. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1002/(ISSN)2573-5152 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.