IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v41y2025i2p487-498.html
   My bibliography  Save this article

How does training improve individual forecasts? Modeling differences in compensatory and non-compensatory biases in geopolitical forecasts

Author

Listed:
  • Karimi Motahhar, Vahid
  • Gruca, Thomas S.

Abstract

Biases in human forecasters lead to poor calibration. We assess how formal training affects two types of bias in probabilistic forecasts of binary outcomes. Compensatory bias occurs when underestimation in one range of probabilities (e.g., less than 50%) is accompanied by overestimation in the opposite range. Non-compensatory bias occurs when the direction of misestimation is consistent throughout the entire range of probabilities. We present a new approach to modeling probabilistic forecasts to determine the extent and direction of compensatory and non-compensatory biases. Using data from the Good Judgment Project, we model the effects of training (randomly assigned) on the calibration of 39,481 initial forecasts from 851 forecasters across two years of the contest. The forecasts exhibit significant indications of both compensatory and non-compensatory biases across all forecasters. Training significantly reduces the compensatory bias in both years. It reduces the non-compensatory bias only in the second year of the contest.

Suggested Citation

  • Karimi Motahhar, Vahid & Gruca, Thomas S., 2025. "How does training improve individual forecasts? Modeling differences in compensatory and non-compensatory biases in geopolitical forecasts," International Journal of Forecasting, Elsevier, vol. 41(2), pages 487-498.
    • Handle: RePEc:eee:intfor:v:41:y:2025:i:2:p:487-498
    DOI: 10.1016/j.ijforecast.2024.12.001
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169207024001298
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijforecast.2024.12.001?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Berg, Joyce E. & Rietz, Thomas A., 2019. "Longshots, overconfidence and efficiency on the Iowa Electronic Market," International Journal of Forecasting, Elsevier, vol. 35(1), pages 271-287.
    2. repec:cup:judgdm:v:11:y:2016:i:5:p:509-526 is not listed on IDEAS
    3. Richard P. Larrick & Jack B. Soll, 2006. "Erratum--Intuitions About Combining Opinions: Misappreciation of the Averaging Principle," Management Science, INFORMS, vol. 52(2), pages 309-310, February.
    4. Merkle, Edgar C. & Steyvers, Mark & Mellers, Barbara & Tetlock, Philip E., 2017. "A neglected dimension of good forecasting judgment: The questions we choose also matter," International Journal of Forecasting, Elsevier, vol. 33(4), pages 817-832.
    5. Ester Faia & Andreas Fuster & Vincenzo Pezone & Basit Zafar, 2024. "Biases in Information Selection and Processing: Survey Evidence from the Pandemic," The Review of Economics and Statistics, MIT Press, vol. 106(3), pages 829-847, May.
    6. Adina D. Sterling & Marissa E. Thompson & Shiya Wang & Abisola Kusimo & Shannon Gilmartin & Sheri Sheppard, 2020. "The confidence gap predicts the gender pay gap among STEM graduates," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 117(48), pages 30303-30308, December.
    7. Welton Chang & Eva Chen & Barbara Mellers & Philip Tetlock, 2016. "Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 11(5), pages 509-526, September.
    8. Jonathan Baron & Barbara A. Mellers & Philip E. Tetlock & Eric Stone & Lyle H. Ungar, 2014. "Two Reasons to Make Aggregated Probability Forecasts More Extreme," Decision Analysis, INFORMS, vol. 11(2), pages 133-145, June.
    9. Pavel Atanasov & Phillip Rescober & Eric Stone & Samuel A. Swift & Emile Servan-Schreiber & Philip Tetlock & Lyle Ungar & Barbara Mellers, 2017. "Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls," Management Science, INFORMS, vol. 63(3), pages 691-706, March.
    10. Erik Snowberg & Justin Wolfers, 2010. "Explaining the Favorite-Long Shot Bias: Is it Risk-Love or Misperceptions?," Journal of Political Economy, University of Chicago Press, vol. 118(4), pages 723-746, August.
    11. Ville A. Satopää & Marat Salikhov & Philip E. Tetlock & Barbara Mellers, 2021. "Bias, Information, Noise: The BIN Model of Forecasting," Management Science, INFORMS, vol. 67(12), pages 7599-7618, December.
    12. Daniel Kahneman & Dan Lovallo, 1993. "Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking," Management Science, INFORMS, vol. 39(1), pages 17-31, January.
    13. Chang, Welton & Chen, Eva & Mellers, Barbara & Tetlock, Philip, 2016. "Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments," Judgment and Decision Making, Cambridge University Press, vol. 11(5), pages 509-526, September.
    14. Don A. Moore & Samuel A. Swift & Angela Minster & Barbara Mellers & Lyle Ungar & Philip Tetlock & Heather H. J. Yang & Elizabeth R. Tenney, 2017. "Confidence Calibration in a Multiyear Geopolitical Forecasting Competition," Management Science, INFORMS, vol. 63(11), pages 3552-3565, November.
    15. Atanasov, Pavel & Witkowski, Jens & Ungar, Lyle & Mellers, Barbara & Tetlock, Philip, 2020. "Small steps to accuracy: Incremental belief updaters are better forecasters," Organizational Behavior and Human Decision Processes, Elsevier, vol. 160(C), pages 19-35.
    16. Arkes, Hal R. & Christensen, Caryn & Lai, Cheryl & Blumer, Catherine, 1987. "Two methods of reducing overconfidence," Organizational Behavior and Human Decision Processes, Elsevier, vol. 39(1), pages 133-144, February.
    17. Stone, Eric R. & Opel, Ryan B., 2000. "Training to Improve Calibration and Discrimination: The Effects of Performance and Environmental Feedback," Organizational Behavior and Human Decision Processes, Elsevier, vol. 83(2), pages 282-309, November.
    18. Richard P. Larrick & Jack B. Soll, 2006. "Intuitions About Combining Opinions: Misappreciation of the Averaging Principle," Management Science, INFORMS, vol. 52(1), pages 111-127, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Don A. Moore & Samuel A. Swift & Angela Minster & Barbara Mellers & Lyle Ungar & Philip Tetlock & Heather H. J. Yang & Elizabeth R. Tenney, 2017. "Confidence Calibration in a Multiyear Geopolitical Forecasting Competition," Management Science, INFORMS, vol. 63(11), pages 3552-3565, November.
    2. Peker, Cem & Wilkening, Tom, 2025. "Robust recalibration of aggregate probability forecasts using meta-beliefs," International Journal of Forecasting, Elsevier, vol. 41(2), pages 613-630.
    3. Robert Mislavsky & Celia Gaertig, 2022. "Combining Probability Forecasts: 60% and 60% Is 60%, but Likely and Likely Is Very Likely," Management Science, INFORMS, vol. 68(1), pages 541-563, January.
    4. Robert L. Winkler & Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose, 2019. "Probability Forecasts and Their Combination: A Research Perspective," Decision Analysis, INFORMS, vol. 16(4), pages 239-260, December.
    5. Asa B. Palley & Jack B. Soll, 2019. "Extracting the Wisdom of Crowds When Information Is Shared," Management Science, INFORMS, vol. 67(5), pages 2291-2309, May.
    6. Mirko Kremer & Brent Moritz & Enno Siemsen, 2011. "Demand Forecasting Behavior: System Neglect and Change Detection," Management Science, INFORMS, vol. 57(10), pages 1827-1843, October.
    7. Atanasov, Pavel & Witkowski, Jens & Ungar, Lyle & Mellers, Barbara & Tetlock, Philip, 2020. "Small steps to accuracy: Incremental belief updaters are better forecasters," Organizational Behavior and Human Decision Processes, Elsevier, vol. 160(C), pages 19-35.
    8. repec:cup:judgdm:v:14:y:2019:i:4:p:395-411 is not listed on IDEAS
    9. Yaniv, Ilan & Choshen-Hillel, Shoham & Milyavsky, Maxim, 2011. "Receiving advice on matters of taste: Similarity, majority influence, and taste discrimination," Organizational Behavior and Human Decision Processes, Elsevier, vol. 115(1), pages 111-120, May.
    10. Ying Han & David Budescu, 2019. "A universal method for evaluating the quality of aggregators," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 14(4), pages 395-411, July.
    11. Ville A. Satopää & Marat Salikhov & Philip E. Tetlock & Barbara Mellers, 2021. "Bias, Information, Noise: The BIN Model of Forecasting," Management Science, INFORMS, vol. 67(12), pages 7599-7618, December.
    12. Jerker Denrell & Christina Fang, 2010. "Predicting the Next Big Thing: Success as a Signal of Poor Judgment," Management Science, INFORMS, vol. 56(10), pages 1653-1667, October.
    13. Rahul Kapoor & Daniel Wilde, 2023. "Peering into a crystal ball: Forecasting behavior and industry foresight," Strategic Management Journal, Wiley Blackwell, vol. 44(3), pages 704-736, March.
    14. Boris Maciejovsky & David V. Budescu, 2020. "Too Much Trust in Group Decisions: Uncovering Hidden Profiles by Groups and Markets," Organization Science, INFORMS, vol. 31(6), pages 1497-1514, November.
    15. Karvetski, Christopher W. & Meinel, Carolyn & Maxwell, Daniel T. & Lu, Yunzi & Mellers, Barbara A. & Tetlock, Philip E., 2022. "What do forecasting rationales reveal about thinking patterns of top geopolitical forecasters?," International Journal of Forecasting, Elsevier, vol. 38(2), pages 688-704.
    16. Armstrong, J. Scott & Green, Kesten C. & Graefe, Andreas, 2015. "Golden rule of forecasting: Be conservative," Journal of Business Research, Elsevier, vol. 68(8), pages 1717-1731.
    17. Ryvkin, Dmitry & Krajč, Marian & Ortmann, Andreas, 2012. "Are the unskilled doomed to remain unaware?," Journal of Economic Psychology, Elsevier, vol. 33(5), pages 1012-1031.
    18. repec:cup:judgdm:v:8:y:2013:i:2:p:91-105 is not listed on IDEAS
    19. Brice Corgnet & Cary Deck & Mark DeSantis & Kyle Hampton & Erik O. Kimbrough, 2023. "When Do Security Markets Aggregate Dispersed Information?," Management Science, INFORMS, vol. 69(6), pages 3697-3729, June.
    20. Julia A. Minson & Jennifer S. Mueller & Richard P. Larrick, 2018. "The Contingent Wisdom of Dyads: When Discussion Enhances vs. Undermines the Accuracy of Collaborative Judgments," Management Science, INFORMS, vol. 64(9), pages 4177-4192, September.
    21. Thomas Astebro & Frank Fossen & Cédric Gutierrez, 2024. "Entrepreneurs: Clueless, Biased, Poor Heuristics, or Bayesian Machines?," Working Papers hal-04759301, HAL.
    22. Yuqian Shi & Sheng Yu & Jie Mei, 2025. "Strategic Decision-Making Enhancement through Graph-Optimized DEMATEL-AHP with Pruning," Group Decision and Negotiation, Springer, vol. 34(1), pages 105-133, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:41:y:2025:i:2:p:487-498. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.