IDEAS home Printed from https://ideas.repec.org/a/wly/fufsci/v3y2021i3-4ne98.html
   My bibliography  Save this article

Tracking accuracy of strategic intelligence forecasts: Findings from a long‐term Canadian study

Author

Listed:
  • David R. Mandel
  • Daniel Irwin

Abstract

Forecasting plays a vital role in intelligence assessment and contributes to national security decision‐making by improving strategic foresight. Remarkably, most intelligence organizations do not proactively track their forecasting accuracy and, therefore, do not know how accurate their forecasts are or what types of biases intelligence analysts (or organizations) might exhibit. We review research on geopolitical forecasting and a roughly decade‐long program of research to assess the accuracy of strategic intelligence forecasts produced by and for the Government of Canada. This research is described in three phases corresponding to previously published research, following which novel analyses (drawing from the data used in the earlier phases) are reported. The findings reveal a high degree of forecasting accuracy as well as significant underconfidence. These results were evident regardless of whether analysts assigned numeric probabilities to their forecasts. However, the novel analyses clarified that there is a substantial cost to accuracy if end‐users rely on their own interpretations of verbal probability terms used in the forecasts. We recommend that intelligence organizations proactively track forecasting accuracy as a means of supporting accountability and organizational learning. We further recommend that intelligence organizations use numeric probabilities in their forecasts to support better comprehension of these estimates by end‐users.

Suggested Citation

  • David R. Mandel & Daniel Irwin, 2021. "Tracking accuracy of strategic intelligence forecasts: Findings from a long‐term Canadian study," Futures & Foresight Science, John Wiley & Sons, vol. 3(3-4), September.
  • Handle: RePEc:wly:fufsci:v:3:y:2021:i:3-4:n:e98
    DOI: 10.1002/ffo2.98
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/ffo2.98
    Download Restriction: no

    File URL: https://libkey.io/10.1002/ffo2.98?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. repec:cup:judgdm:v:12:y:2017:i:4:p:369-381 is not listed on IDEAS
    2. Jonathan Baron & Barbara A. Mellers & Philip E. Tetlock & Eric Stone & Lyle H. Ungar, 2014. "Two Reasons to Make Aggregated Probability Forecasts More Extreme," Decision Analysis, INFORMS, vol. 11(2), pages 133-145, June.
    3. Pavel Atanasov & Phillip Rescober & Eric Stone & Samuel A. Swift & Emile Servan-Schreiber & Philip Tetlock & Lyle Ungar & Barbara Mellers, 2017. "Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls," Management Science, INFORMS, vol. 63(3), pages 691-706, March.
    4. repec:cup:judgdm:v:13:y:2018:i:6:p:607-621 is not listed on IDEAS
    5. repec:cup:judgdm:v:7:y:2012:i:6:p:728-740 is not listed on IDEAS
    6. Christopher W. Karvetski & David R. Mandel & Daniel Irwin, 2020. "Improving Probability Judgment in Intelligence Analysis: From Structured Analysis to Statistical Aggregation," Risk Analysis, John Wiley & Sons, vol. 40(5), pages 1040-1057, May.
    7. repec:cup:judgdm:v:13:y:2018:i:2:p:202-211 is not listed on IDEAS
    8. Lin, Shi-Woei & Bier, Vicki M., 2008. "A study of expert overconfidence," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 711-721.
    9. Thomas Astebro & D. Koehler, 2007. "Calibration Accuracy of a Judgmental Process that Predicts the Commercial Success of New Product Ideas," Post-Print hal-00451653, HAL.
    10. repec:cup:judgdm:v:16:y:2021:i:2:p:363-393 is not listed on IDEAS
    11. David V. Budescu & Han-Hui Por & Stephen B. Broomell & Michael Smithson, 2014. "The interpretation of IPCC probabilistic statements around the world," Nature Climate Change, Nature, vol. 4(6), pages 508-512, June.
    12. Christopher W. Karvetski & Kenneth C. Olson & David R. Mandel & Charles R. Twardy, 2013. "Probabilistic Coherence Weighting for Optimizing Expert Forecasts," Decision Analysis, INFORMS, vol. 10(4), pages 305-326, December.
    13. Eva Chen & David V. Budescu & Shrinidhi K. Lakshmikanth & Barbara A. Mellers & Philip E. Tetlock, 2016. "Validating the Contribution-Weighted Model: Robustness and Cost-Benefit Analyses," Decision Analysis, INFORMS, vol. 13(2), pages 128-152, June.
    14. repec:cup:judgdm:v:14:y:2019:i:3:p:288-292 is not listed on IDEAS
    15. repec:cup:judgdm:v:14:y:2019:i:6:p:683-695 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:cup:judgdm:v:13:y:2018:i:6:p:607-621 is not listed on IDEAS
    2. Spyros Galanis & Sergei Mikhalishchev, 2024. "Information Aggregation with Costly Information Acquisition," Papers 2406.07186, arXiv.org.
    3. Ying Han & David V. Budescu, 2022. "Recalibrating probabilistic forecasts to improve their accuracy," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 17(1), pages 91-123, January.
    4. repec:cup:judgdm:v:15:y:2020:i:6:p:939-958 is not listed on IDEAS
    5. Jared A. Beekman & Ronald F. A. Woodaman & Dennis M. Buede, 2020. "A Review of Probabilistic Opinion Pooling Algorithms with Application to Insider Threat Detection," Decision Analysis, INFORMS, vol. 17(1), pages 39-55, March.
    6. Don A. Moore & Samuel A. Swift & Angela Minster & Barbara Mellers & Lyle Ungar & Philip Tetlock & Heather H. J. Yang & Elizabeth R. Tenney, 2017. "Confidence Calibration in a Multiyear Geopolitical Forecasting Competition," Management Science, INFORMS, vol. 63(11), pages 3552-3565, November.
    7. repec:cup:judgdm:v:14:y:2019:i:4:p:395-411 is not listed on IDEAS
    8. repec:cup:judgdm:v:17:y:2022:i:1:p:91-123 is not listed on IDEAS
    9. Yuyu Fan & David V. Budescu & David Mandel & Mark Himmelstein, 2019. "Improving Accuracy by Coherence Weighting of Direct and Ratio Probability Judgments," Decision Analysis, INFORMS, vol. 16(3), pages 197-217, September.
    10. Ying Han & David Budescu, 2019. "A universal method for evaluating the quality of aggregators," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 14(4), pages 395-411, July.
    11. Ville A. Satopää & Marat Salikhov & Philip E. Tetlock & Barbara Mellers, 2021. "Bias, Information, Noise: The BIN Model of Forecasting," Management Science, INFORMS, vol. 67(12), pages 7599-7618, December.
    12. Christopher W. Karvetski & David R. Mandel, 2020. "Coherence of probability judgments from uncertain evidence: Does ACH help?," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 15(6), pages 939-958, November.
    13. World Health Organization, Foodborne Epidemiology Reference Group, Source Attribution Task Force, 2016. "Research Synthesis Methods in an Age of Globalized Risks: Lessons from the Global Burden of Foodborne Disease Expert Elicitation," Risk Analysis, John Wiley & Sons, vol. 36(2), pages 191-202, February.
    14. Satopää, Ville A. & Salikhov, Marat & Tetlock, Philip E. & Mellers, Barbara, 2023. "Decomposing the effects of crowd-wisdom aggregators: The bias–information–noise (BIN) model," International Journal of Forecasting, Elsevier, vol. 39(1), pages 470-485.
    15. David R. Mandel & Christopher W. Karvetski & Mandeep K. Dhami, 2018. "Boosting intelligence analysts’ judgment accuracy: What works, what fails?," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(6), pages 607-621, November.
    16. Karvetski, Christopher W. & Meinel, Carolyn & Maxwell, Daniel T. & Lu, Yunzi & Mellers, Barbara A. & Tetlock, Philip E., 2022. "What do forecasting rationales reveal about thinking patterns of top geopolitical forecasters?," International Journal of Forecasting, Elsevier, vol. 38(2), pages 688-704.
    17. Bernd Frick & Franziska Prockl, 2018. "Information Precision In Online Communities: Player Valuations On Www.Transfermarkt.De," Working Papers Dissertations 37, Paderborn University, Faculty of Business Administration and Economics.
    18. repec:cup:judgdm:v:8:y:2013:i:5:p:552-560 is not listed on IDEAS
    19. Ajay Agrawal & Joshua S. Gans & Scott Stern, 2021. "Enabling Entrepreneurial Choice," Management Science, INFORMS, vol. 67(9), pages 5510-5524, September.
    20. P A Hancock & William G Volante, 2020. "Quantifying the qualities of language," PLOS ONE, Public Library of Science, vol. 15(5), pages 1-27, May.
    21. Brice Corgnet & Cary Deck & Mark DeSantis & Kyle Hampton & Erik O. Kimbrough, 2023. "When Do Security Markets Aggregate Dispersed Information?," Management Science, INFORMS, vol. 69(6), pages 3697-3729, June.
    22. Hanea, A.M. & McBride, M.F. & Burgman, M.A. & Wintle, B.C. & Fidler, F. & Flander, L. & Twardy, C.R. & Manning, B. & Mascaro, S., 2017. "I nvestigate D iscuss E stimate A ggregate for structured expert judgement," International Journal of Forecasting, Elsevier, vol. 33(1), pages 267-279.
    23. Hanea, D.M. & Jagtman, H.M. & van Alphen, L.L.M.M. & Ale, B.J.M., 2010. "Quantitative and qualitative analysis of the expert and non-expert opinion in fire risk in buildings," Reliability Engineering and System Safety, Elsevier, vol. 95(7), pages 729-741.
    24. Marcellin Martinie & Tom Wilkening & Piers D L Howe, 2020. "Using meta-predictions to identify experts in the crowd when past performance is unknown," PLOS ONE, Public Library of Science, vol. 15(4), pages 1-11, April.
    25. repec:cup:judgdm:v:14:y:2019:i:2:p:135-147 is not listed on IDEAS
    26. Gruetzemacher, Ross & Dorner, Florian E. & Bernaola-Alvarez, Niko & Giattino, Charlie & Manheim, David, 2021. "Forecasting AI progress: A research agenda," Technological Forecasting and Social Change, Elsevier, vol. 170(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:fufsci:v:3:y:2021:i:3-4:n:e98. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1002/(ISSN)2573-5152 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.