IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v93y2008i5p711-721.html
   My bibliography  Save this article

A study of expert overconfidence

Author

Listed:
  • Lin, Shi-Woei
  • Bier, Vicki M.

Abstract

Overconfidence is one of the most common (and potentially severe) problems in expert judgment. To assess the extent of expert overconfidence, we analyzed a large data set on expert opinion compiled by Cooke and colleagues at the Technical University of Delft and elsewhere. This data set contains roughly five thousand 90% confidence intervals of uncertain quantities for which the true values are now known. Our analysis assesses the overall extent of overconfidence in the data set. Significant differences in the extent of overconfidence were found among studies, among experts, and among questions within a study. Moreover, replications (multiple realizations for the same question) allowed a preliminary assessment of whether the question effect is due largely to question difficulty, or merely to random noise in the realizations of the uncertain quantities. The results of this analysis suggest that much of the apparent question effect may be due to noise rather than systematic differences in the difficulty of achieving good calibration for different questions. The results support the differential weighting of experts, since there are significant differences in expert calibration within studies.

Suggested Citation

  • Lin, Shi-Woei & Bier, Vicki M., 2008. "A study of expert overconfidence," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 711-721.
    • Handle: RePEc:eee:reensy:v:93:y:2008:i:5:p:711-721
    DOI: 10.1016/j.ress.2007.03.014
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832007000981
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ress.2007.03.014?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Klayman, Joshua & Soll, Jack B. & Gonzalez-Vallejo, Claudia & Barlas, Sema, 1999. "Overconfidence: It Depends on How, What, and Whom You Ask, , , , , , , , ," Organizational Behavior and Human Decision Processes, Elsevier, vol. 79(3), pages 216-247, September.
    2. Roger Cooke & Eric Jager, 1998. "A Probabilistic Model for the Failure Frequency of Underground Gas Pipelines," Risk Analysis, John Wiley & Sons, vol. 18(4), pages 511-527, August.
    3. Alexander I. Shlyakhter, 1994. "An Improved Framework for Uncertainty Analysis: Accounting for Unsuspected Errors," Risk Analysis, John Wiley & Sons, vol. 14(4), pages 441-447, August.
    4. L.H.J. Goossens & R.M. Cooke & F Woudenberg & P Van Der Torn, 1998. "Expert judgement and lethal toxicity of inhaled chemicals," Journal of Risk Research, Taylor & Francis Journals, vol. 1(2), pages 117-133, April.
    5. Shlyakhter, Alexander I. & Kammen, Daniel M. & Broido, Claire L. & Wilson, Richard, 1994. "Quantifying the credibility of energy projections from trends in past data : The US energy sector," Energy Policy, Elsevier, vol. 22(2), pages 119-130, February.
    6. Robert L. Winkler, 1981. "Combining Probability Distributions from Dependent Information Sources," Management Science, INFORMS, vol. 27(4), pages 479-488, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. World Health Organization, Foodborne Epidemiology Reference Group, Source Attribution Task Force, 2016. "Research Synthesis Methods in an Age of Globalized Risks: Lessons from the Global Burden of Foodborne Disease Expert Elicitation," Risk Analysis, John Wiley & Sons, vol. 36(2), pages 191-202, February.
    2. James K. Hammitt & Yifan Zhang, 2013. "Combining Experts’ Judgments: Comparison of Algorithmic Methods Using Synthetic Data," Risk Analysis, John Wiley & Sons, vol. 33(1), pages 109-120, January.
    3. Bolger, Fergus & Wright, George, 2017. "Use of expert knowledge to anticipate the future: Issues, analysis and directions," International Journal of Forecasting, Elsevier, vol. 33(1), pages 230-243.
    4. Kash Barker & Kaycee J. Wilson, 2012. "Decision Trees with Single and Multiple Interval-Valued Objectives," Decision Analysis, INFORMS, vol. 9(4), pages 348-358, December.
    5. David R. Mandel & Daniel Irwin, 2021. "Tracking accuracy of strategic intelligence forecasts: Findings from a long‐term Canadian study," Futures & Foresight Science, John Wiley & Sons, vol. 3(3-4), September.
    6. Bonnie C. Wintle & Ann Nicholson, 2014. "Exploring Risk Judgments in a Trade Dispute Using Bayesian Networks," Risk Analysis, John Wiley & Sons, vol. 34(6), pages 1095-1111, June.
    7. Erin Baker & Olaitan Olaleye, 2013. "Combining Experts: Decomposition and Aggregation Order," Risk Analysis, John Wiley & Sons, vol. 33(6), pages 1116-1127, June.
    8. Compernolle, Tine & Welkenhuysen, Kris & Petitclerc, Estelle & Maes, Dries & Piessens, Kris, 2019. "The impact of policy measures on profitability and risk in geothermal energy investments," Energy Economics, Elsevier, vol. 84(C).
    9. Das Sourish & Yang Hongxia & Banks David, 2012. "Synthetic Priors that Merge Opinion from Multiple Experts," Statistics, Politics and Policy, De Gruyter, vol. 4(1), pages 1-26, December.
    10. Barker, Kash & Haimes, Yacov Y., 2009. "Assessing uncertainty in extreme events: Applications to risk-based decision making in interdependent infrastructure sectors," Reliability Engineering and System Safety, Elsevier, vol. 94(4), pages 819-829.
    11. Gilberto Montibeller & Detlof von Winterfeldt, 2015. "Cognitive and Motivational Biases in Decision and Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 35(7), pages 1230-1251, July.
    12. Jeremy Rohmer & Eric Chojnacki, 2021. "Forecast of environment systems using expert judgements: performance comparison between the possibilistic and the classical model," Environment Systems and Decisions, Springer, vol. 41(1), pages 131-146, March.
    13. Strigini, Lorenzo & Wright, David, 2014. "Bounds on survival probability given mean probability of failure per demand; and the paradoxical advantages of uncertainty," Reliability Engineering and System Safety, Elsevier, vol. 128(C), pages 66-83.
    14. Morais, Caroline & Estrada-Lugo, Hector Diego & Tolo, Silvia & Jacques, Tiago & Moura, Raphael & Beer, Michael & Patelli, Edoardo, 2022. "Robust data-driven human reliability analysis using credal networks," Reliability Engineering and System Safety, Elsevier, vol. 218(PA).
    15. Fergus Bolger & Gene Rowe, 2015. "There is Data, and then there is Data: Only Experimental Evidence will Determine the Utility of Differential Weighting of Expert Judgment," Risk Analysis, John Wiley & Sons, vol. 35(1), pages 21-26, January.
    16. Cooke, Roger M., 2014. "Deep and Shallow Uncertainty in Messaging Climate Change," RFF Working Paper Series dp-14-11, Resources for the Future.
    17. Baker, Erin & Bosetti, Valentina & Anadon, Laura Diaz & Henrion, Max & Aleluia Reis, Lara, 2015. "Future costs of key low-carbon energy technologies: Harmonization and aggregation of energy technology expert elicitation data," Energy Policy, Elsevier, vol. 80(C), pages 219-232.
    18. Morgan, Horatio M. & Sui, Sui & Baum, Matthias, 2018. "Are SMEs with immigrant owners exceptional exporters?," Journal of Business Venturing, Elsevier, vol. 33(3), pages 241-260.
    19. Roger M. Cooke, 2015. "The Aggregation of Expert Judgment: Do Good Things Come to Those Who Weight?," Risk Analysis, John Wiley & Sons, vol. 35(1), pages 12-15, January.
    20. Hanea, D.M. & Jagtman, H.M. & van Alphen, L.L.M.M. & Ale, B.J.M., 2010. "Quantitative and qualitative analysis of the expert and non-expert opinion in fire risk in buildings," Reliability Engineering and System Safety, Elsevier, vol. 95(7), pages 729-741.
    21. Eggstaff, Justin W. & Mazzuchi, Thomas A. & Sarkani, Shahram, 2014. "The effect of the number of seed variables on the performance of Cooke′s classical model," Reliability Engineering and System Safety, Elsevier, vol. 121(C), pages 72-82.
    22. Erin Baker & Olaitan Olaleye & Lara Aleluia Reis, 2015. "Decision Frameworks and the Investment in R&D," Working Papers 2015.42, Fondazione Eni Enrico Mattei.
    23. Hathout, Michel & Vuillet, Marc & Carvajal, Claudio & Peyras, Laurent & Diab, Youssef, 2019. "Expert judgments calibration and combination for assessment of river levee failure probability," Reliability Engineering and System Safety, Elsevier, vol. 188(C), pages 377-392.
    24. Sulian Wang & Chen Wang, 2021. "Quantile Judgments of Lognormal Losses: An Experimental Investigation," Decision Analysis, INFORMS, vol. 18(1), pages 78-99, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sulian Wang & Chen Wang, 2021. "Quantile Judgments of Lognormal Losses: An Experimental Investigation," Decision Analysis, INFORMS, vol. 18(1), pages 78-99, March.
    2. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    3. James K. Hammitt & Alexander I. Shlyakhter, 1999. "The Expected Value of Information and the Probability of Surprise," Risk Analysis, John Wiley & Sons, vol. 19(1), pages 135-152, February.
    4. Thunström, Linda & Nordström, Jonas & Shogren, Jason F., 2015. "Certainty and overconfidence in future preferences for food," Journal of Economic Psychology, Elsevier, vol. 51(C), pages 101-113.
    5. Daniel Fonseca Costa & Francisval Carvalho & Bruno César Moreira & José Willer Prado, 2017. "Bibliometric analysis on the association between behavioral finance and decision making with cognitive biases such as overconfidence, anchoring effect and confirmation bias," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1775-1799, June.
    6. Herz, Holger & Schunk, Daniel & Zehnder, Christian, 2014. "How do judgmental overconfidence and overoptimism shape innovative activity?," Games and Economic Behavior, Elsevier, vol. 83(C), pages 1-23.
    7. David Hirshleifer & Angie Low & Siew Hong Teoh, 2012. "Are Overconfident CEOs Better Innovators?," Journal of Finance, American Finance Association, vol. 67(4), pages 1457-1498, August.
    8. Tuan Pham, Michel & Meyvis, Tom & Zhou, Rongrong, 2001. "Beyond the Obvious: Chronic Vividness of Imagery and the Use of Information in Decision Making," Organizational Behavior and Human Decision Processes, Elsevier, vol. 84(2), pages 226-253, March.
    9. Juanjuan Qin & Liguo Ren & Liangjie Xia, 2017. "Carbon Emission Reduction and Pricing Strategies of Supply Chain under Various Demand Forecasting Scenarios," Asia-Pacific Journal of Operational Research (APJOR), World Scientific Publishing Co. Pte. Ltd., vol. 34(01), pages 1-27, February.
    10. Brenner, Lyle & Griffin, Dale & Koehler, Derek J., 2005. "Modeling patterns of probability calibration with random support theory: Diagnosing case-based judgment," Organizational Behavior and Human Decision Processes, Elsevier, vol. 97(1), pages 64-81, May.
    11. Anne Sofie Jore & James Mitchell & Shaun P. Vahey, 2010. "Combining forecast densities from VARs with uncertain instabilities," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 621-634.
    12. Hall, Stephen G. & Mitchell, James, 2007. "Combining density forecasts," International Journal of Forecasting, Elsevier, vol. 23(1), pages 1-13.
    13. Victor R. Fuchs & Alan B. Krueger & James M. Poterba, 1997. "Why do Economists Disagree About Policy?," NBER Working Papers 6151, National Bureau of Economic Research, Inc.
    14. Nicolao Bonini & Stefania Pighin & Enrico Rettore & Lucia Savadori & Federico Schena & Sara Tonini & Paolo Tosi, 2019. "Overconfident people are more exposed to “black swan” events: a case study of avalanche risk," Empirical Economics, Springer, vol. 57(4), pages 1443-1467, October.
    15. Stephen C. Hora, 2013. "A Note on the Aggregation of Event Probabilities," Risk Analysis, John Wiley & Sons, vol. 33(5), pages 909-914, May.
    16. Ludwig, Sandra & Fellner-Röhling, Gerlinde & Thoma, Carmen, 2017. "Do women have more shame than men? An experiment on self-assessment and the shame of overestimating oneself," European Economic Review, Elsevier, vol. 92(C), pages 31-46.
    17. McKenzie, Craig R.M. & Liersch, Michael J. & Yaniv, Ilan, 2008. "Overconfidence in interval estimates: What does expertise buy you?," Organizational Behavior and Human Decision Processes, Elsevier, vol. 107(2), pages 179-191, November.
    18. repec:cup:judgdm:v:14:y:2019:i:4:p:395-411 is not listed on IDEAS
    19. Markus Glaser & Thomas Langer & Martin Weber, 2007. "On the Trend Recognition and Forecasting Ability of Professional Traders," Decision Analysis, INFORMS, vol. 4(4), pages 176-193, December.
    20. Ali Mosleh & George Apostolakis, 1986. "The Assessment of Probability Distributions from Expert Opinions with an Application to Seismic Fragility Curves," Risk Analysis, John Wiley & Sons, vol. 6(4), pages 447-461, December.
    21. Fellner-Röhling, Gerlinde & Krügel, Sebastian, 2014. "Judgmental overconfidence and trading activity," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PB), pages 827-842.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:93:y:2008:i:5:p:711-721. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.