IDEAS home Printed from https://ideas.repec.org/p/ags/feemmi/249349.html
   My bibliography  Save this paper

Quantifying the Effects of Expert Selection and Elicitation Design on Experts’ Confidence in their Judgments about Future Energy Technologies

Author

Listed:
  • Nemet, Gregory F.
  • Anadon, Laura Diaz
  • Verdolini, Elena

Abstract

Expert elicitations are frequently used to characterize future technology outcomes. However their usefulness is limited, in part because: estimates across studies are not easily comparable; choices in survey design and expert selection may bias results; and over-confidence is a persistent problem. We provide quantitative evidence of how these choices affect experts’ estimates of the costs of future energy technologies. We harmonize data from 19 elicitations, involving 215 experts, on the 2030 costs of 5 energy technologies: nuclear, biofuels, bioelectricity, solar, and carbon capture. We control for expert characteristics, survey design, and public R&D investment levels on which the elicited values are conditional. We find that, on average, when experts respond to elicitations in person, they ascribe lower confidence (larger uncertainty) to their estimates than when responding via mail or online. In-person interviews also produce more optimistic assessments of best-case (10th percentile) outcomes. The impacts of expert affiliation—government, private sector, or academic—and geography—US or EU—are also significant; academics and US experts have lower confidence than other types of experts. Higher R&D investment levels have no effect on the confidence of experts’ judgments. R&D reduces both the median and breakthrough (10th percentile) cost estimates, although the size of the effect varies across technologies. These results indicate the source, direction, and size of bias in energy technology elicitations. They also point to the technology specificity of some of the effects. These biases should be seriously considered, both in interpreting the results of existing elicitations and in designing new ones.

Suggested Citation

  • Nemet, Gregory F. & Anadon, Laura Diaz & Verdolini, Elena, 2016. "Quantifying the Effects of Expert Selection and Elicitation Design on Experts’ Confidence in their Judgments about Future Energy Technologies," MITP: Mitigation, Innovation and Transformation Pathways 249349, Fondazione Eni Enrico Mattei (FEEM).
  • Handle: RePEc:ags:feemmi:249349
    DOI: 10.22004/ag.econ.249349
    as

    Download full text from publisher

    File URL: https://ageconsearch.umn.edu/record/249349/files/NDL2016-061.pdf
    Download Restriction: no

    File URL: https://libkey.io/10.22004/ag.econ.249349?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Baker, Erin & Chon, Haewon & Keisler, Jeffrey, 2009. "Advanced solar R&D: Combining economic analysis with expert elicitations to inform climate policy," Energy Economics, Elsevier, vol. 31(Supplemen), pages 37-49.
    2. Shital A. Thekdi & James H. Lambert, 2012. "Decision Analysis and Risk Models for Land Development Affecting Infrastructure Systems," Risk Analysis, John Wiley & Sons, vol. 32(7), pages 1253-1269, July.
    3. Verdolini, Elena & Galeotti, Marzio, 2011. "At home and abroad: An empirical analysis of innovation and diffusion in energy technologies," Journal of Environmental Economics and Management, Elsevier, vol. 61(2), pages 119-134, March.
    4. Weyant, John P., 2011. "Accelerating the development and diffusion of new energy technologies: Beyond the "valley of death"," Energy Economics, Elsevier, vol. 33(4), pages 674-682, July.
    5. Luca Zamparini & Aura Reggiani, 2007. "Meta-Analysis and the Value of Travel Time Savings: A Transatlantic Perspective in Passenger Transport," Networks and Spatial Economics, Springer, vol. 7(4), pages 377-396, December.
    6. Yan Cao & William L. McGill, 2013. "LinkIT: A Ludic Elicitation Game for Eliciting Risk Perceptions," Risk Analysis, John Wiley & Sons, vol. 33(6), pages 1066-1082, June.
    7. Verdolini, Elena & Anadon, Laura Diaz & Lu, Jiaqi & Nemet, Gregory F., 2015. "The effects of expert selection, elicitation design, and R&D assumptions on experts' estimates of the future costs of photovoltaics," Energy Policy, Elsevier, vol. 80(C), pages 233-243.
    8. Fergus Bolger & Gene Rowe, 2015. "There is Data, and then there is Data: Only Experimental Evidence will Determine the Utility of Differential Weighting of Expert Judgment," Risk Analysis, John Wiley & Sons, vol. 35(1), pages 21-26, January.
    9. Roger M. Cooke, 2015. "The Aggregation of Expert Judgment: Do Good Things Come to Those Who Weight?," Risk Analysis, John Wiley & Sons, vol. 35(1), pages 12-15, January.
    10. Fiorese, Giulia & Catenacci, Michela & Verdolini, Elena & Bosetti, Valentina, 2013. "Advanced biofuels: Future perspectives from an expert elicitation survey," Energy Policy, Elsevier, vol. 56(C), pages 293-311.
    11. Bosetti, Valentina & Catenacci, Michela & Fiorese, Giulia & Verdolini, Elena, 2012. "The future prospect of PV and CSP solar technologies: An expert elicitation survey," Energy Policy, Elsevier, vol. 49(C), pages 308-317.
    12. Terry Barker and Katie Jenkins, 2007. "The Costs of Avoiding Dangerous Climate Change: Estimates Derived from a Meta-Analysis of the Literature," Human Development Occasional Papers (1992-2007) HDOCPA-2007-02, Human Development Report Office (HDRO), United Nations Development Programme (UNDP).
    13. Goldberg, Pinelopi K. & Verboven, Frank, 2005. "Market integration and convergence to the Law of One Price: evidence from the European car market," Journal of International Economics, Elsevier, vol. 65(1), pages 49-73, January.
    14. Nigel W. Arnell & Emma L. Tompkins & W. Neil Adger, 2005. "Eliciting Information from Experts on the Likelihood of Rapid Climate Change," Risk Analysis, John Wiley & Sons, vol. 25(6), pages 1419-1431, December.
    15. Didier Dubois, 2010. "Representation, Propagation, and Decision Issues in Risk Analysis Under Incomplete Probabilistic Information," Risk Analysis, John Wiley & Sons, vol. 30(3), pages 361-368, March.
    16. Henry A. Roman & James K. Hammitt & Tyra L. Walsh & David M. Stieb, 2012. "Expert Elicitation of the Value per Statistical Life in an Air Pollution Context," Risk Analysis, John Wiley & Sons, vol. 32(12), pages 2133-2151, December.
    17. Andrew Speirs‐Bridge & Fiona Fidler & Marissa McBride & Louisa Flander & Geoff Cumming & Mark Burgman, 2010. "Reducing Overconfidence in the Interval Judgments of Experts," Risk Analysis, John Wiley & Sons, vol. 30(3), pages 512-523, March.
    18. Diaz Anadon, Laura & Nemet, Gregory & Verdolini, Elena, 2013. "The Future Costs of Nuclear Power Using Multiple Expert Elicitations: Effects of RD&D and Elicitation Design," Climate Change and Sustainable Development 158747, Fondazione Eni Enrico Mattei (FEEM).
    19. Adam Rose & Noah Dormady, 2011. "A Meta-Analysis of the Economic Impacts of Climate Change Policy in the United States," The Energy Journal, International Association for Energy Economics, vol. 0(Number 2), pages 143-166.
    20. Seel, Joachim & Barbose, Galen L. & Wiser, Ryan H., 2014. "An analysis of residential PV system price differences between the United States and Germany," Energy Policy, Elsevier, vol. 69(C), pages 216-226.
    21. Juliana Martins Ruzante & Valerie J. Davidson & Julie Caswell & Aamir Fazil & John A. L. Cranfield & Spencer J. Henson & Sven M. Anders & Claudia Schmidt & Jeffrey M. Farber, 2010. "A Multifactorial Risk Prioritization Framework for Foodborne Pathogens," Risk Analysis, John Wiley & Sons, vol. 30(5), pages 724-742, May.
    22. Koomey, Jonathan & Hultman, Nathan E., 2007. "A reactor-level analysis of busbar costs for US nuclear plants, 1970-2005," Energy Policy, Elsevier, vol. 35(11), pages 5630-5642, November.
    23. Havranek, Tomas & Irsova, Zuzana & Janda, Karel, 2012. "Demand for gasoline is more price-inelastic than commonly thought," Energy Economics, Elsevier, vol. 34(1), pages 201-207.
    24. Havranek, Tomas & Irsova, Zuzana & Janda, Karel, 2012. "Demand for gasoline is more price-inelastic than commonly thought," Energy Economics, Elsevier, vol. 34(1), pages 201-207.
    25. Stephanie E. Chang & Timothy McDaniels & Jana Fox & Rajan Dhariwal & Holly Longstaff, 2014. "Toward Disaster‐Resilient Cities: Characterizing Resilience of Infrastructure Systems with Expert Judgments," Risk Analysis, John Wiley & Sons, vol. 34(3), pages 416-434, March.
    26. John Paul Gosling & Andy Hart & David C. Mouat & Mirzet Sabirovic & Simon Scanlan & Alick Simmons, 2012. "Quantifying Experts’ Uncertainty About the Future Cost of Exotic Diseases," Risk Analysis, John Wiley & Sons, vol. 32(5), pages 881-893, May.
    27. Terje Aven, 2013. "On How to Deal with Deep Uncertainties in a Risk Assessment and Management Context," Risk Analysis, John Wiley & Sons, vol. 33(12), pages 2082-2091, December.
    28. Elmar Kriegler & John Weyant & Geoffrey Blanford & Volker Krey & Leon Clarke & Jae Edmonds & Allen Fawcett & Gunnar Luderer & Keywan Riahi & Richard Richels & Steven Rose & Massimo Tavoni & Detlef Vuu, 2014. "The role of technology for achieving climate policy objectives: overview of the EMF 27 study on global technology and climate policy strategies," Climatic Change, Springer, vol. 123(3), pages 353-367, April.
    29. M. Granger Morgan, 2015. "Our Knowledge of the World is Often Not Simple: Policymakers Should Not Duck that Fact, But Should Deal with It," Risk Analysis, John Wiley & Sons, vol. 35(1), pages 19-20, January.
    30. Harry Otway & Detlof von Winterfeldt, 1992. "Expert Judgment in Risk Analysis and Management: Process, Context, and Pitfalls," Risk Analysis, John Wiley & Sons, vol. 12(1), pages 83-93, March.
    31. Robert L. Winkler*, 2015. "Equal Versus Differential Weighting in Combining Forecasts," Risk Analysis, John Wiley & Sons, vol. 35(1), pages 16-18, January.
    32. Gilberto Montibeller & Detlof von Winterfeldt, 2015. "Cognitive and Motivational Biases in Decision and Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 35(7), pages 1230-1251, July.
    33. James K. Hammitt & Alexander I. Shlyakhter, 1999. "The Expected Value of Information and the Probability of Surprise," Risk Analysis, John Wiley & Sons, vol. 19(1), pages 135-152, February.
    34. Timothy McDaniels & Tamsin Mills & Robin Gregory & Dan Ohlson, 2012. "Using Expert Judgments to Explore Robust Alternatives for Forest Management under Climate Change," Risk Analysis, John Wiley & Sons, vol. 32(12), pages 2098-2112, December.
    35. Kenneth Gillingham, Hao Deng, Ryan Wiser, Naim Darghouth, Gregory Nemet, Galen Barbose, Varun Rai, and Changgui Dong, 2016. "Deconstructing Solar Photovoltaic Pricing," The Energy Journal, International Association for Energy Economics, vol. 0(Number 3).
    36. Jesus Rios & David Rios Insua, 2012. "Adversarial Risk Analysis for Counterterrorism Modeling," Risk Analysis, John Wiley & Sons, vol. 32(5), pages 894-915, May.
    37. Baker, Erin & Keisler, Jeffrey M., 2011. "Cellulosic biofuels: Expert views on prospects for advancement," Energy, Elsevier, vol. 36(1), pages 595-605.
    38. David M. Reiner, 2015. "Where can I go to see one? Risk communications for an 'imaginary technology'," Journal of Risk Research, Taylor & Francis Journals, vol. 18(6), pages 710-713, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Giacomo Marangoni & Gauthier De Maere & Valentina Bosetti, 2017. "Optimal Clean Energy R&D Investments Under Uncertainty," MITP: Mitigation, Innovation and Transformation Pathways 256056, Fondazione Eni Enrico Mattei (FEEM).
    2. Laura Diaz Anadon & Erin Baker & Valentina Bosetti & Lara Aleluia Reis, 2016. "Expert views - and disagreements - about the potential of energy technology R&D," Climatic Change, Springer, vol. 136(3), pages 677-691, June.
    3. Elena Verdolini & Laura Díaz Anadón & Erin Baker & Valentina Bosetti & Lara Aleluia Reis, 2018. "Future Prospects for Energy Technologies: Insights from Expert Elicitations," Review of Environmental Economics and Policy, Association of Environmental and Resource Economists, vol. 12(1), pages 133-153.
    4. Prager, Fynnwin & Rhoads, Mohja & Martínez, Jose N., 2022. "The COVID-19 economic shutdown and the future of flexible workplace practices in the South Bay region of Los Angeles County," Transport Policy, Elsevier, vol. 125(C), pages 241-255.
    5. Steffen, Bjarne, 2020. "Estimating the cost of capital for renewable energy projects," Energy Economics, Elsevier, vol. 88(C).
    6. Small, Mitchell J. & Wong-Parodi, Gabrielle & Kefford, Benjamin M. & Stringer, Martin & Schmeda-Lopez, Diego R. & Greig, Chris & Ballinger, Benjamin & Wilson, Stephen & Smart, Simon, 2019. "Generating linked technology-socioeconomic scenarios for emerging energy transitions," Applied Energy, Elsevier, vol. 239(C), pages 1402-1423.
    7. Gruetzemacher, Ross & Paradice, David & Lee, Kang Bok, 2020. "Forecasting extreme labor displacement: A survey of AI practitioners," Technological Forecasting and Social Change, Elsevier, vol. 161(C).
    8. Franklyn Kanyako & Erin Baker, 2021. "Uncertainty analysis of the future cost of wind energy on climate change mitigation," Climatic Change, Springer, vol. 166(1), pages 1-17, May.
    9. Ohlendorf, Nils & Flachsland, Christian & Nemet, Gregory F. & Steckel, Jan Christoph, 2022. "Carbon price floors and low-carbon investment: A survey of German firms," Energy Policy, Elsevier, vol. 169(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Laura Diaz Anadon & Erin Baker & Valentina Bosetti & Lara Aleluia Reis, 2016. "Expert views - and disagreements - about the potential of energy technology R&D," Climatic Change, Springer, vol. 136(3), pages 677-691, June.
    2. Diaz Anadon, Laura & Bosetti, Valentina & Chan, Gabriel & Nemet, Gregory & Verdolini, Elena, 2014. "Energy Technology Expert Elicitations for Policy: Workshops, Modeling, and Meta-analysis," Working Paper Series rwp14-054, Harvard University, John F. Kennedy School of Government.
    3. Vicki Bier, 2020. "The Role of Decision Analysis in Risk Analysis: A Retrospective," Risk Analysis, John Wiley & Sons, vol. 40(S1), pages 2207-2217, November.
    4. Anca M. Hanea & Marissa F. McBride & Mark A. Burgman & Bonnie C. Wintle, 2018. "The Value of Performance Weights and Discussion in Aggregated Expert Judgments," Risk Analysis, John Wiley & Sons, vol. 38(9), pages 1781-1794, September.
    5. Elena Verdolini & Laura Díaz Anadón & Erin Baker & Valentina Bosetti & Lara Aleluia Reis, 2018. "Future Prospects for Energy Technologies: Insights from Expert Elicitations," Review of Environmental Economics and Policy, Association of Environmental and Resource Economists, vol. 12(1), pages 133-153.
    6. Laura Diaz Anadon & Erin Baker & Valentina Bosetti & Lara Aleluia Reis, 2016. "Too Early to Pick Winners: Disagreement across Experts Implies the Need to Diversify R&D Investment," Working Papers 2016.22, Fondazione Eni Enrico Mattei.
    7. Baker, Erin & Bosetti, Valentina & Salo, Ahti, 2016. "Finding Common Ground when Experts Disagree: Belief Dominance over Portfolios of Alternatives," MITP: Mitigation, Innovation and Transformation Pathways 243147, Fondazione Eni Enrico Mattei (FEEM).
    8. Erin Baker & Olaitan Olaleye & Lara Aleluia Reis, 2015. "Decision Frameworks and the Investment in R&D," Working Papers 2015.42, Fondazione Eni Enrico Mattei.
    9. Fiorese, Giulia & Catenacci, Michela & Bosetti, Valentina & Verdolini, Elena, 2014. "The power of biomass: Experts disclose the potential for success of bioenergy technologies," Energy Policy, Elsevier, vol. 65(C), pages 94-114.
    10. Catenacci, Michela & Verdolini, Elena & Bosetti, Valentina & Fiorese, Giulia, 2013. "Going electric: Expert survey on the future of battery technologies for electric vehicles," Energy Policy, Elsevier, vol. 61(C), pages 403-413.
    11. Stergios Athanassoglou & Valentina Bosetti, 2015. "Setting Environmental Policy When Experts Disagree," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 61(4), pages 497-516, August.
    12. Bosetti, Valentina & Catenacci, Michela & Fiorese, Giulia & Verdolini, Elena, 2012. "The future prospect of PV and CSP solar technologies: An expert elicitation survey," Energy Policy, Elsevier, vol. 49(C), pages 308-317.
    13. Baker, Erin & Bosetti, Valentina & Anadon, Laura Diaz & Henrion, Max & Aleluia Reis, Lara, 2015. "Future costs of key low-carbon energy technologies: Harmonization and aggregation of energy technology expert elicitation data," Energy Policy, Elsevier, vol. 80(C), pages 219-232.
    14. Bistline, John E., 2014. "Energy technology expert elicitations: An application to natural gas turbine efficiencies," Technological Forecasting and Social Change, Elsevier, vol. 86(C), pages 177-187.
    15. Nemet, Gregory F. & O’Shaughnessy, Eric & Wiser, Ryan & Darghouth, Naïm & Barbose, Galen & Gillingham, Ken & Rai, Varun, 2017. "Characteristics of low-priced solar PV systems in the U.S," Applied Energy, Elsevier, vol. 187(C), pages 501-513.
    16. Erin Baker & Valentina Bosetti & Karen E. Jenni & Elena Claire Ricci, 2014. "Facing the Experts: Survey Mode and Expert Elicitation," Working Papers 2014.01, Fondazione Eni Enrico Mattei.
    17. Verdolini, Elena & Anadon, Laura Diaz & Lu, Jiaqi & Nemet, Gregory F., 2015. "The effects of expert selection, elicitation design, and R&D assumptions on experts' estimates of the future costs of photovoltaics," Energy Policy, Elsevier, vol. 80(C), pages 233-243.
    18. Barron, Robert & McJeon, Haewon, 2015. "The differential impact of low-carbon technologies on climate change mitigation cost under a range of socioeconomic and climate policy scenarios," Energy Policy, Elsevier, vol. 80(C), pages 264-274.
    19. Fiorese, Giulia & Catenacci, Michela & Verdolini, Elena & Bosetti, Valentina, 2013. "Advanced biofuels: Future perspectives from an expert elicitation survey," Energy Policy, Elsevier, vol. 56(C), pages 293-311.
    20. Julia Reis & Julie Shortridge, 2020. "Impact of Uncertainty Parameter Distribution on Robust Decision Making Outcomes for Climate Change Adaptation under Deep Uncertainty," Risk Analysis, John Wiley & Sons, vol. 40(3), pages 494-511, March.

    More about this item

    Keywords

    Research and Development/Tech Change/Emerging Technologies;

    JEL classification:

    • O13 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development - - - Agriculture; Natural Resources; Environment; Other Primary Products
    • O14 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development - - - Industrialization; Manufacturing and Service Industries; Choice of Technology
    • Q4 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - Energy

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ags:feemmi:249349. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: AgEcon Search (email available below). General contact details of provider: https://edirc.repec.org/data/feemmit.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.