IDEAS home Printed from https://ideas.repec.org/p/pdn/dispap/147.html
   My bibliography  Save this paper

Are numerical or verbal explanations of AI the key to appropriate user reliance and error detection? An experimental study with a classification algorithm

Author

Listed:
  • Jörg Papenkordt

    (Paderborn University)

  • Axel-Cyrille Ngonga Ngomo

    (Paderborn University)

  • Kirsten Thommes

    (Paderborn University)

Abstract

Advances in AI and our limited human capabilities have made AI decision-making opaque to humans. One prerequisite for enhancing the transparency of AI recommendations is improving AI explainability as humans need to be enabled to take responsibility for their actions even with AI support. Our study aims to tackle this issue by investigating two basic approaches to explainability: We evaluate numerical explanations, such as certainty measures, against verbal explanations, such as those provided by LLM as explanatory agents. Specifically, we examine whether verbal or numerical (or both) explanations in tasks of high uncertainty lure users into false beliefs or, on the contrary, promote appropriate reliance. Drawing on an experiment with 441 participants, we explore the dynamics of non-expert users' interactions with AI under varying explanatory conditions. Results show that explanations significantly improve reliance and decision accuracy. Numerical explanations aid in identifying uncertainties and errors, but the users' reliance on the advice falls far behind the given numerical certainty. Verbal explanations foster higher reliance while increasing the risk of over-reliance. Combining both explanation types enhances reliance but further amplifies blind trust in AI.

Suggested Citation

  • Jörg Papenkordt & Axel-Cyrille Ngonga Ngomo & Kirsten Thommes, 2025. "Are numerical or verbal explanations of AI the key to appropriate user reliance and error detection? An experimental study with a classification algorithm," Working Papers Dissertations 147, Paderborn University, Faculty of Business Administration and Economics.
    • Handle: RePEc:pdn:dispap:147
    as

    Download full text from publisher

    File URL: http://groups.uni-paderborn.de/wp-wiwi/RePEc/pdf/dispap/DP147.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Berkeley J. Dietvorst & Joseph P. Simmons & Cade Massey, 2018. "Overcoming Algorithm Aversion: People Will Use Imperfect Algorithms If They Can (Even Slightly) Modify Them," Management Science, INFORMS, vol. 64(3), pages 1155-1170, March.
    2. Bell, Andrew & Jones, Kelvyn, 2015. "Explaining Fixed Effects: Random Effects Modeling of Time-Series Cross-Sectional and Panel Data," Political Science Research and Methods, Cambridge University Press, vol. 3(1), pages 133-153, January.
    3. Thomas S. Wallsten & David V. Budescu & Rami Zwick, 1993. "Comparing the Calibration and Coherence of Numerical and Verbal Probability Judgments," Management Science, INFORMS, vol. 39(2), pages 176-190, February.
    4. Runshan Fu & Manmohan Aseri & Param Vir Singh & Kannan Srinivasan, 2022. "“Un”Fair Machine Learning Algorithms," Management Science, INFORMS, vol. 68(6), pages 4173-4195, June.
    5. Robert Mislavsky & Celia Gaertig, 2022. "Combining Probability Forecasts: 60% and 60% Is 60%, but Likely and Likely Is Very Likely," Management Science, INFORMS, vol. 68(1), pages 541-563, January.
    6. Isaac M. Lipkus, 2007. "Numeric, Verbal, and Visual Formats of Conveying Health Risks: Suggested Best Practices and Future Recommendations," Medical Decision Making, , vol. 27(5), pages 696-713, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gutiérrez-Romero, Roxana & Ahamed, Mostak, 2021. "COVID-19 response needs to broaden financial inclusion to curb the rise in poverty," World Development, Elsevier, vol. 138(C).
    2. Wu, Yu & Mullan, Katrina & Biggs, Trent & Caviglia-Harris, Jill L. & Harris, Daniel & Sills, Erin O., "undated". "Do Forests Provide Watershed Services to Local Populations in the Humid Tropics? Evidence from the Brazilian Amazon," 2018 Annual Meeting, August 5-7, Washington, D.C. 274012, Agricultural and Applied Economics Association.
    3. Kasraian, Dena & Maat, Kees & van Wee, Bert, 2018. "Urban developments and daily travel distances: Fixed, random and hybrid effects models using a Dutch pseudo-panel over three decades," Journal of Transport Geography, Elsevier, vol. 72(C), pages 228-236.
    4. Bruno S. Frey & Anthony Gullo, 2021. "Does Sports Make People Happier, or Do Happy People More Sports?," Journal of Sports Economics, , vol. 22(4), pages 432-458, May.
    5. Panarello, Demetrio, 2021. "Economic insecurity, conservatism, and the crisis of environmentalism: 30 years of evidence," Socio-Economic Planning Sciences, Elsevier, vol. 73(C).
    6. Arjan H Schakel & A J Brown, 2022. "Dissecting Public Opinion on Regional Authority: Four Types of Regionalists Based on Citizens’ Preferences for Self-Rule and Shared Rule," Publius: The Journal of Federalism, CSF Associates Inc., vol. 52(2), pages 310-328.
    7. Ryan H. Murphy & Colin O’Reilly, 2023. "Freedom through taxation: the effect of fiscal capacity on the rule of law," European Journal of Law and Economics, Springer, vol. 56(1), pages 69-90, August.
    8. Dimitris Bertsimas & Agni Orfanoudaki, 2021. "Algorithmic Insurance," Papers 2106.00839, arXiv.org, revised Dec 2022.
    9. Jan Hoffmann & Naima Saeed & Sigbjørn Sødal, 2020. "Liner shipping bilateral connectivity and its impact on South Africa’s bilateral trade flows," Maritime Economics & Logistics, Palgrave Macmillan;International Association of Maritime Economists (IAME), vol. 22(3), pages 473-499, September.
    10. Bryce McLaughlin & Jann Spiess, 2022. "Algorithmic Assistance with Recommendation-Dependent Preferences," Papers 2208.07626, arXiv.org, revised Oct 2025.
    11. Markus Jung & Mischa Seiter, 2021. "Towards a better understanding on mitigating algorithm aversion in forecasting: an experimental study," Journal of Management Control: Zeitschrift für Planung und Unternehmenssteuerung, Springer, vol. 32(4), pages 495-516, December.
    12. Tse, Tiffany Tsz Kwan & Hanaki, Nobuyuki & Mao, Bolin, 2024. "Beware the performance of an algorithm before relying on it: Evidence from a stock price forecasting experiment," Journal of Economic Psychology, Elsevier, vol. 102(C).
    13. Julien Hanoteau, 2023. "Do foreign MNEs alleviate multidimensional poverty in developing countries?," Eurasian Business Review, Springer;Eurasia Business and Economics Society, vol. 13(4), pages 719-749, December.
    14. Elisa Barbieri & Manli Huang & Shenglei Pi & Mattia Tassinari, 2017. "Restructuring the Production of Medicines: An Investigation on the Pharmaceutical Sector in China and the Role of Mergers and Acquisitions," IJERPH, MDPI, vol. 14(10), pages 1-21, October.
    15. Kohei Kawaguchi, 2021. "When Will Workers Follow an Algorithm? A Field Experiment with a Retail Business," Management Science, INFORMS, vol. 67(3), pages 1670-1695, March.
    16. Vial, Virginie & Hanoteau, Julien, 2015. "Returns to Micro-Entrepreneurship in an Emerging Economy: A Quantile Study of Entrepreneurial Indonesian Households’ Welfare," World Development, Elsevier, vol. 74(C), pages 142-157.
    17. David V. Budescu & Timothy R. Johnson, 2011. "A model-based approach for the analysis of the calibration of probability judgments," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 6(8), pages 857-869, December.
    18. Isabel Neira & Fernando Bruna & Marta Portela & Adela García-Aracil, 2018. "Individual Well-Being, Geographical Heterogeneity and Social Capital," Journal of Happiness Studies, Springer, vol. 19(4), pages 1067-1090, April.
    19. Damien Rousselière, 2019. "A Flexible Approach to Age Dependence in Organizational Mortality: Comparing the Life Duration for Cooperative and Non-Cooperative Enterprises Using a Bayesian Generalized Additive Discrete Time Survi," Journal of Quantitative Economics, Springer;The Indian Econometric Society (TIES), vol. 17(4), pages 829-855, December.
    20. T.E. Uberti & M.A. Maggioni & E. Marrocu & S. Usai, 2023. "The role of localised, recombinant and exogenous technological change in European regions," Working Paper CRENoS 202303, Centre for North South Economic Research, University of Cagliari and Sassari, Sardinia.

    More about this item

    Keywords

    ;
    ;
    ;

    JEL classification:

    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • D81 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Criteria for Decision-Making under Risk and Uncertainty
    • C88 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Other Computer Software
    • O33 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Technological Change: Choices and Consequences; Diffusion Processes

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pdn:dispap:147. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: WP-WiWi-Info (email available below). General contact details of provider: https://edirc.repec.org/data/fwpadde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.