IDEAS home Printed from https://ideas.repec.org/a/eee/tefoso/v219y2025ics0040162525002963.html

Unlocking the black box: Enhancing human-AI collaboration in high-stakes healthcare scenarios through explainable AI

Author

Listed:
  • Hassan, Reda
  • Nguyen, Nhien
  • Finserås, Stine Rasdal
  • Adde, Lars
  • Strümke, Inga
  • Støen, Ragnhild

Abstract

Despite the advanced predictive capabilities of artificial intelligence (AI) systems, their inherent opacity often leaves users confused about the rationale behind their outputs. We investigate the challenge of AI opacity, which undermines user trust and the effectiveness of clinical judgment in healthcare. We demonstrate how human experts make judgment in high-stakes scenarios where their judgment diverges from AI predictions, emphasizing the need for explainability to enhance clinical judgment and trust in AI systems. We used a scenario-based methodology, conducting 28 semi-structured interviews and observations with clinicians from Norway and Egypt. Our analysis revealed that, during the process of forming judgments, human experts engage in AI interrogation practices when faced with opaque AI systems. Obtaining explainability from AI systems leads to increased interrogation practices aimed at gaining a deeper understanding of AI predictions. With the introduction of explainable AI (XAI), experts demonstrate greater trust in the AI system, show a readiness to learn from AI, and may reconsider or update their initial judgments when they contradict AI predictions.

Suggested Citation

  • Hassan, Reda & Nguyen, Nhien & Finserås, Stine Rasdal & Adde, Lars & Strümke, Inga & Støen, Ragnhild, 2025. "Unlocking the black box: Enhancing human-AI collaboration in high-stakes healthcare scenarios through explainable AI," Technological Forecasting and Social Change, Elsevier, vol. 219(C).
    • Handle: RePEc:eee:tefoso:v:219:y:2025:i:c:s0040162525002963
    DOI: 10.1016/j.techfore.2025.124265
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0040162525002963
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.techfore.2025.124265?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Natalia Levina, 2005. "Collaborating on Multiparty Information Systems Development Projects: A Collective Reflection-in-Action View," Information Systems Research, INFORMS, vol. 16(2), pages 109-130, June.
    2. Haque, AKM Bahalul & Islam, A.K.M. Najmul & Mikalef, Patrick, 2023. "Explainable Artificial Intelligence (XAI) from a user perspective: A synthesis of prior literature and problematizing avenues for future research," Technological Forecasting and Social Change, Elsevier, vol. 186(PA).
    3. Nripendra P. Rana & Sheshadri Chatterjee & Yogesh K. Dwivedi & Shahriar Akter, 2022. "Understanding dark side of artificial intelligence (AI) integrated business analytics: assessing firm’s operational inefficiency and competitiveness," European Journal of Information Systems, Taylor & Francis Journals, vol. 31(3), pages 364-387, May.
    4. Ryan Allen & Prithwiraj (Raj) Choudhury, 2022. "Algorithm-Augmented Work and Domain Experience: The Countervailing Forces of Ability and Aversion," Organization Science, INFORMS, vol. 33(1), pages 149-169, January.
    5. Paul R. Carlile, 2004. "Transferring, Translating, and Transforming: An Integrative Framework for Managing Knowledge Across Boundaries," Organization Science, INFORMS, vol. 15(5), pages 555-568, October.
    6. Niyi Ogunbiyi & Artie Basukoski & Thierry Chaussalet, 2021. "An Exploration of Ethical Decision Making with Intelligence Augmentation," Social Sciences, MDPI, vol. 10(2), pages 1-14, February.
    7. Kevin Bauer & Moritz von Zahn & Oliver Hinz, 2023. "Expl(AI)ned: The Impact of Explainable Artificial Intelligence on Users’ Information Processing," Information Systems Research, INFORMS, vol. 34(4), pages 1582-1602, December.
    8. Gillner, Sandra, 2024. "We're implementing AI now, so why not ask us what to do? – How AI providers perceive and navigate the spread of diagnostic AI in complex healthcare systems," Social Science & Medicine, Elsevier, vol. 340(C).
    9. Logg, Jennifer M. & Minson, Julia A. & Moore, Don A., 2019. "Algorithm appreciation: People prefer algorithmic to human judgment," Organizational Behavior and Human Decision Processes, Elsevier, vol. 151(C), pages 90-103.
    10. Sarah Lebovitz & Hila Lifshitz-Assaf & Natalia Levina, 2022. "To Engage or Not to Engage with AI for Critical Judgments: How Professionals Deal with Opacity When Using AI for Medical Diagnosis," Organization Science, INFORMS, vol. 33(1), pages 126-148, January.
    11. Meyer, Lea Mareen & Stead, Susan & Salge, Torsten Oliver & Antons, David, 2024. "Artificial intelligence in acute care: A systematic review, conceptual synthesis, and research agenda," Technological Forecasting and Social Change, Elsevier, vol. 206(C).
    12. Daniel A. Levinthal & James G. March, 1993. "The myopia of learning," Strategic Management Journal, Wiley Blackwell, vol. 14(S2), pages 95-112, December.
    13. Véronique Ambrosini & Cliff Bowman, 2001. "Tacit Knowledge: Some Suggestions for Operationalization," Journal of Management Studies, Wiley Blackwell, vol. 38(6), pages 811-829, September.
    14. Kim, Doha & Song, Yeosol & Kim, Songyie & Lee, Sewang & Wu, Yanqin & Shin, Jungwoo & Lee, Daeho, 2023. "How should the results of artificial intelligence be explained to users? - Research on consumer preferences in user-centered explainable artificial intelligence," Technological Forecasting and Social Change, Elsevier, vol. 188(C).
    15. Lauren Waardenburg & Marleen Huysman & Anastasia V. Sergeeva, 2022. "In the Land of the Blind, the One-Eyed Man Is King: Knowledge Brokerage in the Age of Learning Algorithms," Organization Science, INFORMS, vol. 33(1), pages 59-82, January.
    16. Berkeley J. Dietvorst & Joseph P. Simmons & Cade Massey, 2018. "Overcoming Algorithm Aversion: People Will Use Imperfect Algorithms If They Can (Even Slightly) Modify Them," Management Science, INFORMS, vol. 64(3), pages 1155-1170, March.
    17. Kevin Bauer & Oliver Hinz & Wil Aalst & Christof Weinhardt, 2021. "Expl(AI)n It to Me – Explainable AI and Information Systems Research," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(2), pages 79-82, April.
    18. Wang, Weisha & Wang, Yichuan & Chen, Long & Ma, Rui & Zhang, Minhao, 2024. "Justice at the Forefront: Cultivating felt accountability towards Artificial Intelligence among healthcare professionals," Social Science & Medicine, Elsevier, vol. 347(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ben Greiner & Philipp Grünwald & Thomas Lindner & Georg Lintner & Martin Wiernsperger, 2026. "Incentives, Framing, and Reliance on Algorithmic Advice: An Experimental Study," Management Science, INFORMS, vol. 72(1), pages 302-322, January.
    2. Hongchang Wang & Yingjie Zhang & Tian Lu, 2026. "The Power of Disagreement: A Field Experiment to Investigate Human–Algorithm Collaboration in Loan Evaluations," Management Science, INFORMS, vol. 72(1), pages 96-118, January.
    3. Sebastian Krakowski & Darek Haftor & Johannes Luger & Natallia Pashkevich & Sebastian Raisch, 2026. "Human-Centered Artificial Intelligence: A Field Experiment," Management Science, INFORMS, vol. 72(1), pages 57-72, January.
    4. Mancuso, Ilaria & Petruzzelli, Antonio Messeni & Panniello, Umberto & Frattini, Federico, 2025. "The role of explainable artificial intelligence (XAI) in innovation processes: a knowledge management perspective," Technology in Society, Elsevier, vol. 82(C).
    5. Sarah Lebovitz & Hila Lifshitz-Assaf & Natalia Levina, 2022. "To Engage or Not to Engage with AI for Critical Judgments: How Professionals Deal with Opacity When Using AI for Medical Diagnosis," Organization Science, INFORMS, vol. 33(1), pages 126-148, January.
    6. Jiamin Yin & Kee Yuan Ngiam & Sharon Swee-Lin Tan & Hock Hai Teo, 2025. "Designing AI-Based Work Processes: How the Timing of AI Advice Affects Diagnostic Decision Making," Management Science, INFORMS, vol. 71(11), pages 9361-9383, November.
    7. Shirish Sundaresan & Isin Guler, 2025. "Algorithmic Recommendation Tools and Experiential Learning in Clinical Care," Organization Science, INFORMS, vol. 36(5), pages 1786-1802, September.
    8. Khosrowabadi, Naghmeh & Hoberg, Kai & Lee, Yun Shin, 2025. "Guiding supervisors in artificial intelligence-enabled forecasting: Understanding the impacts of salience and detail on decision-making," International Journal of Forecasting, Elsevier, vol. 41(2), pages 716-732.
    9. Ilanit SimanTov-Nachlieli, 2025. "More to Lose: The Adverse Effect of High Performance Ranking on Employees’ Preimplementation Attitudes Toward the Integration of Powerful AI Aids," Organization Science, INFORMS, vol. 36(1), pages 1-20, January.
    10. Ding, Zhenbin & Zhang, Yali & Sun, Jun & Goh, Mark & Yang, Zhaojun, 2025. "To reveal or conceal: AI identity disclosure strategies for merchants," International Journal of Production Economics, Elsevier, vol. 283(C).
    11. Tian Lu & Yingjie Zhang, 2025. "1 + 1 > 2? Information, Humans, and Machines," Information Systems Research, INFORMS, vol. 36(1), pages 394-418, March.
    12. Meng Liu & Xiaocheng Tang & Siyuan Xia & Shuo Zhang & Yuting Zhu & Qianying Meng, 2026. "Algorithm Aversion: Evidence from Ridesharing Drivers," Management Science, INFORMS, vol. 72(1), pages 193-203, January.
    13. Kevin Bauer & Moritz von Zahn & Oliver Hinz, 2023. "Expl(AI)ned: The Impact of Explainable Artificial Intelligence on Users’ Information Processing," Information Systems Research, INFORMS, vol. 34(4), pages 1582-1602, December.
    14. Hamsa Bastani & Osbert Bastani & Wichinpong Park Sinchaisri, 2026. "Improving Human Sequential Decision Making with Reinforcement Learning," Management Science, INFORMS, vol. 72(1), pages 733-755, January.
    15. Samuel N. Kirshner, 2025. "Psychological Distance and Algorithm Aversion: Congruency and Advisor Confidence," Service Science, INFORMS, vol. 17(2-3), pages 74-91, June.
    16. Mahmud, Hasan & Islam, A.K.M. Najmul & Ahmed, Syed Ishtiaque & Smolander, Kari, 2022. "What influences algorithmic decision-making? A systematic literature review on algorithm aversion," Technological Forecasting and Social Change, Elsevier, vol. 175(C).
    17. Felipe A. Csaszar & Harsh Ketkar & Hyunjin Kim, 2024. "Artificial Intelligence and Strategic Decision-Making: Evidence from Entrepreneurs and Investors," Strategy Science, INFORMS, vol. 9(4), pages 322-345, December.
    18. Bryce McLaughlin & Jann Spiess, 2022. "Algorithmic Assistance with Recommendation-Dependent Preferences," Papers 2208.07626, arXiv.org, revised Oct 2025.
    19. Markus Jung & Mischa Seiter, 2021. "Towards a better understanding on mitigating algorithm aversion in forecasting: an experimental study," Journal of Management Control: Zeitschrift für Planung und Unternehmenssteuerung, Springer, vol. 32(4), pages 495-516, December.
    20. Zhu, Yimin & Zhang, Jiemin & Wu, Jifei & Liu, Yingyue, 2022. "AI is better when I'm sure: The influence of certainty of needs on consumers' acceptance of AI chatbots," Journal of Business Research, Elsevier, vol. 150(C), pages 642-652.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:tefoso:v:219:y:2025:i:c:s0040162525002963. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.sciencedirect.com/science/journal/00401625 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.