IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/31422.html
   My bibliography  Save this paper

Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology

Author

Listed:
  • Nikhil Agarwal
  • Alex Moehring
  • Pranav Rajpurkar
  • Tobias Salz

Abstract

Full automation using Artificial Intelligence (AI) predictions may not be optimal if humans have information not available to the AI (contextual information). We study human-AI collaboration using an information experiment with professional radiologists. Results show that providing (i) AI predictions does not improve performance on average, whereas (ii) contextual information does. Radiologists do not realize the gains from AI assistance because of errors in belief updating – they underweight AI predictions and treat their own information and AI predictions as statistically independent. Unless these mistakes can be corrected, the optimal human-AI collaboration design delegates cases either to humans or to AI, but rarely to AI assisted humans.

Suggested Citation

  • Nikhil Agarwal & Alex Moehring & Pranav Rajpurkar & Tobias Salz, 2023. "Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology," NBER Working Papers 31422, National Bureau of Economic Research, Inc.
    • Handle: RePEc:nbr:nberwo:31422
    Note: EH IO PR
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w31422.pdf
    Download Restriction: no
    ---><---

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Fouarge, Didier & Fregin, Marie-Christine & Janssen, Simon & Levels, Mark & Montizaan, Raymond & Özgül, Pelin & Rounding, Nicholas & Stops, Michael, 2025. "How AI-Augmented Training Improves Worker Productivity," IZA Discussion Papers 18224, Institute of Labor Economics (IZA).
    2. Felix Chopra & Ingar Haaland, 2023. "Conducting qualitative interviews with AI," CEBI working paper series 23-06, University of Copenhagen. Department of Economics. The Center for Economic Behavior and Inequality (CEBI).
    3. Martin Baily & David Byrne & Aidan Kane & Paul Soto, 2025. "Generative AI at the Crossroads: Light Bulb, Dynamo, or Microscope?," Papers 2505.14588, arXiv.org, revised Sep 2025.
    4. Foucart, Renaud & Zeng, Fanqi & Wang, Shidong, 2025. "The Social Importance of Being Stubborn When an Organization Meets AI," SocArXiv nfgy3_v1, Center for Open Science.
    5. Marie-Pascale Grimon & Christopher Mills, 2025. "Better Together? A Field Experiment on Human-Algorithm Interaction in Child Protection," Papers 2502.08501, arXiv.org, revised Aug 2025.
    6. Obidzinski, Marie & Oytana, Yves, 2024. "Artificial intelligence, inattention and liability rules," International Review of Law and Economics, Elsevier, vol. 79(C).
    7. L. Elisa Celis & Lingxiao Huang & Nisheeth K. Vishnoi, 2025. "A Mathematical Framework for AI-Human Integration in Work," Papers 2505.23432, arXiv.org, revised May 2025.
    8. Martin Neil Baily & David M. Byrne & Aidan T. Kane & Paul E. Soto, 2025. "Generative AI at the Crossroads: Light Bulb, Dynamo, or Microscope?," Finance and Economics Discussion Series 2025-053, Board of Governors of the Federal Reserve System (U.S.).
    9. Riccardo Zanardelli, 2025. "Navigating the safe harbor paradox in human-machine systems," Papers 2509.14057, arXiv.org, revised Jan 2026.
    10. Marie Obidzinski & Yves Oytana, 2025. "Advisory algorithms, automation bias and liability rules," Working Papers 2025-08, CRESE.
    11. Yuhao Fu & Nobuyuki Hanaki, 2024. "Do people rely on ChatGPT more than their peers to detect deepfake news?," ISER Discussion Paper 1233r, Institute of Social and Economic Research, The University of Osaka, revised Dec 2024.
    12. Andrei Iakovlev & Annie Liang, 2024. "The Value of Context: Human versus Black Box Evaluators," Papers 2402.11157, arXiv.org, revised Jun 2024.
    13. Tom Suhr & Samira Samadi & Chiara Farronato, 2024. "A Dynamic Model of Performative Human-ML Collaboration: Theory and Empirical Evidence," Papers 2405.13753, arXiv.org, revised Oct 2024.
    14. Bouacida, Elias & Foucart, Renaud & Jalloul, Maya, 2025. "When expert advice fails to reduce the productivity gap: Experimental evidence from chess players," Journal of Economic Behavior & Organization, Elsevier, vol. 236(C).
    15. R. Maria del Rio-Chanona & Ekkehard Ernst & Rossana Merola & Daniel Samaan & Ole Teutloff, 2025. "AI and jobs. A review of theory, estimates, and evidence," Papers 2509.15265, arXiv.org.
    16. Hanzhe Li & Jin Li & Ye Luo & Xiaowei Zhang, 2024. "AI Persuasion, Bayesian Attribution, and Career Concerns of Doctors," Papers 2410.01114, arXiv.org.
    17. Michele Fioretti & Junnan He & Jorge Tamayo, 2024. "Prices and Concentration: A U-Shape? Theory and Evidence from Renewables," Working Papers hal-04631762, HAL.
    18. David Almog & Romain Gauriot & Lionel Page & Daniel Martin, 2024. "AI Oversight and Human Mistakes: Evidence from Centre Court," Papers 2401.16754, arXiv.org, revised Feb 2025.
    19. Caplin, Andrew & Martin, Daniel & Marx, Philip, 2025. "Modeling machine learning: A cognitive economic approach," Journal of Economic Theory, Elsevier, vol. 224(C).
    20. Walter, Johannes & Biermann, Jan & Horton, John, 2024. "Advised by an Algorithm: Learning with Different Informational Resources," VfS Annual Conference 2024 (Berlin): Upcoming Labor Market Challenges 302407, Verein für Socialpolitik / German Economic Association.
    21. Huelden, Tobias & Jascisens, Vitalijs & Roemheld, Lars & Werner, Tobias, 2024. "Human-machine interactions in pricing: Evidence from two large-scale field experiments," DICE Discussion Papers 412, Heinrich Heine University Düsseldorf, Düsseldorf Institute for Competition Economics (DICE).
    22. Yang, Nanyin & Palma, Marco & Drichoutis, Andreas C., 2023. "Humanization of Virtual Assistants and Delegation Choices," MPRA Paper 119275, University Library of Munich, Germany.
    23. Jingyi Cui & Gabriel Dias & Justin Ye, 2025. "Signaling in the Age of AI: Evidence from Cover Letters," Papers 2509.25054, arXiv.org, revised Nov 2025.
    24. Hannes Ullrich & Michael Allan Ribers, 2023. "Machine predictions and human decisions with variation in payoffs and skill: the case of antibiotic prescribing," Berlin School of Economics Discussion Papers 0027, Berlin School of Economics.
    25. Chen, Qin & Ge, Jinfeng & Xie, Huaqing & Xu, Xingcheng & Yang, Yanqing, 2025. "Large language models at work in China’s labor market," China Economic Review, Elsevier, vol. 92(C).

    More about this item

    JEL classification:

    • C50 - Mathematical and Quantitative Methods - - Econometric Modeling - - - General
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • D47 - Microeconomics - - Market Structure, Pricing, and Design - - - Market Design
    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:31422. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.