IDEAS home Printed from https://ideas.repec.org/a/inm/ormnsc/v72y2026i1p73-95.html

Engaging Customers with AI in Online Chats: Evidence from a Randomized Field Experiment

Author

Listed:
  • Shunyuan Zhang

    (Marketing Department, Harvard Business School, Boston, Massachusetts 02163)

  • Das Narayandas

    (Marketing Department, Harvard Business School, Boston, Massachusetts 02163)

Abstract

We examine how artificial intelligence (AI) affected the productivity of customer service agents and customer sentiment in online interactions. Collaborating with a meal delivery company, we conducted a randomized field experiment that exploited exogenous variation in giving agents access to AI-generated suggestions. We found that AI improved both the efficiency and effectiveness of the interactions: AI-assisted agents responded faster, engaged customers more deeply, and achieved greater improvements in customer sentiment. The benefits were most pronounced for less-experienced agents. However, AI’s impact varied by conversation type: It improved efficiency and customer sentiment in subscription cancellation requests but was the least effective in repeat complaint scenarios because of systemic issues beyond the AI’s capability. A text analysis of agent messages suggests that improved customer sentiment was explained by AI-assisted agents exhibiting higher levels of key response characteristics: empathy, information, and solution. Furthermore, we exploit a unique data feature: Customers first chatted with an automated chatbot without any human intervention before they were transferred to human agents (who may or may not have had AI assistance). We found that if customers who had experienced chatbot comprehension failures were then connected to AI-assisted human agents, the involvement of AI negatively affected customer sentiment. This is because unusually rapid responses in the latter scenario led customers to believe they were still communicating with a chatbot only, suggesting a spillover from their initial negative chatbot experiences. Companies should understand the conversation contexts, such as customer intent and chatbot interactions, when integrating AI into their customer support strategies.

Suggested Citation

  • Shunyuan Zhang & Das Narayandas, 2026. "Engaging Customers with AI in Online Chats: Evidence from a Randomized Field Experiment," Management Science, INFORMS, vol. 72(1), pages 73-95, January.
    • Handle: RePEc:inm:ormnsc:v:72:y:2026:i:1:p:73-95
    DOI: 10.1287/mnsc.2022.03920
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mnsc.2022.03920
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mnsc.2022.03920?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Clément de Chaisemartin & Xavier D'Haultfœuille, 2020. "Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects," American Economic Review, American Economic Association, vol. 110(9), pages 2964-2996, September.
    2. Agrawal, Ajay & Gans, Joshua S. & Goldfarb, Avi, 2019. "Exploring the impact of artificial Intelligence: Prediction versus judgment," Information Economics and Policy, Elsevier, vol. 47(C), pages 1-6.
    3. Kirill Borusyak & Xavier Jaravel & Jann Spiess, 2024. "Revisiting Event-Study Designs: Robust and Efficient Estimation," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 91(6), pages 3253-3285.
    4. Shahidur R. Khandker & Gayatri B. Koolwal & Hussain A. Samad, 2010. "Handbook on Impact Evaluation : Quantitative Methods and Practices," World Bank Publications - Books, The World Bank Group, number 2693, April.
    5. Allen, Ryan T. & Choudhury, Prithwiraj, 2021. "Algorithm-augmented work and domain experience: the countervailing forces of ability and aversion," LSE Research Online Documents on Economics 128784, London School of Economics and Political Science, LSE Library.
    6. Shunyuan Zhang & Nitin Mehta & Param Vir Singh & Kannan Srinivasan, 2021. "Frontiers: Can an Artificial Intelligence Algorithm Mitigate Racial Economic Inequality? An Analysis in the Context of Airbnb," Marketing Science, INFORMS, vol. 40(5), pages 813-820, September.
    7. Xueming Luo & Siliang Tong & Zheng Fang & Zhe Qu, 2019. "Frontiers: Machines vs. Humans: The Impact of Artificial Intelligence Chatbot Disclosure on Customer Purchases," Marketing Science, INFORMS, vol. 38(6), pages 937-947, November.
    8. Prithwiraj Choudhury & Evan Starr & Rajshree Agarwal, 2020. "Machine learning and human capital complementarities: Experimental evidence on bias mitigation," Strategic Management Journal, Wiley Blackwell, vol. 41(8), pages 1381-1411, August.
    9. Robert Seamans & Feng Zhu, 2014. "Responses to Entry in Multi-Sided Markets: The Impact of Craigslist on Local Newspapers," Management Science, INFORMS, vol. 60(2), pages 476-493, February.
    10. Goodman-Bacon, Andrew, 2021. "Difference-in-differences with variation in treatment timing," Journal of Econometrics, Elsevier, vol. 225(2), pages 254-277.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bernd Irlenbusch, 2026. "Human Trust in AI: Evidence from Experimental Economics," ECONtribute Discussion Papers Series 417, University of Bonn and University of Cologne, Germany.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hahm, Dong Woo, 2026. "From curriculum to career: Early-career labor market effects of the Common Core," Economics of Education Review, Elsevier, vol. 110(C).
    2. Shipeng Yan & Wei Jiang & Yue Xu, 2026. "Global investors, hidden suppliers: how institutions shape the impact of stock market liberalization programs on corporate responsibility," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 57(2), pages 197-219, March.
    3. Xu, Kefan & Yuan, Peng & Yu, Renjie, 2025. "The effects of mixed ownership reforms on Chinese firms’ emissions," Economic Analysis and Policy, Elsevier, vol. 86(C), pages 191-209.
    4. Gupta, Tanu & Khan, Md. Tajuddin & Negi, Digvijay Singh, 2025. "Agriculture, electrification and gendered time use in rural Bangladesh," Energy Economics, Elsevier, vol. 150(C).
    5. Brehm, Johannes & Pestel, Nico & Schaffner, Sandra & Schmitz, Laura, 2025. "From Low Emission Zone to academic track: Environmental policy effects on educational achievement in elementary school," Journal of Environmental Economics and Management, Elsevier, vol. 132(C).
    6. Calderón, Mariana & Cortés, Josué & Pérez Pérez, Jorge & Salcedo, Alejandrina, 2023. "Disentangling the Effects of Large Minimum Wage and VAT Changes on Prices: Evidence from Mexico," Labour Economics, Elsevier, vol. 80(C).
    7. Hu, Changyu & Zhao, Qicheng & Ma, Ruiyun & Yin, Yugang, 2025. "Civil recognition and urban entrepreneurship: Evidence from China," Economic Analysis and Policy, Elsevier, vol. 87(C), pages 1395-1414.
    8. Coraggio, Luca & Pagano, Marco & Scognamiglio, Annalisa & Tåg, Joacim, 2025. "JAQ of all trades: Job mismatch, firm productivity and managerial quality," Journal of Financial Economics, Elsevier, vol. 164(C).
    9. Burton, Anne M. & Churchill, Brandyn F., 2025. "Supply-side opioid restrictions and the retail pharmacy market," Journal of Health Economics, Elsevier, vol. 104(C).
    10. Martin Huber & Sarina Joy Oberhansli, 2026. "Difference-in-differences for mediation analysis using double machine learning," Papers 2602.23877, arXiv.org.
    11. Amarasinghe, Ashani & Raschky, Paul A. & Zenou, Yves & Zhou, Junjie, 2026. "How natural disasters spread conflict," European Economic Review, Elsevier, vol. 181(C).
    12. Prem, Mounu & Purroy, Miguel E. & Vargas, Juan F., 2025. "Landmines: The local effects of demining," Journal of Public Economics, Elsevier, vol. 247(C).
    13. Padilla-Romo, María & Peluffo, Cecilia & Viollaz, Mariana, 2025. "Parents’ effective time endowment and divorce: Evidence from extended school days," Journal of Public Economics, Elsevier, vol. 242(C).
    14. Huang, Wei & Luo, Mi & Ta, Yuqi & Wang, Boxian, 2024. "Land expropriation, household behaviors, and health outcomes: Evidence from China," Journal of Development Economics, Elsevier, vol. 171(C).
    15. Li, Pei & Liu, Kaihao & Lu, Yi & Peng, Lu, 2025. "Organizing regulatory structure and local air quality: Evidence from the environmental vertical management reform in China," Journal of Comparative Economics, Elsevier, vol. 53(1), pages 139-164.
    16. Shibashish Mukherjee & Sorin M.S. Krammer, 2024. "When the going gets tough : Board gender diversity in the wake of a major crisis," Post-Print hal-04522722, HAL.
    17. de Assis, Dércio & Ghosh, Arpita & Oreffice, Sonia & Quintana-Domeque, Climent, 2025. "Non-Fatal Strangulation Laws and Intimate Partner Homicides," IZA Discussion Papers 18006, IZA Network @ LISER.
    18. Kang, Yankun & Liu, Ruiming & Yang, Bingyan, 2025. "Bridging the ivory tower and industry: How university science parks promote university-industry collaboration?," Research Policy, Elsevier, vol. 54(6).
    19. Umblijs, Janis & Hermansen, Are Skeie, 2025. "Can A New Name Open Closed Doors? Foreign-Sounding Names and Immigrant Earnings," SocArXiv rd3gv_v1, Center for Open Science.
    20. Jaworski, Krystian & Olipra, Jakub, 2025. "Cutting VAT rate on food products in a high-inflation environment. Does it work out?," Food Policy, Elsevier, vol. 131(C).

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:72:y:2026:i:1:p:73-95. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.