IDEAS home Printed from https://ideas.repec.org/a/eee/jbrese/v190y2025ics0148296324006623.html
   My bibliography  Save this article

Language translation effects in Chatbots: Evidence from a randomized field experiment on a mobile commerce platform

Author

Listed:
  • Nayak, Ashutosh
  • Nair, Ashwin Aravindakshan

Abstract

This study investigates the impact of language translation innovations in artificial intelligence (AI) digital assistants. We use data from a mobile com- merce platform application in India that introduced a Hindi and English version of its previously English-only language chatbot. The data, obtained from a randomized field experiment conducted on new users, help determine the impact of introducing language translation in conversational chatbots by quantifying its effect on user metrics such as purchases and uninstalls. In the experiment, the firm only altered the language of interaction, from English- only to Hindi and English. The firm did not make any other changes in the design or purchase flow within the application. We find that language trans- lation innovations significantly increase the number of user sessions and also improve user purchases and engagement. The increase in the engagement did not emerge from an increase in the number of sessions but from an increase in interactions within a session in the bilingual app. We also observe a sharp rise in uninstalls for the population that received the bilingual app. We find that in the bilingual chatbot, uninstalls rise with increased user interactions in a high involvement product category. In sum, the results from the field experiment show that while language translation in artificially intelligent as- sistants leads to greater purchases, it could also lead to increased uninstalls. This result suggests that implementing similar language translation innova- tions in isolation, without any modifications to in-application experience, has the potential to yield negative outcomes for the firm.

Suggested Citation

  • Nayak, Ashutosh & Nair, Ashwin Aravindakshan, 2025. "Language translation effects in Chatbots: Evidence from a randomized field experiment on a mobile commerce platform," Journal of Business Research, Elsevier, vol. 190(C).
    • Handle: RePEc:eee:jbrese:v:190:y:2025:i:c:s0148296324006623
    DOI: 10.1016/j.jbusres.2024.115158
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0148296324006623
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.jbusres.2024.115158?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Stefan Wager & Susan Athey, 2018. "Estimation and Inference of Heterogeneous Treatment Effects using Random Forests," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(523), pages 1228-1242, July.
    2. Koslow, Scott & Shamdasani, Prem N & Touchstone, Ellen E, 1994. "Exploring Language Effects in Ethnic Advertising: A Sociolinguistic Perspective," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 20(4), pages 575-585, March.
    3. Luna, David & Peracchio, Laura A, 2001. "Moderators of Language Effects in Advertising to Bilinguals: A Psycholinguistic Approach," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 28(2), pages 284-295, September.
    4. Erik Brynjolfsson & Xiang Hui & Meng Liu, 2019. "Does Machine Translation Affect International Trade? Evidence from a Large Digital Platform," Management Science, INFORMS, vol. 65(12), pages 5449-5460, December.
    5. Erik Brynjolfsson & Daniel Rock & Chad Syverson, 2018. "Artificial Intelligence and the Modern Productivity Paradox: A Clash of Expectations and Statistics," NBER Chapters, in: The Economics of Artificial Intelligence: An Agenda, pages 23-57, National Bureau of Economic Research, Inc.
    6. Mariani, Marcello M. & Hashemi, Novin & Wirtz, Jochen, 2023. "Artificial intelligence empowered conversational agents: A systematic literature review and research agenda," Journal of Business Research, Elsevier, vol. 161(C).
    7. Sun, Yuan & Chen, Jin & Sundar, S. Shyam, 2024. "Chatbot ads with a human touch: A test of anthropomorphism, interactivity, and narrativity," Journal of Business Research, Elsevier, vol. 172(C).
    8. Ascarza, & Neslin, & Netzer, & Lemmens, Aurélie & Anderson, Zachery & Fader, Peter S. & Gupta, S. & Hardie, B.G.S. & Libai, Barak & Neal, David & Provost, Foster, 2018. "In pursuit of enhanced customer retention management : Review, key issues, and future directions," Other publications TiSEM 28a90d28-6daf-42f1-bd8e-e, Tilburg University, School of Economics and Management.
    9. Juquelier, Antoine & Poncin, Ingrid & Hazée, Simon, 2025. "Empathic chatbots: A double-edged sword in customer experiences," Journal of Business Research, Elsevier, vol. 188(C).
    10. Aradhna Krishna & Rohini Ahluwalia, 2008. "Language Choice in Advertising to Bilinguals: Asymmetric Effects for Multinationals versus Local Firms," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 35(4), pages 692-705, August.
    11. Chakraborty, Debarun & Kumar Kar, Arpan & Patre, Smruti & Gupta, Shivam, 2024. "Enhancing trust in online grocery shopping through generative AI chatbots," Journal of Business Research, Elsevier, vol. 180(C).
    12. Ping Jin & Humza Haider & Russell Greiner & Sarah Wei & Gerald Häubl, 2021. "Using survival prediction techniques to learn consumer-specific reservation price distributions," PLOS ONE, Public Library of Science, vol. 16(4), pages 1-26, April.
    13. Seetharaman, P B & Chintagunta, Pradeep K, 2003. "The Proportional Hazard Model for Purchase Timing: A Comparison of Alternative Specifications," Journal of Business & Economic Statistics, American Statistical Association, vol. 21(3), pages 368-382, July.
    14. Gabriela Gutierrez-Huerter O & Jeremy Moon & Stefan Gold & Wendy Chapple, 2020. "Micro-processes of translation in the transfer of practices from MNE headquarters to foreign subsidiaries: The role of subsidiary translators," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 51(3), pages 389-413, April.
    15. Viswanath Venkatesh & Fred D. Davis, 2000. "A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies," Management Science, INFORMS, vol. 46(2), pages 186-204, February.
    16. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, Enero-Abr.
    17. Rutz, Oliver & Aravindakshan, Ashwin & Rubel, Olivier, 2019. "Measuring and forecasting mobile game app engagement," International Journal of Research in Marketing, Elsevier, vol. 36(2), pages 185-199.
    18. David Luna & Laura A. Peracchio, 2005. "Advertising to Bilingual Consumers: The Impact of Code-Switching on Persuasion," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 31(4), pages 760-765, March.
    19. Eva Ascarza & Scott A. Neslin & Oded Netzer & Zachery Anderson & Peter S. Fader & Sunil Gupta & Bruce G. S. Hardie & Aurélie Lemmens & Barak Libai & David Neal & Foster Provost & Rom Schrift, 2018. "In Pursuit of Enhanced Customer Retention Management: Review, Key Issues, and Future Directions," Customer Needs and Solutions, Springer;Institute for Sustainable Innovation and Growth (iSIG), vol. 5(1), pages 65-81, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alvarez, Cecilia M.O. & Miniard, Paul W. & Jaccard, James, 2017. "How Hispanic bilinguals' cultural stereotypes shape advertising persuasiveness," Journal of Business Research, Elsevier, vol. 75(C), pages 29-36.
    2. Jos Hornikx & Frank Meurs & Helene Tenzer, 2024. "Foreign languages in advertising: Theoretical implications for language-related IB research," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 55(2), pages 270-279, March.
    3. Aurélie Lemmens & Sunil Gupta, 2020. "Managing Churn to Maximize Profits," Marketing Science, INFORMS, vol. 39(5), pages 956-973, September.
    4. Kubat, Umut & Swaminathan, Vanitha, 2015. "Crossing the cultural divide through bilingual advertising: The moderating role of brand cultural symbolism," International Journal of Research in Marketing, Elsevier, vol. 32(4), pages 354-362.
    5. Huda Khan & Richard Lee, 2020. "A sociolinguistic perspective of the effects of packaging in bilingual markets," Journal of Brand Management, Palgrave Macmillan, vol. 27(2), pages 130-142, March.
    6. Holmqvist, Jonas & Van Vaerenbergh, Yves & Lunardo, Renaud & Dahlén, Micael, 2019. "The Language Backfire Effect: How Frontline Employees Decrease Customer Satisfaction through Language Use," Journal of Retailing, Elsevier, vol. 95(2), pages 115-129.
    7. Dwight R. Merunka, 2013. "Reinterpreting cultural priming effects in cross-cultural consumer research," AMS Review, Springer;Academy of Marketing Science, vol. 3(4), pages 232-248, December.
    8. Dimitris Bertsimas & Agni Orfanoudaki & Rory B. Weiner, 2020. "Personalized treatment for coronary artery disease patients: a machine learning approach," Health Care Management Science, Springer, vol. 23(4), pages 482-506, December.
    9. Davide Viviano & Jelena Bradic, 2019. "Synthetic learner: model-free inference on treatments over time," Papers 1904.01490, arXiv.org, revised Aug 2022.
    10. Koen W. de Bock & Arno de Caigny, 2021. "Spline-rule ensemble classifiers with structured sparsity regularization for interpretable customer churn modeling," Post-Print hal-03391564, HAL.
    11. Undral Byambadalai & Tatsushi Oka & Shota Yasui, 2024. "Estimating Distributional Treatment Effects in Randomized Experiments: Machine Learning for Variance Reduction," Papers 2407.16037, arXiv.org.
    12. Ta-Wei Huang & Eva Ascarza, 2024. "Doing More with Less: Overcoming Ineffective Long-Term Targeting Using Short-Term Signals," Marketing Science, INFORMS, vol. 43(4), pages 863-884, July.
    13. Chen, Zhenshan & Towe, Charles & He, Xi, 2025. "Heterogeneous flood zone effects on coastal housing prices - Risk signal and mandatory costs," Journal of Environmental Economics and Management, Elsevier, vol. 131(C).
    14. Julius Owusu, 2023. "Randomization Inference of Heterogeneous Treatment Effects under Network Interference," Papers 2308.00202, arXiv.org, revised Jun 2025.
    15. Ajay Agrawal & Joshua Gans & Avi Goldfarb, 2019. "Economic Policy for Artificial Intelligence," Innovation Policy and the Economy, University of Chicago Press, vol. 19(1), pages 139-159.
    16. Guido W. Imbens, 2022. "Causality in Econometrics: Choice vs Chance," Econometrica, Econometric Society, vol. 90(6), pages 2541-2566, November.
    17. Mark Kattenberg & Bas Scheer & Jurre Thiel, 2023. "Causal forests with fixed effects for treatment effect heterogeneity in difference-in-differences," CPB Discussion Paper 452, CPB Netherlands Bureau for Economic Policy Analysis.
    18. Michael C Knaus, 2022. "Double machine learning-based programme evaluation under unconfoundedness [Econometric methods for program evaluation]," The Econometrics Journal, Royal Economic Society, vol. 25(3), pages 602-627.
    19. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," SciencePo Working papers Main hal-03455978, HAL.
    20. Arthur Charpentier & Emmanuel Flachaire & Ewen Gallic, 2023. "Optimal Transport for Counterfactual Estimation: A Method for Causal Inference," Papers 2301.07755, arXiv.org.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jbrese:v:190:y:2025:i:c:s0148296324006623. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jbusres .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.