IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2608.27364.html

Sophistication in GenAI Use: Field Evidence from a Large Firm

Author

Listed:
  • Nicholas J. Hallman
  • Zachary T. Kowaleski
  • Anu Puvvada
  • Jaime J. Schmidt

Abstract

We study how sophistication in generative AI (genAI) use varies among the back-office workforce of a large firm. Using proprietary data, we observe 713,564 employee prompts and their corresponding large language model responses from nearly 4,000 back-office employees across 15 functional areas over eight months in 2025. We document three main findings. First, senior employees exhibit more sophisticated genAI use, consistent with domain expertise complementing genAI capabilities. Second, sophistication varies considerably across functions and is highest in Strategy, Digital Innovation, and Project Management, three groups that share a focus on firmwide strategic initiatives and organizational change. Third, we observe neither improvements in sophistication over time nor lasting improvements following formal AI training, suggesting that sophisticated use can be difficult to change. Together, our study provides measures of and insights into sophisticated genAI use that managers can use to improve outcomes and that researchers can use in future research.

Suggested Citation

  • Nicholas J. Hallman & Zachary T. Kowaleski & Anu Puvvada & Jaime J. Schmidt, 2026. "Sophistication in GenAI Use: Field Evidence from a Large Firm," Papers 2608.27364, arXiv.org.
    • Handle: RePEc:arx:papers:2608.27364
    as

    Download full text from publisher

    File URL: https://arxiv.org/pdf/2608.27364
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Andrew Burton-Jones & Camille Grange, 2013. "From Use to Effective Use: A Representation Theory Perspective," Information Systems Research, INFORMS, vol. 24(3), pages 632-658, September.
    2. Cassandra Estep & Emily E. Griffith & Nikki L. MacKenzie, 2024. "How do financial executives respond to the use of artificial intelligence in financial reporting and auditing?," Review of Accounting Studies, Springer, vol. 29(3), pages 2798-2831, September.
    3. Wanda J. Orlikowski, 2000. "Using Technology and Constituting Structures: A Practice Lens for Studying Technology in Organizations," Organization Science, INFORMS, vol. 11(4), pages 404-428, August.
    4. Jung Ho Choi & Chloe L. Xie, 2026. "Human + AI in Accounting: Early Evidence from the Field," Journal of Accounting Research, John Wiley & Sons, Ltd., vol. 64(3), pages 1333-1373, June.
    5. Weiguang Wang & Guodong (Gordon) Gao & Ritu Agarwal, 2024. "Friend or Foe? Teaming Between Artificial Intelligence and Workers with Variation in Experience," Management Science, INFORMS, vol. 70(9), pages 5753-5775, September.
    6. Anders Humlum & Emilie Vestergaard, 2025. "The unequal adoption of ChatGPT exacerbates existing inequalities among workers," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 122(1), pages 2414972121-, January.
    7. Scott Counts & Yan Chen & Jing Dong & Himanshu Sharma & Andrey Zaikin & Rui Hu & Alperen Kok & Gorkem Ozer Yilmaz & Siddharth Suri & Kiran Tomlinson & Sonia Jaffe & Will Wang, 2026. "AI in the Enterprise: How People Use M365 Copilot Chat," Papers 2605.23958, arXiv.org.
    8. Marc Eulerich & Aida Sanatizadeh & Hamid Vakilzadeh & David A. Wood, 2024. "Is it all hype? ChatGPT’s performance and disruptive potential in the accounting and auditing industries," Review of Accounting Studies, Springer, vol. 29(3), pages 2318-2349, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jeewon Cho & Insu Park, 2022. "Does Information Systems Support for Creativity Enhance Effective Information Systems Use and Job Satisfaction in Virtual Work?," Information Systems Frontiers, Springer, vol. 24(6), pages 1865-1886, December.
    2. Stratopoulos, Theophanis C. & Wang, Victor Xiaoqi, 2025. "Artificial intelligence and accounting research: a framework and agenda," International Journal of Accounting Information Systems, Elsevier, vol. 56(C).
    3. Candi, Marina & Beltagui, Ahmad, 2019. "Effective use of 3D printing in the innovation process," Technovation, Elsevier, vol. 80, pages 63-73.
    4. Ben Greiner & Philipp Grünwald & Thomas Lindner & Georg Lintner & Martin Wiernsperger, 2026. "Incentives, Framing, and Reliance on Algorithmic Advice: An Experimental Study," Management Science, INFORMS, vol. 72(1), pages 302-322, January.
    5. Andrew Burton-Jones & Olga Volkoff, 2017. "How Can We Develop Contextualized Theories of Effective Use? A Demonstration in the Context of Community-Care Electronic Health Records," Information Systems Research, INFORMS, vol. 28(3), pages 468-489, September.
    6. Jason A. Williams & Saurabh Gupta, 2025. "Integrating the IT Use Literature: Construct Validity and a Holistic Nomological Framework," Information Systems Frontiers, Springer, vol. 27(2), pages 563-583, April.
    7. Kertcher, Zack & Venkatraman, Rohan & Coslor, Erica, 2020. "Pleasingly parallel: Early cross-disciplinary work for innovation diffusion across boundaries in grid computing," Journal of Business Research, Elsevier, vol. 116(C), pages 581-594.
    8. Christos Makridis, 2026. "The Sum of All (Workplace) Fears: How Managers Mediate the Fear of AI Job Displacement," CESifo Working Paper Series 12678, CESifo.
    9. Pandza, Krsto & Ellwood, Paul, 2013. "Strategic and ethical foundations for responsible innovation," Research Policy, Elsevier, vol. 42(5), pages 1112-1125.
    10. Jin P. Gerlach & Ronald T. Cenfetelli, 2022. "Overcoming the Single-IS Paradigm in Individual-Level IS Research," Information Systems Research, INFORMS, vol. 33(2), pages 476-488, June.
    11. Aaltonen, Aleksi Ville & Alaimo, Cristina & Kallinikos, Jannis, 2021. "The making of data commodities: data analytics as an embedded process," LSE Research Online Documents on Economics 110296, London School of Economics and Political Science, LSE Library.
    12. Christoph Riedl & Eric Bogert, 2024. "Who Benefits from AI? Self-Selection, Skill Gap, and the Hidden Costs of AI Feedback," Papers 2409.18660, arXiv.org, revised Apr 2026.
    13. Weiling Ke & Lele Kang & Chuan-Hoo Tan & Chih-Hung Peng, 2021. "User Competence with Enterprise Systems: The Effects of Work Environment Factors," Information Systems Research, INFORMS, vol. 32(3), pages 860-875, September.
    14. Philipp Hansmeier & Philipp zur Heiden & Daniel Beverungen, 2026. "A domain-specific modeling language for customer journeys in digital data ecosystems," Working Papers Dissertations 183, Paderborn University, Faculty of Business Administration and Economics.
    15. Sulin Ba & Jan Stallaert & Andrew B. Whinston, 2001. "Research Commentary: Introducing a Third Dimension in Information Systems Design—The Case for Incentive Alignment," Information Systems Research, INFORMS, vol. 12(3), pages 225-239, September.
    16. Sean Shun Cao & Wilbur Xinyuan Chen & Guang Ma & Suraj Srinivasan, 2026. "Generative AI in Capital Markets: Information Production, Dissemination, and Processing," Journal of Accounting Research, John Wiley & Sons, Ltd., vol. 64(3), pages 1427-1450, June.
    17. Tammar B. Zilber, 2011. "Institutional Multiplicity in Practice: A Tale of Two High-Tech Conferences in Israel," Organization Science, INFORMS, vol. 22(6), pages 1539-1559, December.
    18. Lise Arena & Anthony Hussenot, 2021. "From Innovations at Work to Innovative Ways of Conceptualizing Organization: A Brief History of Organization Studies," Post-Print hal-03290300, HAL.
    19. François-Xavier de Vaujany & Sabine Carton & Carine Dominguez-Perry & Emmanuelle Vaast, 2012. "Performativity and Information Technologies: An inter-organizational perspective," Post-Print halshs-00851315, HAL.
    20. EuiBeom Jeong & DonHee Lee, 2025. "AI–human collaboration in services: an integrative framework to uncover the key success factors," Service Business, Springer;Pan-Pacific Business Association, vol. 19(3), pages 1-45, September.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2608.27364. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: https://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.