IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2411.01067.html
   My bibliography  Save this paper

Randomized Controlled Trials for Security Copilot for IT Administrators

Author

Listed:
  • James Bono
  • Alec Xu

Abstract

As generative AI (GAI) tools become increasingly integrated into workplace environments, it is essential to measure their impact on productivity across specific domains. This study evaluates the effects of Microsoft's Security Copilot ("Copilot") on information technology administrators ("IT admins") through randomized controlled trials. Participants were divided into treatment and control groups, with the former granted access to Copilot within Microsoft's Entra and Intune admin centers. Across three IT admin scenarios - sign-in troubleshooting, device policy management, and device troubleshooting - Copilot users demonstrated substantial improvements in both accuracy and speed. Across all scenarios and tasks, Copilot subjects experienced a 34.53% improvement in overall accuracy and a 29.79% reduction in task completion time. We also find that the productivity benefits vary by task type, with more complex tasks showing greater improvement. In free response tasks, Copilot users identified 146.07% more relevant facts and reduced task completion time by 61.14%. Subject satisfaction with Copilot was high, with participants reporting reduced effort and a strong preference for using the tool in future tasks. These findings suggest that GAI tools like Copilot can significantly enhance the productivity and efficiency of IT admins, especially in scenarios requiring information synthesis and complex decision-making.

Suggested Citation

  • James Bono & Alec Xu, 2024. "Randomized Controlled Trials for Security Copilot for IT Administrators," Papers 2411.01067, arXiv.org, revised Nov 2024.
    • Handle: RePEc:arx:papers:2411.01067
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2411.01067
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Erik Brynjolfsson & Danielle Li & Lindsey Raymond, 2025. "Generative AI at Work," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 140(2), pages 889-942.
    2. Daron Acemoglu, 2025. "The simple macroeconomics of AI," Economic Policy, CEPR, CESifo, Sciences Po;CES;MSH, vol. 40(121), pages 13-58.
    3. Parteka, Aleksandra & Kordalska, Aleksandra, 2023. "Artificial intelligence and productivity: global evidence from AI patent and bibliometric data," Technovation, Elsevier, vol. 125(C).
    4. Alexander Bick & Adam Blandin & David Deming, 2023. "The Rapid Adoption of Generative AI," On the Economy 98843, Federal Reserve Bank of St. Louis.
    5. Ajay Agrawal & Joshua S. Gans & Avi Goldfarb, 2019. "Artificial Intelligence: The Ambiguous Labor Market Impact of Automating Prediction," Journal of Economic Perspectives, American Economic Association, vol. 33(2), pages 31-50, Spring.
    6. Tyna Eloundou & Sam Manning & Pamela Mishkin & Daniel Rock, 2023. "GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models," Papers 2303.10130, arXiv.org, revised Aug 2023.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Carlo Drago & Alberto Costantiello & Marco Savorgnan & Angelo Leogrande, 2025. "Macroeconomic and Labor Market Drivers of AI Adoption in Europe: A Machine Learning and Panel Data Approach," Economies, MDPI, vol. 13(8), pages 1-62, August.
    2. Carlo Drago & Alberto Costantiello & Marco Savorgnan & Angelo Leogrande, 2025. "Driving AI Adoption in the EU: A Quantitative Analysis of Macroeconomic Influences," Working Papers hal-05102974, HAL.
    3. Piyush Gulati & Arianna Marchetti & Phanish Puranam & Victoria Sevcenko, 2025. "Generative AI Adoption and Higher Order Skills," Papers 2503.09212, arXiv.org, revised Jun 2025.
    4. L. Elisa Celis & Lingxiao Huang & Nisheeth K. Vishnoi, 2025. "A Mathematical Framework for AI-Human Integration in Work," Papers 2505.23432, arXiv.org, revised May 2025.
    5. Lukas B. Freund & Lukas F. Mann, 2025. "Job Transformation, Specialization, and the Labor Market Effects of AI," CESifo Working Paper Series 12072, CESifo.
    6. Zara Contractor & Germ'an Reyes, 2025. "Generative AI in Higher Education: Evidence from an Elite College," Papers 2508.00717, arXiv.org.
    7. Christoph Riedl & Eric Bogert, 2024. "Effects of AI Feedback on Learning, the Skill Gap, and Intellectual Diversity," Papers 2409.18660, arXiv.org.
    8. Carvajal, Daniel & Franco, Catalina & Isaksson, Siri, 2024. "Will Artificial Intelligence Get in the Way of Achieving Gender Equality?," Discussion Paper Series in Economics 3/2024, Norwegian School of Economics, Department of Economics, revised 28 Apr 2025.
    9. Naomi Hausman & Oren Rigbi & Sarit Weisburd, 2025. "Generative AI’s Impact on Student Achievement and Implications for Worker Productivity," CESifo Working Paper Series 11843, CESifo.
    10. Tatsuru Kikuchi, 2025. "Weather-Aware AI Systems versus Route-Optimization AI: A Comprehensive Analysis of AI Applications in Transportation Productivity," Papers 2507.17099, arXiv.org.
    11. Leonardo Banh & Gero Strobel, 2023. "Generative artificial intelligence," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-17, December.
    12. Kiran Tomlinson & Sonia Jaffe & Will Wang & Scott Counts & Siddharth Suri, 2025. "Working with AI: Measuring the Applicability of Generative AI to Occupations," Papers 2507.07935, arXiv.org, revised Sep 2025.
    13. Frank M. Fossen & Trevor McLemore & Alina Sorgner, 2024. "Artificial Intelligence and Entrepreneurship," Foundations and Trends(R) in Entrepreneurship, now publishers, vol. 20(8), pages 781-904, December.
    14. Amali Matharaarachchi & Wishmitha Mendis & Kanishka Randunu & Daswin De Silva & Gihan Gamage & Harsha Moraliyage & Nishan Mills & Andrew Jennings, 2024. "Optimizing Generative AI Chatbots for Net-Zero Emissions Energy Internet-of-Things Infrastructure," Energies, MDPI, vol. 17(8), pages 1-19, April.
    15. Becker, Dominik & Deck, Luca & Feulner, Simon & Gutheil, Niklas & Schüll, Moritz & Decker, Stefan & Eymann, Torsten & Gimpel, Henner & Pippow, Andreas & Röglinger, Maximilian & Urbach, Nils, 2024. "Lohnt sich Microsoft 365 Copilot? Eine Potenzialanalyse für Unternehmen und Bildungseinrichtungen," Bayreuth Reports on Information Systems Management 72, University of Bayreuth, Chair of Information Systems Management.
    16. Jacob Dominski & Yong Suk Lee, 2025. "Advancing AI Capabilities and Evolving Labor Outcomes," Papers 2507.08244, arXiv.org.
    17. Harry Moroz & Mariana Viollaz, 2024. "The Future of Work in Central America and the Dominican Republic," World Bank Publications - Reports 42043, The World Bank Group.
    18. Christian Peukert & Florian Abeillon & Jérémie Haese & Franziska Kaiser & Alexander Staub, 2024. "Strategic Behavior and AI Training Data," CESifo Working Paper Series 11099, CESifo.
    19. Nida Çakır Melek, 2025. "Marshall B. Reinsdorf and Louise Sheiner (editors): The Measure of Economies: Measuring Productivity in an Age of Technological Change," Business Economics, Palgrave Macmillan;National Association for Business Economics, vol. 60(3), pages 175-178, July.
    20. Tatsuru Kikuchi, 2024. "Impact Evaluation on the European Privacy Laws governing generative-AI models -- Evidence in Relation between Internet Censorship and the Ban of ChatGPT in Italy," Papers 2407.06495, arXiv.org.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2411.01067. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.