IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2511.13865.html

Randomized Controlled Trials for Conditional Access Optimization Agent

Author

Listed:
  • James Bono
  • Beibei Cheng
  • Joaquin Lozano

Abstract

AI agents are increasingly deployed to automate complex enterprise workflows, yet evidence of their effectiveness in identity governance is limited. We report results from the first randomized controlled trial (RCT) evaluating an AI agent for Conditional Access (CA) policy management in Microsoft Entra. The agent assists with four high-value tasks: policy merging, Zero-Trust baseline gap detection, phased rollout planning, and user-policy alignment. In a production-grade environment, 162 identity administrators were randomly assigned to a control group (no agent) or treatment group (agent-assisted) and asked to perform these tasks. Agent access produced substantial gains: accuracy improved by 48% and task completion time decreased by 43% while holding accuracy constant. The largest benefits emerged on cognitively demanding tasks such as baseline gap detection. These findings demonstrate that purpose-built AI agents can significantly enhance both speed and accuracy in identity administration.

Suggested Citation

  • James Bono & Beibei Cheng & Joaquin Lozano, 2025. "Randomized Controlled Trials for Conditional Access Optimization Agent," Papers 2511.13865, arXiv.org.
    • Handle: RePEc:arx:papers:2511.13865
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2511.13865
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Erik Brynjolfsson & Danielle Li & Lindsey Raymond, 2025. "Generative AI at Work," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 140(2), pages 889-942.
    2. Anton Korinek, 2025. "AI Agents for Economic Research," NBER Working Papers 34202, National Bureau of Economic Research, Inc.
    3. James Bono & Justin Grana & Alec Xu, 2024. "Generative AI and Security Operations Center Productivity: Evidence from Live Operations," Papers 2411.03116, arXiv.org, revised Nov 2024.
    4. Flavio Calvino & Jelmer Reijerink & Lea Samek, 2025. "The effects of generative AI on productivity, innovation and entrepreneurship," OECD Artificial Intelligence Papers 39, OECD Publishing.
    5. Daron Acemoglu, 2025. "The simple macroeconomics of AI," Economic Policy, CEPR, CESifo, Sciences Po;CES;MSH, vol. 40(121), pages 13-58.
    6. James Bono & Alec Xu, 2024. "Randomized Controlled Trials for Security Copilot for IT Administrators," Papers 2411.01067, arXiv.org, revised Nov 2024.
    7. Alexander Bick & Adam Blandin & David Deming, 2023. "The Rapid Adoption of Generative AI," On the Economy 98843, Federal Reserve Bank of St. Louis.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. James Bono, 2025. "Randomized Controlled Trials for Phishing Triage Agent," Papers 2511.13860, arXiv.org.
    2. James Bono & Alec Xu, 2024. "Randomized Controlled Trials for Security Copilot for IT Administrators," Papers 2411.01067, arXiv.org, revised Nov 2024.
    3. Lu Fang & Zhe Yuan & Kaifu Zhang & Dante Donati & Miklos Sarvary, 2025. "Generative AI and Firm Productivity: Field Experiments in Online Retail," Papers 2510.12049, arXiv.org, revised Oct 2025.
    4. Lu Fang & Zhe Yuan & Kaifu Zhang & Dante Donati & Miklos Sarvary, 2025. "Generative AI and Firm Productivity: Field Experiments in Online Retail," CESifo Working Paper Series 12201, CESifo.
    5. Zara Contractor & Germ'an Reyes, 2025. "Generative AI in Higher Education: Evidence from an Elite College," Papers 2508.00717, arXiv.org.
    6. Riccardo Zanardelli, 2025. "Navigating the safe harbor paradox in human-machine systems," Papers 2509.14057, arXiv.org, revised Jan 2026.
    7. Carlo Drago & Alberto Costantiello & Marco Savorgnan & Angelo Leogrande, 2025. "Macroeconomic and Labor Market Drivers of AI Adoption in Europe: A Machine Learning and Panel Data Approach," Economies, MDPI, vol. 13(8), pages 1-62, August.
    8. Kiran Tomlinson & Sonia Jaffe & Will Wang & Scott Counts & Siddharth Suri, 2025. "Working with AI: Measuring the Applicability of Generative AI to Occupations," Papers 2507.07935, arXiv.org, revised Dec 2025.
    9. Carlo Drago & Alberto Costantiello & Marco Savorgnan & Angelo Leogrande, 2025. "Driving AI Adoption in the EU: A Quantitative Analysis of Macroeconomic Influences," Working Papers hal-05102974, HAL.
    10. Nida Çakır Melek, 2025. "Marshall B. Reinsdorf and Louise Sheiner (editors): The Measure of Economies: Measuring Productivity in an Age of Technological Change," Business Economics, Palgrave Macmillan;National Association for Business Economics, vol. 60(3), pages 175-178, July.
    11. Contractor, Zara & Reyes, Germán, 2025. "Generative AI in Higher Education: Evidence from an Elite College," IZA Discussion Papers 18055, Institute of Labor Economics (IZA).
    12. Piyush Gulati & Arianna Marchetti & Phanish Puranam & Victoria Sevcenko, 2025. "Generative AI Adoption and Higher Order Skills," Papers 2503.09212, arXiv.org, revised Jun 2025.
    13. Matthew O. Jackson & Qiaozhu Me & Stephanie W. Wang & Yutong Xie & Walter Yuan & Seth Benzell & Erik Brynjolfsson & Colin F. Camerer & James Evans & Brian Jabarian & Jon Kleinberg & Juanjuan Meng & Se, 2025. "AI Behavioral Science," Papers 2509.13323, arXiv.org.
    14. Nickel, Christiane & Kilponen, Juha & Moral-Benito, Enrique & Koester, Gerrit & Ciccarelli, Matteo & Enders, Almira & Holton, Sarah & Landau, Bettina & Venditti, Fabrizio & Bobeica, Elena & Brand, Cla, 2025. "A strategic view on the economic and inflation environment in the euro area," Occasional Paper Series 371, European Central Bank.
    15. L. Elisa Celis & Lingxiao Huang & Nisheeth K. Vishnoi, 2025. "A Mathematical Framework for AI-Human Integration in Work," Papers 2505.23432, arXiv.org, revised May 2025.
    16. Eleanor W. Dillon & Sonia Jaffe & Nicole Immorlica & Christopher T. Stanton, 2025. "Shifting Work Patterns with Generative AI," NBER Working Papers 33795, National Bureau of Economic Research, Inc.
    17. Peeyush Agarwal & Harsh Agarwal & Akshat Rana, 2025. "What Work is AI Actually Doing? Uncovering the Drivers of Generative AI Adoption," Papers 2510.23669, arXiv.org, revised Oct 2025.
    18. Fasheng Xu & Xiaoyu Wang & Wei Chen & Karen Xie, 2025. "The Economics of AI Foundation Models: Openness, Competition, and Governance," Papers 2510.15200, arXiv.org.
    19. Yongheng Hu, 2025. "Heterogeneous Agents in the Data Economy," Papers 2509.09656, arXiv.org.
    20. Leonardo Gambacorta & Tullio Jappelli & Tommaso Oliviero, 2025. "Exploring household adoption and usage of generative AI: new evidence from Italy," BIS Working Papers 1298, Bank for International Settlements.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2511.13865. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.