IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2603.02076.html

When an AI Judges Your Work: The Hidden Costs of Algorithmic Assessment

Author

Listed:
  • David Almog
  • Lucas Lippman
  • Daniel Martin

Abstract

We use an online experiment with a real work task to study whether workers change their behavior when they know AI will be used to judge their work instead of humans. We find that individuals produce a higher quantity of output when they are assigned an AI evaluator. However, controlling for quantity, the quality of their output is lower, regardless of whether quality is measured using humans or LLM grades. We also find that workers are more likely to use external tools, including LLMs, when they know AI is used to judge their work instead of humans. However, the increase in external tool use does not appear to explain the differences in quantity or quality across treatments.

Suggested Citation

  • David Almog & Lucas Lippman & Daniel Martin, 2026. "When an AI Judges Your Work: The Hidden Costs of Algorithmic Assessment," Papers 2603.02076, arXiv.org.
    • Handle: RePEc:arx:papers:2603.02076
    as

    Download full text from publisher

    File URL: https://arxiv.org/pdf/2603.02076
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Daron Acemoglu & Pascual Restrepo, 2019. "Automation and New Tasks: How Technology Displaces and Reinstates Labor," Journal of Economic Perspectives, American Economic Association, vol. 33(2), pages 3-30, Spring.
    2. Gary Charness & Brian Jabarian & John A. List, 2023. "Generation Next: Experimentation with AI," NBER Working Papers 31679, National Bureau of Economic Research, Inc.
    3. Jens Ludwig & Sendhil Mullainathan & Ashesh Rambachan, 2024. "The Unreasonable Effectiveness of Algorithms," AEA Papers and Proceedings, American Economic Association, vol. 114, pages 623-627, May.
    4. Simone Zhang & Janet Xu & AJ Alvero, 2025. "Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization," Sociological Methods & Research, , vol. 54(3), pages 1197-1242, August.
    5. Anton Korinek, 2023. "Generative AI for Economic Research: Use Cases and Implications for Economists," Journal of Economic Literature, American Economic Association, vol. 61(4), pages 1281-1317, December.
    6. Erik Brynjolfsson & Danielle Li & Lindsey Raymond, 2025. "Generative AI at Work," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 140(2), pages 889-942.
    7. Corgnet, Brice & Hernán-González, Roberto & Mateo, Ricardo, 2023. "Peer effects in an automated world," Labour Economics, Elsevier, vol. 85(C).
    8. Anders Humlum & Emilie Vestergaard, 2025. "The unequal adoption of ChatGPT exacerbates existing inequalities among workers," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 122(1), pages 2414972121-, January.
    9. March, Christoph, 2021. "Strategic interactions between humans and artificial intelligence: Lessons from experiments with computer players," Journal of Economic Psychology, Elsevier, vol. 87(C).
    10. Frey, Bruno S, 1993. "Does Monitoring Increase Work Effort? The Rivalry with Trust and Loyalty," Economic Inquiry, Western Economic Association International, vol. 31(4), pages 663-670, October.
    11. Enrique Ide & Eduard Talamas, 2023. "Artificial Intelligence in the Knowledge Economy," Papers 2312.05481, arXiv.org, revised May 2025.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chang, Samuel & Kennedy, Andrew & Leonard, Aaron & List, John A., 2026. "12 best practices for leveraging Generative AI in experimental research," Journal of Economic Behavior & Organization, Elsevier, vol. 246(C).
    2. Buchanan, Joy & Hickman, William, 2024. "Do people trust humans more than ChatGPT?," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 112(C).
    3. Alex Farach, 2026. "AI as Coordination-Compressing Capital: Task Reallocation, Organizational Redesign, and the Regime Fork," Papers 2602.16078, arXiv.org, revised Mar 2026.
    4. Rosa-García, Alfonso, 2024. "Student Reactions to AI-Replicant Professor in an Econ101 Teaching Video," MPRA Paper 120135, University Library of Munich, Germany.
    5. Čeryová, Barbara & Árendáš, Peter & Kotlebová, Jana, 2026. "Connectedness and risk transmission across artificial intelligence industries," Research in International Business and Finance, Elsevier, vol. 84(C).
    6. Tatsuru Kikuchi, 2025. "Weather-Aware AI Systems versus Route-Optimization AI: A Comprehensive Analysis of AI Applications in Transportation Productivity," Papers 2507.17099, arXiv.org.
    7. Francesco Venturini, 2025. "Generative AI and Income Growth: Early Evidence on Global Data," Gospodarka Narodowa. The Polish Journal of Economics, Warsaw School of Economics, issue 3, pages 31-46.
    8. Ide, Enrique & Talamàs, Eduard, 2026. "The impact of AI on global knowledge work," Journal of Monetary Economics, Elsevier, vol. 157(C).
    9. Giampaolo Bonomi, 2026. "The Division of Understanding: Specialization and Democratic Accountability," Papers 2604.09871, arXiv.org, revised Sep 2026.
    10. Soohyoung Lee & Dawoon Jeong & Jeong-Dong Lee, 2026. "Contrasting pathways of automation: routine task substitution vs. AI complementarity," Humanities and Social Sciences Communications, Palgrave Macmillan, vol. 13(1), pages 1-11, December.
    11. Kiran Tomlinson & Sonia Jaffe & Will Wang & Scott Counts & Siddharth Suri, 2025. "Working with AI: Measuring the Applicability of Generative AI to Occupations," Papers 2507.07935, arXiv.org, revised Dec 2025.
    12. Seth Gordon Benzell & Kyle R. Myers, 2026. "Automation Experiments and Inequality," NBER Working Papers 34668, National Bureau of Economic Research, Inc.
    13. Junhui Jeff Cai & Xian Gu & Liugang Sheng & Mengjia Xia & Linda Zhao & Wu Zhu, 2025. "AI as "Co-founder": GenAI for Entrepreneurship," Papers 2512.06506, arXiv.org.
    14. Fabrizio Dell’Acqua & Edward McFowland & Ethan Mollick & Hila Lifshitz & Katherine C. Kellogg & Saran Rajendran & Lisa Krayer & François Candelon & Karim R. Lakhani, 2026. "Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality," Organization Science, INFORMS, vol. 37(2), pages 403-423, March.
    15. FUKAO, Kyoji & IKEUCHI, Kenta & NAGAYA, Yoshiaki & PERUGINI, Cristiano & POMPEI, Fabrizio, 2026. "Automation Risk, Innovative Capital, and Labour Market Outcomes : Evidence from Japan," CEI Working Paper Series 2026-03, Center for Economic Institutions, Institute of Economic Research, Hitotsubashi University.
    16. Nathalie Greenan & Dario Guarascio & Jelena Reljic, 2025. "AI and the labour market: opening the black box," Eurasian Business Review, Springer;Eurasia Business and Economics Society, vol. 15(4), pages 925-951, December.
    17. Sharique Hasan & Alexander Oettl & Sampsa Samila, 2025. "From Model Design to Organizational Design: Complexity Redistribution and Trade-Offs in Generative AI," Papers 2506.22440, arXiv.org, revised May 2026.
    18. Piyush Gulati & Arianna Marchetti & Victoria Sevcenko & Phanish Puranam, 2025. "How Generative AI Adoption Alters the Demand for Cognitive and Social Skills Within Roles: A Skill-Centric Analysis," Papers 2503.09212, arXiv.org, revised Aug 2026.
    19. Fabian Stephany & Ole Teutloff & Angelo Leone, 2026. "AI Skills Improve Job Prospects: Causal Evidence from a Hiring Experiment," Papers 2601.13286, arXiv.org, revised Mar 2026.
    20. Udayan Mukherjee & Harry Nicholls, 2024. "The impact of artificial intelligence – an economic analysis," Treasury Analytical Notes Series an24/06, New Zealand Treasury.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2603.02076. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: https://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.