IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2610.07003.html

Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program

Author

Listed:
  • Liu Zhang
  • Mark Esposito

Abstract

Firms increasingly evaluate deployed AI agents using repeated human ratings, yet observed score changes may reflect the measurement process as much as changes in the agent itself. Firms also rarely know how long an evaluation remains informative as deployed systems evolve. We study reliability as a latent state using 2,611 rubric-scored production interviews from a deployed voice-and-video interviewing agent, overlapping human reviewers, the agent's change log, and support tickets. Reviewer severity varied substantially: two reviewers assessing the same batches differed by 0.79 standard deviations on the composite index, and a shift in reviewer composition flattened the raw trend. After adjusting for reviewer effects and smoothing batch-level estimates, evaluated reliability increased by 0.53 standard deviations from March to August 2026. Reliability also shifted materially across logged deploys. At the estimated rate of unaccounted movement, forecast uncertainty reaches the magnitude of the largest observed deploy contrast after about five weeks, although this horizon is imprecisely estimated and based on a small number of step-like changes. An independent operational outcome moved in the same direction: interview-related support tickets fell by about 10% per month relative to technical-issue tickets. These data do not identify a causal effect of evaluation on performance. Instead, they show that longitudinal AI evaluation requires three checks: comparability across raters, evidence currency under drift, and corroboration with consequential operational outcomes.

Suggested Citation

  • Liu Zhang & Mark Esposito, 2026. "Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program," Papers 2610.07003, arXiv.org.
    • Handle: RePEc:arx:papers:2610.07003
    as

    Download full text from publisher

    File URL: https://arxiv.org/pdf/2610.07003
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Whitney Newey & Kenneth West, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Russian Academy of National Economy and Public Administration (RANEPA), vol. 33(1), pages 125-132.
    2. Jeffrey R. Kling, 2006. "Incarceration Length, Employment, and Earnings," American Economic Review, American Economic Association, vol. 96(3), pages 863-876, June.
    3. John M. Abowd & Robert H. Creecy & Francis Kramarz, 2002. "Computing Person and Firm Effects Using Linked Longitudinal Employer-Employee Data," Longitudinal Employer-Household Dynamics Technical Papers 2002-06, Center for Economic Studies, U.S. Census Bureau.
    4. James G. MacKinnon & Matthew D. Webb, 2018. "The wild bootstrap for few (treated) clusters," Econometrics Journal, Royal Economic Society, vol. 21(2), pages 114-135, June.
    5. Matthew D. Webb, 2023. "Reworking wild bootstrap‐based inference for clustered errors," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 56(3), pages 839-858, August.
    6. John M. Abowd & Francis Kramarz & David N. Margolis, 1999. "High Wage Workers and High Wage Firms," Econometrica, Econometric Society, vol. 67(2), pages 251-334, March.
    7. Will Dobbie & Jacob Goldin & Crystal S. Yang, 2018. "The Effects of Pretrial Detention on Conviction, Future Crime, and Employment: Evidence from Randomly Assigned Judges," American Economic Review, American Economic Association, vol. 108(2), pages 201-240, February.
    8. A. Colin Cameron & Jonah B. Gelbach & Douglas L. Miller, 2008. "Bootstrap-Based Improvements for Inference with Clustered Errors," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 414-427, August.
    9. James G. MacKinnon & Matthew D. Webb, 2019. "Wild Bootstrap Randomization Inference for Few Treated Clusters," Advances in Econometrics, in: The Econometrics of Complex Survey Data, volume 39, pages 61-85, Emerald Group Publishing Limited.
    10. Durbin, James & Koopman, Siem Jan, 2012. "Time Series Analysis by State Space Methods," OUP Catalogue, Oxford University Press, edition 2, number 9780199641178.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. James G. MacKinnon & Morten Ørregaard Nielsen & Matthew D. Webb, 2023. "Fast and reliable jackknife and bootstrap methods for cluster‐robust inference," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(5), pages 671-694, August.
    2. MacKinnon, James G., 2023. "Fast cluster bootstrap methods for linear regression models," Econometrics and Statistics, Elsevier, vol. 26(C), pages 52-71.
    3. James G. MacKinnon, 2019. "How cluster-robust inference is changing applied econometrics," Canadian Journal of Economics, Canadian Economics Association, vol. 52(3), pages 851-881, August.
    4. Dorner, Matthias & Görlitz, Katja, 2020. "Training, wages and a missing school graduation cohort," IAB-Discussion Paper 202028, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany].
    5. James G. MacKinnon & Morten Ørregaard Nielsen & Matthew D. Webb, 2021. "Wild Bootstrap and Asymptotic Inference With Multiway Clustering," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(2), pages 505-519, March.
    6. James G. MacKinnon & Morten Ørregaard Nielsen & Matthew D. Webb, 2024. "Cluster-Robust Jackknife and Bootstrap Inference for Binary Response Models," Working Paper 1515, Economics Department, Queen's University.
    7. James G. MacKinnon, 2025. "When can we trust cluster-robust inference?," Canadian Stata Users' Group Meetings 2025 11, Stata Users Group.
    8. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Cluster-robust inference: A guide to empirical practice," Journal of Econometrics, Elsevier, vol. 232(2), pages 272-299.
    9. James G. MacKinnon & Morten {O}rregaard Nielsen & Matthew D. Webb, 2024. "Cluster-robust jackknife and bootstrap inference for logistic regression models," Papers 2406.00650, arXiv.org, revised May 2025.
    10. Goussé, Marion & Leturcq, Marion, 2022. "More or less unmarried. The impact of legal settings of cohabitation on labour market outcomes," European Economic Review, Elsevier, vol. 149(C).
    11. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Testing for the appropriate level of clustering in linear regression models," Journal of Econometrics, Elsevier, vol. 235(2), pages 2027-2056.
    12. Dorner, Matthias & Görlitz, Katja & Jahn, Elke J., 2024. "The impact of a missing school graduation cohort on the training market," Economics of Education Review, Elsevier, vol. 103(C).
    13. James G. MacKinnon & Matthew D. Webb, 2020. "When and How to Deal with Clustered Errors in Regression Models," Working Paper 1421, Economics Department, Queen's University.
    14. Hagemann, Andreas, 2019. "Placebo inference on treatment effects when the number of clusters is small," Journal of Econometrics, Elsevier, vol. 213(1), pages 190-209.
    15. Carpenter, Christopher S. & Gonzales, Gilbert & McKay, Tara & Sansone, Dario, 2020. "Effects of the Affordable Care Act Dependent Coverage Mandate on Health Insurance Coverage for Individuals in Same-Sex Couples," IZA Discussion Papers 13119, IZA Network @ LISER.
    16. Megan T Stevenson, 2018. "Distortion of Justice: How the Inability to Pay Bail Affects Case Outcomes," The Journal of Law, Economics, and Organization, Oxford University Press, vol. 34(4), pages 511-542.
    17. Prettyman, Alexa, 2024. "Underreporting child maltreatment during the pandemic: Evidence from Colorado," Children and Youth Services Review, Elsevier, vol. 156(C).
    18. Doremus, Jacqueline, 2019. "Unintended impacts from forest certification: Evidence from indigenous Aka households in Congo," Ecological Economics, Elsevier, vol. 166(C), pages 1-1.
    19. Catalina Amuedo-Dorantes & Agnese Romiti, 2024. "International student applications in the UK after Brexit," Journal of Economic Geography, Oxford University Press, vol. 24(5), pages 637-662.
    20. García-Ramos, Aixa, 2021. "Divorce laws and intimate partner violence: Evidence from Mexico," Journal of Development Economics, Elsevier, vol. 150(C).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2610.07003. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: https://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.