IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2605.30916.html

Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregation

Author

Listed:
  • Andreas Haupt
  • Justin Hartenstein
  • Anka Reuel
  • Mykel Kochenderfer
  • Sanmi Koyejo

Abstract

AI benchmarks have well-documented limitations, with prior work examining contamination, saturation, and construct underspecification. Aggregation has received far less attention: benchmarks are typically summarized by uniformly averaging item-level scores, implicitly treating every test item as equally valuable. We model benchmarking as a multitask principal-agent game and show that the welfare loss from a benchmark is determined jointly by three item-level primitives: alignment with normative welfare priorities, marginal improvability, and performance variance. We translate the theory into an audit framework that ranks items along each of these three axes, and apply it to OLMES items using WORKBank for welfare, the EvoLM 4B suite for improvability, and the PolyPythias 410M panel for variance. The framework surfaces items that are Pareto-inferior within OLMES subject to a pro-worker welfare operationalization. All code is available at https://github.com/stair-lab/principal-agent-benchmarks.

Suggested Citation

  • Andreas Haupt & Justin Hartenstein & Anka Reuel & Mykel Kochenderfer & Sanmi Koyejo, 2026. "Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregation," Papers 2605.30916, arXiv.org.
    • Handle: RePEc:arx:papers:2605.30916
    as

    Download full text from publisher

    File URL: https://arxiv.org/pdf/2605.30916
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Erik Brynjolfsson, 2022. "The Turing Trap: The Promise & Peril of Human-Like Artificial Intelligence," Papers 2201.04200, arXiv.org.
    2. Konrad, Kai A., 2009. "Strategy and Dynamics in Contests," OUP Catalogue, Oxford University Press, number 9780199549603.
    3. George Baker, 2002. "Distortion and Risk in Optimal Incentive Contracts," Journal of Human Resources, University of Wisconsin Press, vol. 37(4), pages 728-751.
    4. Simon Ott & Adriano Barbosa-Silva & Kathrin Blagec & Jan Brauner & Matthias Samwald, 2022. "Mapping global dynamics of benchmark creation and saturation in artificial intelligence," Nature Communications, Nature, vol. 13(1), pages 1-11, December.
    5. Brian A. Jacob & Steven D. Levitt, 2003. "Rotten Apples: An Investigation of the Prevalence and Predictors of Teacher Cheating," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 118(3), pages 843-877.
    6. Holmstrom, Bengt & Milgrom, Paul, 1987. "Aggregation and Linearity in the Provision of Intertemporal Incentives," Econometrica, Econometric Society, vol. 55(2), pages 303-328, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Carolyn J. Heinrich & Gerald Marschke, 2010. "Incentives and their dynamics in public sector performance management systems," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 29(1), pages 183-208.
    2. Thiele, Veikko, 2010. "Task-specific abilities in multi-task principal-agent relationships," Labour Economics, Elsevier, vol. 17(4), pages 690-698, August.
    3. Nick Zubanov & W.S. Siebert, 2009. "Management economics in a large UK retailer," CPB Discussion Paper 125, CPB Netherlands Bureau for Economic Policy Analysis.
    4. Dietrichson, Jens, 2013. "Coordination Incentives, Performance Measurement and Resource Allocation in Public Sector Organizations," Working Papers 2013:26, Lund University, Department of Economics.
    5. Sloof, Randolph & van Praag, C. Mirjam, 2008. "Performance measurement, expectancy and agency theory: An experimental study," Journal of Economic Behavior & Organization, Elsevier, vol. 67(3-4), pages 794-809, September.
    6. Roland G. Fryer, 2013. "Teacher Incentives and Student Achievement: Evidence from New York City Public Schools," Journal of Labor Economics, University of Chicago Press, vol. 31(2), pages 373-407.
    7. Felipe Balmaceda, 2012. "On the Optimality of One-size-fits-all Contracts: The Limited Liability Case," Documentos de Trabajo 291, Centro de Economía Aplicada, Universidad de Chile.
    8. Derek Neal, 2011. "The Design of Performance Pay in Education," NBER Working Papers 16710, National Bureau of Economic Research, Inc.
    9. Kyota Eguchi, 2017. "Guilty Conscience And Incentives With Performance Assessment Errors," Economic Inquiry, Western Economic Association International, vol. 55(1), pages 434-450, January.
    10. Siebert, W. Stanley & Zubanov, Nick, 2008. "Management Economics in a Large Retail Organization," IZA Discussion Papers 3645, IZA Network @ LISER.
    11. Andersson Fredrik, 2011. "Make-or-Buy Decisions and the Manipulability of Performance Measures," The B.E. Journal of Theoretical Economics, De Gruyter, vol. 11(1), pages 1-30, December.
    12. Courty, Pascal & Marschke, Gerald, 2004. "A General Test of Gaming," CEPR Discussion Papers 4514, Centre for Economic Policy Research.
    13. Anthony M. Marino & Ján Zábojník, 2008. "Work‐related perks, agency problems, and optimal incentive contracts," RAND Journal of Economics, RAND Corporation, vol. 39(2), pages 565-585, June.
    14. Alberto Bayo-Moriones & Jose E. Galdon-Sanchez & Sara Martinez-de-Morentin, 2017. "Performance Measurement and Incentive Intensity," Journal of Labor Research, Springer, vol. 38(4), pages 496-546, December.
    15. Lacetera, Nicola & Zirulia, Lorenzo, 2012. "Individual preferences, organization, and competition in a model of R&D incentive provision," Journal of Economic Behavior & Organization, Elsevier, vol. 84(2), pages 550-570.
    16. Kubick, Thomas R. & Lockhart, G. Brandon, 2016. "Do external labor market incentives motivate CEOs to adopt more aggressive corporate tax reporting preferences?," Journal of Corporate Finance, Elsevier, vol. 36(C), pages 255-277.
    17. Christian Zehnder & Holger Herz & Jean-Philippe Bonardi, 2016. "A Productive Clash of Cultures: Injecting Economics into Leadership Research," CESifo Working Paper Series 6175, CESifo.
    18. Bengt Holmström, 2017. "Pay for Performance and Beyond," American Economic Review, American Economic Association, vol. 107(7), pages 1753-1777, July.
    19. Zabel, Astrid & Roe, Brian, 2009. "Optimal design of pro-conservation incentives," Ecological Economics, Elsevier, vol. 69(1), pages 126-134, November.
    20. Roland G. Fryer, 2011. "Teacher Incentives and Student Achievement: Evidence from New York City Public Schools," NBER Working Papers 16850, National Bureau of Economic Research, Inc.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2605.30916. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: https://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.