IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2610.16863.html

Better together? A performance theory of human-machine skill policies under generalization difficulty

Author

Listed:
  • Riccardo Zanardelli

Abstract

Organizations pairing a human and a machine on a task decide, often by default, which executes each instance and what happens when both do. We account for what this attains under the squared rule, with performance distributed over instances to capture generalization difficulty. A gross stake lies between the skills' mean and best per instance. It splits at the best single skill into a systematic part, known from separate assessments and secured by commitment, and a residual, the expected value of perfect information on which skill each instance favors, known only from paired observation and split further by classes. Skills set the stake; workflows decide the take, via two channels. Routing directs work, weighting or selecting responses; with binary truth it is capped by the best skill per instance. Reading responses as joint evidence is not routing; what they jointly reveal caps it. Synergy lets skills interact before responding; bounded by no skill either way, its lift over independent responses must clear a break-even and can fall below one. As algorithms, arrangements form a ladder stating what each reads, holds and records, and a procedure whose checks keep earlier gains when class means are known and lose little on finite records given a margin. The theory extends to many skills, each worth what it adds where it beats all others held, and to competing skills, where at parity a single failure's allowed cost decides. In a public chest-radiograph study with 227 radiologists, they are near parity with the model and the stake is mostly residual; assisted reading, no better on average originally, returned less than averaging both responses; recalibrating the model alone beats any deployable arrangement of raw responses. Organizations can learn from their records what is at stake, which arrangement collects it and what evidence each step needs; pricing costs and errors is future work.

Suggested Citation

  • Riccardo Zanardelli, 2026. "Better together? A performance theory of human-machine skill policies under generalization difficulty," Papers 2610.16863, arXiv.org.
  • Handle: RePEc:arx:papers:2610.16863
    as

    Download full text from publisher

    File URL: https://arxiv.org/pdf/2610.16863
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Herbert A. Simon, 1955. "A Behavioral Model of Rational Choice," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 69(1), pages 99-118.
    2. Daron Acemoglu, 2025. "The simple macroeconomics of AI," Economic Policy, CEPR, CESifo, Sciences Po;CES;MSH, vol. 40(121), pages 13-58.
    3. Charles E. Clark, 1961. "The Greatest of a Finite Set of Random Variables," Operations Research, INFORMS, vol. 9(2), pages 145-162, April.
    4. Toru Kitagawa & Aleksey Tetenov, 2018. "Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice," Econometrica, Econometric Society, vol. 86(2), pages 591-616, March.
    5. Agrawal, Ajay & Gans, Joshua S. & Goldfarb, Avi, 2019. "Exploring the impact of artificial Intelligence: Prediction versus judgment," Information Economics and Policy, Elsevier, vol. 47(C), pages 1-6.
    6. Susan C. Athey & Kevin A. Bryan & Joshua S. Gans, 2020. "The Allocation of Decision Authority to Human and Artificial Intelligence," AEA Papers and Proceedings, American Economic Association, vol. 110, pages 80-84, May.
    7. Roopesh Ranjan & Tilmann Gneiting, 2010. "Combining probability forecasts," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 72(1), pages 71-91, January.
    8. Kenneth C. Lichtendahl & Yael Grushka-Cockayne & Victor Richmond Jose & Robert L. Winkler, 2022. "Extremizing and Antiextremizing in Bayesian Ensembles of Binary-Event Forecasts," Operations Research, INFORMS, vol. 70(5), pages 2998-3014, September.
    9. Patrick Hemmer & Max Schemmer & Niklas Kühl & Michael Vössing & Gerhard Satzger, 2025. "Complementarity in human-AI collaboration: concept, sources, and evidence," European Journal of Information Systems, Taylor & Francis Journals, vol. 34(6), pages 979-1002, November.
    10. David H. Autor & Frank Levy & Richard J. Murnane, 2003. "The skill content of recent technological change: an empirical exploration," Proceedings, Federal Reserve Bank of San Francisco, issue Nov.
    11. Jeremy Smith & Kenneth F. Wallis, 2009. "A Simple Explanation of the Forecast Combination Puzzle," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 71(3), pages 331-355, June.
    12. James J. Heckman & Jeffrey Smith & Nancy Clements, 1997. "Making The Most Out Of Programme Evaluations and Social Experiments: Accounting For Heterogeneity in Programme Impacts," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 487-535.
    13. Richard P. Larrick & Jack B. Soll, 2006. "Erratum--Intuitions About Combining Opinions: Misappreciation of the Averaging Principle," Management Science, INFORMS, vol. 52(2), pages 309-310, February.
    14. G. Elliott & C. Granger & A. Timmermann (ed.), 2006. "Handbook of Economic Forecasting," Handbook of Economic Forecasting, Elsevier, edition 1, volume 1, number 1.
    15. Fan, Yanqin & Park, Sang Soo, 2010. "Sharp Bounds On The Distribution Of Treatment Effects And Their Statistical Inference," Econometric Theory, Cambridge University Press, vol. 26(3), pages 931-951, June.
    16. Daron Acemoglu & Pascual Restrepo, 2022. "Tasks, Automation, and the Rise in U.S. Wage Inequality," Econometrica, Econometric Society, vol. 90(5), pages 1973-2016, September.
    17. Susan Athey & Stefan Wager, 2021. "Policy Learning With Observational Data," Econometrica, Econometric Society, vol. 89(1), pages 133-161, January.
    18. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    19. Fabrizio Dell’Acqua & Edward McFowland & Ethan Mollick & Hila Lifshitz & Katherine C. Kellogg & Saran Rajendran & Lisa Krayer & François Candelon & Karim R. Lakhani, 2026. "Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality," Organization Science, INFORMS, vol. 37(2), pages 403-423, March.
    20. James E. Smith & Robert L. Winkler, 2006. "The Optimizer's Curse: Skepticism and Postdecision Surprise in Decision Analysis," Management Science, INFORMS, vol. 52(3), pages 311-322, March.
    21. Charles F. Manski, 2004. "Statistical Treatment Rules for Heterogeneous Populations," Econometrica, Econometric Society, vol. 72(4), pages 1221-1246, July.
    22. Michelle Vaccaro & Abdullah Almaatouq & Thomas Malone, 2024. "When combinations of humans and AI are useful: A systematic review and meta-analysis," Nature Human Behaviour, Nature, vol. 8(12), pages 2293-2303, December.
    23. Richard P. Larrick & Jack B. Soll, 2006. "Intuitions About Combining Opinions: Misappreciation of the Averaging Principle," Management Science, INFORMS, vol. 52(1), pages 111-127, January.
    24. Charles F. Manski, 1997. "The Mixing Problem in Programme Evaluation," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 537-553.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    2. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    3. Neil Christy & Amanda Ellen Kowalski, 2024. "Counting Defiers: A Design-Based Model of an Experiment Can Reveal Evidence Beyond the Average Effect," Papers 2412.16352, arXiv.org, revised Jun 2026.
    4. Christopher Adjaho & Timothy Christensen, 2022. "Externally Valid Policy Choice," Papers 2205.05561, arXiv.org, revised Nov 2025.
    5. Firpo, Sergio & Galvao, Antonio F. & Kobus, Martyna & Parker, Thomas & Rosa-Dias, Pedro, 2025. "Loss aversion and the welfare ranking of policy interventions," Journal of Econometrics, Elsevier, vol. 252(PB).
    6. Peker, Cem & Wilkening, Tom, 2025. "Robust recalibration of aggregate probability forecasts using meta-beliefs," International Journal of Forecasting, Elsevier, vol. 41(2), pages 613-630.
    7. Johannes Müller-Trede & Shoham Choshen-Hillel & Meir Barneron & Ilan Yaniv, 2018. "The Wisdom of Crowds in Matters of Taste," Management Science, INFORMS, vol. 64(4), pages 1779-1803, April.
    8. Vira Semenova, 2023. "Debiased Machine Learning of Aggregated Intersection Bounds and Other Causal Parameters," Papers 2303.00982, arXiv.org, revised May 2025.
    9. Alejandro Sanchez-Becerra, 2023. "Robust inference for the treatment effect variance in experiments using machine learning," Papers 2306.03363, arXiv.org.
    10. Timothy B. Armstrong & Shu Shen, 2023. "Inference on optimal treatment assignments," The Japanese Economic Review, Springer, vol. 74(4), pages 471-500, October.
    11. Augustine Denteh & Helge Liebert, 2022. "Who Increases Emergency Department Use? New Insights from the Oregon Health Insurance Experiment," CESifo Working Paper Series 9664, CESifo.
    12. David Soule & Yael Grushka-Cockayne & Jason Merrick, 2024. "A Heuristic for Combining Correlated Experts When There Are Few Data," Management Science, INFORMS, vol. 70(10), pages 6637-6668, October.
    13. Egana-delSol, Pablo & Bravo-Ortega, Claudio, 2026. "Artificial intelligence exposure and labor market transformations in Latin America," Technological Forecasting and Social Change, Elsevier, vol. 230(C).
    14. Taylor, James W. & Taylor, Kathryn S., 2023. "Combining probabilistic forecasts of COVID-19 mortality in the United States," European Journal of Operational Research, Elsevier, vol. 304(1), pages 25-41.
    15. Neil Christy & A. E. Kowalski, 2024. "Starting Small: Prioritizing Safety over Efficacy in Randomized Experiments Using the Exact Finite Sample Likelihood," Papers 2407.18206, arXiv.org.
    16. Johannes Müller-Trede & Shoham Choshen-Hillel & Meir Barneron & Ilan Yaniv, 2017. "The Wisdom of Crowds in Matters of Taste," Discussion Paper Series dp709, The Federmann Center for the Study of Rationality, the Hebrew University, Jerusalem.
    17. Ying Jin & Naoki Egami, 2026. "Conformal Policy Learning with Distribution-Free Safety Guarantees," Papers 2609.17296, arXiv.org.
    18. Robert L. Winkler & Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose, 2019. "Probability Forecasts and Their Combination: A Research Perspective," Decision Analysis, INFORMS, vol. 16(4), pages 239-260, December.
    19. Kenneth C. Lichtendahl & Yael Grushka-Cockayne & Victor Richmond Jose & Robert L. Winkler, 2022. "Extremizing and Antiextremizing in Bayesian Ensembles of Binary-Event Forecasts," Operations Research, INFORMS, vol. 70(5), pages 2998-3014, September.
    20. von der Gracht, Heiko A. & Hommel, Ulrich & Prokesch, Tobias & Wohlenberg, Holger, 2016. "Testing weighting approaches for forecasting in a Group Wisdom Support System environment," Journal of Business Research, Elsevier, vol. 69(10), pages 4081-4094.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2610.16863. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: https://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.