IDEAS home Printed from https://ideas.repec.org/a/oup/qjecon/v133y2018i1p237-293..html
   My bibliography  Save this article

Human Decisions and Machine Predictions

Author

Listed:
  • Jon Kleinberg
  • Himabindu Lakkaraju
  • Jure Leskovec
  • Jens Ludwig
  • Sendhil Mullainathan

Abstract

Can machine learning improve human decision making? Bail decisions provide a good test case. Millions of times each year, judges make jail-or-release decisions that hinge on a prediction of what a defendant would do if released. The concreteness of the prediction task combined with the volume of data available makes this a promising machine-learning application. Yet comparing the algorithm to judges proves complicated. First, the available data are generated by prior judge decisions. We only observe crime outcomes for released defendants, not for those judges detained. This makes it hard to evaluate counterfactual decision rules based on algorithmic predictions. Second, judges may have a broader set of preferences than the variable the algorithm predicts; for instance, judges may care specifically about violent crimes or about racial inequities. We deal with these problems using different econometric strategies, such as quasi-random assignment of cases to judges. Even accounting for these concerns, our results suggest potentially large welfare gains: one policy simulation shows crime reductions up to 24.7% with no change in jailing rates, or jailing rate reductions up to 41.9% with no increase in crime rates. Moreover, all categories of crime, including violent crimes, show reductions; these gains can be achieved while simultaneously reducing racial disparities. These results suggest that while machine learning can be valuable, realizing this value requires integrating these tools into an economic framework: being clear about the link between predictions and decisions; specifying the scope of payoff functions; and constructing unbiased decision counterfactuals.

Suggested Citation

  • Jon Kleinberg & Himabindu Lakkaraju & Jure Leskovec & Jens Ludwig & Sendhil Mullainathan, 2018. "Human Decisions and Machine Predictions," The Quarterly Journal of Economics, Oxford University Press, vol. 133(1), pages 237-293.
  • Handle: RePEc:oup:qjecon:v:133:y:2018:i:1:p:237-293.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/qje/qjx032
    Download Restriction: Access to full text is restricted to subscribers.

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Will Dobbie & Jacob Goldin & Crystal Yang, 2016. "The Effects of Pre-Trial Detention on Conviction, Future Crime, and Employment: Evidence from Randomly Assigned Judges," Working Papers id:11212, eSocialSciences.
    2. Mendel, Brock & Shleifer, Andrei, 2012. "Chasing noise," Journal of Financial Economics, Elsevier, vol. 104(2), pages 303-320.
    3. Pedro Bordalo & Nicola Gennaioli & Andrei Shleifer, 2012. "Salience Theory of Choice Under Risk," The Quarterly Journal of Economics, Oxford University Press, vol. 127(3), pages 1243-1285.
    4. Shiller, Robert J, 1981. "Do Stock Prices Move Too Much to be Justified by Subsequent Changes in Dividends?," American Economic Review, American Economic Association, vol. 71(3), pages 421-436, June.
    5. Rafael Di Tella & Ernesto Schargrodsky, 2013. "Criminal Recidivism after Prison and Electronic Monitoring," Journal of Political Economy, University of Chicago Press, vol. 121(1), pages 28-73.
    6. Jeffrey R. Kling, 2006. "Incarceration Length, Employment, and Earnings," American Economic Review, American Economic Association, vol. 96(3), pages 863-876, June.
    7. Alexandre Belloni & Victor Chernozhukov & Christian Hansen, 2014. "High-Dimensional Methods and Inference on Structural and Treatment Effects," Journal of Economic Perspectives, American Economic Association, vol. 28(2), pages 29-50, Spring.
    8. Will Dobbie & Jacob Goldin & Crystal Yang, 2016. "The Effects of Pre-Trial Detention on Conviction, Future Crime, and Employment: Evidence from Randomly Assigned Judges," Working Papers 601, Princeton University, Department of Economics, Industrial Relations Section..
    9. Anna Aizer & Joseph J. Doyle, 2015. "Juvenile Incarceration, Human Capital, and Future Crime: Evidence from Randomly Assigned Judges," The Quarterly Journal of Economics, Oxford University Press, vol. 130(2), pages 759-803.
    10. Arpit Gupta & Christopher Hansman & Ethan Frenchman, 2016. "The Heavy Costs of High Bail: Evidence from Judge Randomization," The Journal of Legal Studies, University of Chicago Press, vol. 45(2), pages 471-505.
    11. Will Dobbie & Jacob Goldin & Crystal Yang, 2016. "The Effects of Pre-Trial Detention on Conviction, Future Crime, and Employment: Evidence from Randomly Assigned Judges," NBER Working Papers 22511, National Bureau of Economic Research, Inc.
    12. David S. Abrams & Chris Rohlfs, 2011. "Optimal Bail And The Value Of Freedom: Evidence From The Philadelphia Bail Experiment," Economic Inquiry, Western Economic Association International, vol. 49(3), pages 750-770, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. MIYAKAWA Daisuke, 2019. "Shocks to Supply Chain Networks and Firm Dynamics: An Application of Double Machine Learning," Discussion papers 19100, Research Institute of Economy, Trade and Industry (RIETI).
    2. Will Dobbie & Andres Liberman & Daniel Paravisini & Vikram Pathania, 2018. "Measuring Bias in Consumer Lending," Working Papers 623, Princeton University, Department of Economics, Industrial Relations Section..
    3. McKenzie, David & Sansone, Dario, 2019. "Predicting entrepreneurial success is hard: Evidence from a business plan competition in Nigeria," Journal of Development Economics, Elsevier, vol. 141(C).
    4. Liyang Tang, 2020. "Application of Nonlinear Autoregressive with Exogenous Input (NARX) neural network in macroeconomic forecasting, national goal setting and global competitiveness assessment," Papers 2005.08735, arXiv.org.
    5. Ajay Agrawal & Joshua S. Gans & Avi Goldfarb, 2019. "Artificial Intelligence: The Ambiguous Labor Market Impact of Automating Prediction," Journal of Economic Perspectives, American Economic Association, vol. 33(2), pages 31-50, Spring.
    6. Christoph Engel, 2018. "Empirical Methods for the Law," Journal of Institutional and Theoretical Economics (JITE), Mohr Siebeck, Tübingen, vol. 174(1), pages 5-23, March.
    7. Andini, Monica & Ciani, Emanuele & de Blasio, Guido & D'Ignazio, Alessio & Salvestrini, Viola, 2018. "Targeting with machine learning: An application to a tax rebate program in Italy," Journal of Economic Behavior & Organization, Elsevier, vol. 156(C), pages 86-102.
    8. Achim Ahrens & Christian B. Hansen & Mark E. Schaffer, 2020. "lassopack: Model selection and prediction with regularized regression in Stata," Stata Journal, StataCorp LP, vol. 20(1), pages 176-235, March.
    9. Athey, Susan & Wager, Stefan, 2017. "Efficient Policy Learning," Research Papers 3506, Stanford University, Graduate School of Business.
    10. Anthony Niblett, 2018. "Regulatory Reform in Ontario: Machine Learning and Regulation," C.D. Howe Institute Commentary, C.D. Howe Institute, issue 507, March.
    11. McKenzie, David J. & Sansone, Dario, 2017. "Man vs. Machine in Predicting Successful Entrepreneurs: Evidence from a Business Plan Competition in Nigeria," CEPR Discussion Papers 12523, C.E.P.R. Discussion Papers.
    12. Monica Andini & Emanuele Ciani & Guido de Blasio & Alessio D'Ignazio & Viola Salvestrini, 2017. "Targeting policy-compliers with machine learning: an application to a tax rebate programme in Italy," Temi di discussione (Economic working papers) 1158, Bank of Italy, Economic Research and International Relations Area.
    13. Jongbin Jung & Connor Concannon & Ravi Shroff & Sharad Goel & Daniel G. Goldstein, 2020. "Simple rules to guide expert classifications," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(3), pages 771-800, June.
    14. Tzai-Shuen Chen, 2018. "Evaluating Conditional Cash Transfer Policies with Machine Learning Methods," Papers 1803.06401, arXiv.org.
    15. Isil Erel & Léa H. Stern & Chenhao Tan & Michael S. Weisbach, 2018. "Selecting Directors Using Machine Learning," NBER Working Papers 24435, National Bureau of Economic Research, Inc.
    16. Jon Kleinberg & Sendhil Mullainathan, 2019. "Simplicity Creates Inequity: Implications for Fairness, Stereotypes, and Interpretability," NBER Working Papers 25854, National Bureau of Economic Research, Inc.
    17. Ballestar, María Teresa & Doncel, Luis Miguel & Sainz, Jorge & Ortigosa-Blanch, Arturo, 2019. "A novel machine learning approach for evaluation of public policies: An application in relation to the performance of university researchers," Technological Forecasting and Social Change, Elsevier, vol. 149(C).
    18. Sendhil Mullainathan & Ziad Obermeyer, 2019. "A Machine Learning Approach to Low-Value Health Care: Wasted Tests, Missed Heart Attacks and Mis-Predictions," NBER Working Papers 26168, National Bureau of Economic Research, Inc.
    19. Ginther, Donna K. & Heggeness, Misty L., 2020. "Administrative discretion in scientific funding: Evidence from a prestigious postdoctoral training program✰," Research Policy, Elsevier, vol. 49(4).
    20. Ian Lundberg & Arvind Narayanan & Karen Levy & Matthew Salganik, 2018. "Privacy, ethics, and data access: A case study of the Fragile Families Challenge," Working Papers wp18-09-ff, Princeton University, Woodrow Wilson School of Public and International Affairs, Center for Research on Child Wellbeing..
    21. Jason Anastasopoulos & George J. Borjas & Gavin G. Cook & Michael Lachanski, 2018. "Job Vacancies, the Beveridge Curve, and Supply Shocks: The Frequency and Content of Help-Wanted Ads in Pre- and Post-Mariel Miami," NBER Working Papers 24580, National Bureau of Economic Research, Inc.
    22. Monica Andini & Michela Boldrini & Emanuele Ciani & Guido de Blasio & Alessio D'Ignazio & Andrea Paladini, 2019. "Machine learning in the service of policy targeting: the case of public credit guarantees," Temi di discussione (Economic working papers) 1206, Bank of Italy, Economic Research and International Relations Area.

    More about this item

    JEL classification:

    • C01 - Mathematical and Quantitative Methods - - General - - - Econometrics
    • C54 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Quantitative Policy Modeling
    • C55 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Large Data Sets: Modeling and Analysis
    • D8 - Microeconomics - - Information, Knowledge, and Uncertainty
    • H0 - Public Economics - - General
    • K0 - Law and Economics - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:qjecon:v:133:y:2018:i:1:p:237-293.. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Oxford University Press). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.