IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/27802.html
   My bibliography  Save this paper

On the Use of Outcome Tests for Detecting Bias in Decision Making

Author

Listed:
  • Ivan A. Canay
  • Magne Mogstad
  • Jack Mountjoy

Abstract

The decisions of judges, lenders, journal editors, and other gatekeepers often lead to significant disparities across affected groups. An important question is whether, and to what extent, these group-level disparities are driven by relevant differences in underlying individual characteristics, or by biased decision makers. Becker (1957, 1993) proposed an outcome test of bias based on differences in post-decision outcomes across groups, inspiring a large and growing empirical literature. The goal of our paper is to offer a methodological blueprint for empirical work that seeks to use outcome tests to detect bias. We show that models of decision making underpinning outcome tests can be usefully recast as Roy models, since heterogeneous potential outcomes enter directly into the decision maker's choice equation. Different members of the Roy model family, however, are distinguished by the tightness of the link between potential outcomes and decisions. We show that these distinctions have important implications for defining bias, deriving logically valid outcome tests of such bias, and identifying the marginal outcomes that the test requires.

Suggested Citation

  • Ivan A. Canay & Magne Mogstad & Jack Mountjoy, 2020. "On the Use of Outcome Tests for Detecting Bias in Decision Making," NBER Working Papers 27802, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:27802
    Note: LE LS PE TWP
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w27802.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Will Dobbie & Jae Song, 2015. "Debt Relief and Debtor Outcomes: Measuring the Effects of Consumer Bankruptcy Protection," American Economic Review, American Economic Association, vol. 105(3), pages 1272-1311, March.
    2. Manudeep Bhuller & Gordon B. Dahl & Katrine V. Løken & Magne Mogstad, 2020. "Incarceration, Recidivism, and Employment," Journal of Political Economy, University of Chicago Press, vol. 128(4), pages 1269-1324.
    3. Willis, Robert J & Rosen, Sherwin, 1979. "Education and Self-Selection," Journal of Political Economy, University of Chicago Press, vol. 87(5), pages 7-36, October.
    4. David Arnold & Will Dobbie & Crystal S Yang, 2018. "Racial Bias in Bail Decisions," The Quarterly Journal of Economics, Oxford University Press, vol. 133(4), pages 1885-1932.
    5. James J. Heckman & Vytlacil, Edward J., 2007. "Econometric Evaluation of Social Programs, Part II: Using the Marginal Treatment Effect to Organize Alternative Econometric Estimators to Evaluate Social Programs, and to Forecast their Effects in New," Handbook of Econometrics, in: J.J. Heckman & E.E. Leamer (ed.), Handbook of Econometrics, edition 1, volume 6, chapter 71, Elsevier.
    6. David Arnold & Will Dobbie & Peter Hull, 2022. "Measuring Racial Discrimination in Bail Decisions," American Economic Review, American Economic Association, vol. 112(9), pages 2992-3038, September.
    7. James J. Heckman & Edward Vytlacil, 2005. "Structural Equations, Treatment Effects, and Econometric Policy Evaluation," Econometrica, Econometric Society, vol. 73(3), pages 669-738, May.
    8. Will Dobbie & Jacob Goldin & Crystal S. Yang, 2018. "The Effects of Pretrial Detention on Conviction, Future Crime, and Employment: Evidence from Randomly Assigned Judges," American Economic Review, American Economic Association, vol. 108(2), pages 201-240, February.
    9. Will Dobbie & Andres Liberman & Daniel Paravisini & Vikram Pathania, 2021. "Measuring Bias in Consumer Lending [Loan Prospecting and the Loss of Soft Information]," Review of Economic Studies, Oxford University Press, vol. 88(6), pages 2799-2832.
    10. Galasso, Alberto & Schankerman, Mark, 2015. "Patents and cumulative innovation: causal evidence from the courts," LSE Research Online Documents on Economics 61614, London School of Economics and Political Science, LSE Library.
    11. A. Belloni & D. Chen & V. Chernozhukov & C. Hansen, 2012. "Sparse Models and Methods for Optimal Instruments With an Application to Eminent Domain," Econometrica, Econometric Society, vol. 80(6), pages 2369-2429, November.
    12. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    13. Nicole Maestas & Kathleen J. Mullen & Alexander Strand, 2013. "Does Disability Insurance Receipt Discourage Work? Using Examiner Assignment to Estimate Causal Effects of SSDI Receipt," American Economic Review, American Economic Association, vol. 103(5), pages 1797-1829, August.
    14. Emily Leslie & Nolan G. Pope, 2017. "The Unintended Impact of Pretrial Detention on Case Outcomes: Evidence from New York City Arraignments," Journal of Law and Economics, University of Chicago Press, vol. 60(3), pages 529-557.
    15. Kevin Lang & Ariella Kahn-Lang Spitzer, 2020. "Race Discrimination: An Economic Perspective," Journal of Economic Perspectives, American Economic Association, vol. 34(2), pages 68-89, Spring.
    16. Dolton, P J & Makepeace, G H & Van Der Klaauw, W, 1989. "Occupational Choice and Earnings Determination: The Role of Sample Selection and Non-pecuniary Factors," Oxford Economic Papers, Oxford University Press, vol. 41(3), pages 573-594, July.
    17. Jeffrey R. Kling, 2006. "Incarceration Length, Employment, and Earnings," American Economic Review, American Economic Association, vol. 96(3), pages 863-876, June.
    18. Kevin Lang & Jee-Yeon K. Lehmann, 2012. "Racial Discrimination in the Labor Market: Theory and Empirics," Journal of Economic Literature, American Economic Association, vol. 50(4), pages 959-1006, December.
    19. James J. Heckman, 1998. "Detecting Discrimination," Journal of Economic Perspectives, American Economic Association, vol. 12(2), pages 101-116, Spring.
    20. D’Haultfœuille, Xavier & Maurel, Arnaud, 2013. "Inference on an extended Roy model, with an application to schooling decisions in France," Journal of Econometrics, Elsevier, vol. 174(2), pages 95-106.
    21. A. D. Roy, 1951. "Some Thoughts On The Distribution Of Earnings," Oxford Economic Papers, Oxford University Press, vol. 3(2), pages 135-146.
    22. George J. Borjas, 2021. "Self-Selection and the Earnings of Immigrants," World Scientific Book Chapters, in: Foundational Essays in Immigration Economics, chapter 4, pages 69-91, World Scientific Publishing Co. Pte. Ltd..
    23. Bhaven Sampat & Heidi L. Williams, 2019. "How Do Patents Affect Follow-On Innovation? Evidence from the Human Genome," American Economic Review, American Economic Association, vol. 109(1), pages 203-236, January.
    24. Joseph J. Doyle Jr., 2007. "Child Protection and Child Outcomes: Measuring the Effects of Foster Care," American Economic Review, American Economic Association, vol. 97(5), pages 1583-1610, December.
    25. Nicola Persico, 2009. "Racial Profiling? Detecting Bias Using Statistical Evidence," Annual Review of Economics, Annual Reviews, vol. 1(1), pages 229-254, May.
    26. Shamena Anwar & Hanming Fang, 2006. "An Alternative Test of Racial Prejudice in Motor Vehicle Searches: Theory and Evidence," American Economic Review, American Economic Association, vol. 96(1), pages 127-151, March.
    27. John Knowles & Nicola Persico & Petra Todd, 2001. "Racial Bias in Motor Vehicle Searches: Theory and Evidence," Journal of Political Economy, University of Chicago Press, vol. 109(1), pages 203-232, February.
    28. Chris Robinson & Nigel Tomes, 1982. "Self-Selection and Interprovincial Migration in Canada," Canadian Journal of Economics, Canadian Economics Association, vol. 15(3), pages 474-502, August.
    29. Christian N. Brinch & Magne Mogstad & Matthew Wiswall, 2017. "Beyond LATE with a Discrete Instrument," Journal of Political Economy, University of Chicago Press, vol. 125(4), pages 985-1039.
    30. Magne Mogstad & Andres Santos & Alexander Torgovitsky, 2018. "Using Instrumental Variables for Inference About Policy Relevant Treatment Parameters," Econometrica, Econometric Society, vol. 86(5), pages 1589-1619, September.
    31. Lee, Lung-Fei, 1978. "Unionism and Wage Rates: A Simultaneous Equations Model with Qualitative and Limited Dependent Variables," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 19(2), pages 415-433, June.
    32. Jon Kleinberg & Himabindu Lakkaraju & Jure Leskovec & Jens Ludwig & Sendhil Mullainathan, 2018. "Human Decisions and Machine Predictions," The Quarterly Journal of Economics, Oxford University Press, vol. 133(1), pages 237-293.
    33. Brock, William A. & Cooley, Jane & Durlauf, Steven N. & Navarro, Salvador, 2012. "On the observational implications of taste-based discrimination in racial profiling," Journal of Econometrics, Elsevier, vol. 166(1), pages 66-78.
    34. Heckman, James J, 1974. "Shadow Prices, Market Wages, and Labor Supply," Econometrica, Econometric Society, vol. 42(4), pages 679-694, July.
    35. Bjorklund, Anders & Moffitt, Robert, 1987. "The Estimation of Wage Gains and Welfare Gains in Self-selection," The Review of Economics and Statistics, MIT Press, vol. 69(1), pages 42-49, February.
    36. Edward Vytlacil, 2002. "Independence, Monotonicity, and Latent Index Models: An Equivalence Result," Econometrica, Econometric Society, vol. 70(1), pages 331-341, January.
    37. McElroy, Marjorie B & Horney, Mary Jean, 1981. "Nash-Bargained Household Decisions: Toward a Generalization of the Theory of Demand," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 22(2), pages 333-349, June.
    38. Alberto Galasso & Mark Schankerman, 2015. "Patents and Cumulative Innovation: Causal Evidence from the Courts," The Quarterly Journal of Economics, Oxford University Press, vol. 130(1), pages 317-369.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Arcidiacono, Peter & Kinsler, Josh & Ransom, Tyler, 2022. "Asian American Discrimination in Harvard Admissions," European Economic Review, Elsevier, vol. 144(C).
    2. David Arnold & Will Dobbie & Peter Hull, 2022. "Measuring Racial Discrimination in Bail Decisions," American Economic Review, American Economic Association, vol. 112(9), pages 2992-3038, September.
    3. Rahul Deb & Ludovic Renou, 2022. "Which wage distributions are consistent with statistical discrimination?," Working Papers tecipa-736, University of Toronto, Department of Economics.
    4. Laura Blattner & Scott Nelson, 2021. "How Costly is Noise? Data and Disparities in Consumer Credit," Papers 2105.07554, arXiv.org.
    5. Ashesh Rambachan, 2022. "Identifying Prediction Mistakes in Observational Data," NBER Chapters, in: Economics of Artificial Intelligence, National Bureau of Economic Research, Inc.
    6. Will Dobbie & Crystal S. Yang, 2021. "The US Pretrial System: Balancing Individual Rights and Public Interests," Journal of Economic Perspectives, American Economic Association, vol. 35(4), pages 49-70, Fall.
    7. Lee, Ji Hyung & Park, Byoung G., 2023. "Nonparametric identification and estimation of the extended Roy model," Journal of Econometrics, Elsevier, vol. 235(2), pages 1087-1113.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Brigham Frandsen & Lars Lefgren & Emily Leslie, 2023. "Judging Judge Fixed Effects," American Economic Review, American Economic Association, vol. 113(1), pages 253-277, January.
    2. Domenico Depalo, 2020. "Explaining the causal effect of adherence to medication on cholesterol through the marginal patient," Health Economics, John Wiley & Sons, Ltd., vol. 29(S1), pages 110-126, October.
    3. Robert A. Moffitt & Matthew V. Zahn, 2019. "The Marginal Labor Supply Disincentives of Welfare: Evidence from Administrative Barriers to Participation," NBER Working Papers 26028, National Bureau of Economic Research, Inc.
    4. Manudeep Bhuller & Gordon B. Dahl & Katrine V. Løken & Magne Mogstad, 2020. "Incarceration, Recidivism, and Employment," Journal of Political Economy, University of Chicago Press, vol. 128(4), pages 1269-1324.
    5. Nicolás Grau & Damián Vergara, "undated". "A Simple Test for Prejudice in Decision Processes: The Prediction-Based Outcome Test," Working Papers wp493, University of Chile, Department of Economics.
    6. Michelle Yin & Garima Siwach & Dajun Lin, 2023. "Vocational Rehabilitation Services and Labor Market Outcomes for Transition‐Age Youth with Disabilities in Maine," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 42(1), pages 166-197, January.
    7. Pereda-Fernández, Santiago, 2023. "Identification and estimation of triangular models with a binary treatment," Journal of Econometrics, Elsevier, vol. 234(2), pages 585-623.
    8. Philipp Eisenhauer & James J. Heckman & Edward Vytlacil, 2015. "The Generalized Roy Model and the Cost-Benefit Analysis of Social Programs," Journal of Political Economy, University of Chicago Press, vol. 123(2), pages 413-443.
    9. Gathmann, Christina & Vonnahme, Christina & Kim, Jongoh & Busse, Anna, 2021. "Marginal Returns to Citizenship and Educational Performance," CEPR Discussion Papers 16636, C.E.P.R. Discussion Papers.
    10. James J. Heckman, 2005. "Micro Data, Heterogeneity and the Evaluation of Public Policy Part 2," The American Economist, Sage Publications, vol. 49(1), pages 16-44, March.
    11. Huber, Martin & Wüthrich, Kaspar, 2017. "Evaluating local average and quantile treatment effects under endogeneity based on instruments: a review," FSES Working Papers 479, Faculty of Economics and Social Sciences, University of Freiburg/Fribourg Switzerland.
    12. Robert Collinson & John Eric Humphries & Nicholas S. Mader & Davin K. Reed & Daniel I. Tannenbaum & Winnie van Dijk, 2022. "Eviction and Poverty in American Cities," NBER Working Papers 30382, National Bureau of Economic Research, Inc.
    13. Patrick Gaulé, 2018. "Patents and the Success of Venture‐Capital Backed Startups: Using Examiner Assignment to Estimate Causal Effects," Journal of Industrial Economics, Wiley Blackwell, vol. 66(2), pages 350-376, June.
    14. Matthias Westphal & Daniel A Kamhöfer & Hendrik Schmitz, 2022. "Marginal College Wage Premiums Under Selection Into Employment," The Economic Journal, Royal Economic Society, vol. 132(646), pages 2231-2272.
    15. Cornelissen, Thomas & Dustmann, Christian & Raute, Anna & Schönberg, Uta, 2016. "From LATE to MTE: Alternative methods for the evaluation of policy interventions," Labour Economics, Elsevier, vol. 41(C), pages 47-60.
    16. Black, Dan A. & Joo, Joonhwi & LaLonde, Robert & Smith, Jeffrey A. & Taylor, Evan J., 2022. "Simple Tests for Selection: Learning More from Instrumental Variables," Labour Economics, Elsevier, vol. 79(C).
    17. Gerten, Elisa & Beckmann, Michael & Kräkel, Matthias, 2022. "Information and Communication Technology, Hierarchy, and Job Design," IZA Discussion Papers 15491, Institute of Labor Economics (IZA).
    18. Amanda E Kowalski, 2023. "Behaviour within a Clinical Trial and Implications for Mammography Guidelines," Review of Economic Studies, Oxford University Press, vol. 90(1), pages 432-462.
    19. Patrick Kline & Christopher R. Walters, 2019. "On Heckits, LATE, and Numerical Equivalence," Econometrica, Econometric Society, vol. 87(2), pages 677-696, March.
    20. D'Haultfoeuille, Xavier & Maurel, Arnaud, 2009. "Inference on a Generalized Roy Model, with an Application to Schooling Decisions in France," IZA Discussion Papers 4606, Institute of Labor Economics (IZA).

    More about this item

    JEL classification:

    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C26 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Instrumental Variables (IV) Estimation
    • C51 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Construction and Estimation
    • J15 - Labor and Demographic Economics - - Demographic Economics - - - Economics of Minorities, Races, Indigenous Peoples, and Immigrants; Non-labor Discrimination
    • J16 - Labor and Demographic Economics - - Demographic Economics - - - Economics of Gender; Non-labor Discrimination
    • J71 - Labor and Demographic Economics - - Labor Discrimination - - - Hiring and Firing
    • K14 - Law and Economics - - Basic Areas of Law - - - Criminal Law
    • K42 - Law and Economics - - Legal Procedure, the Legal System, and Illegal Behavior - - - Illegal Behavior and the Enforcement of Law

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:27802. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.