IDEAS home Printed from https://ideas.repec.org/p/arx/papers/1907.06622.html
   My bibliography  Save this paper

Audits as Evidence: Experiments, Ensembles, and Enforcement

Author

Listed:
  • Patrick Kline
  • Christopher Walters

Abstract

We develop tools for utilizing correspondence experiments to detect illegal discrimination by individual employers. Employers violate US employment law if their propensity to contact applicants depends on protected characteristics such as race or sex. We establish identification of higher moments of the causal effects of protected characteristics on callback rates as a function of the number of fictitious applications sent to each job ad. These moments are used to bound the fraction of jobs that illegally discriminate. Applying our results to three experimental datasets, we find evidence of significant employer heterogeneity in discriminatory behavior, with the standard deviation of gaps in job-specific callback probabilities across protected groups averaging roughly twice the mean gap. In a recent experiment manipulating racially distinctive names, we estimate that at least 85% of jobs that contact both of two white applications and neither of two black applications are engaged in illegal discrimination. To assess the tradeoff between type I and II errors presented by these patterns, we consider the performance of a series of decision rules for investigating suspicious callback behavior under a simple two-type model that rationalizes the experimental data. Though, in our preferred specification, only 17% of employers are estimated to discriminate on the basis of race, we find that an experiment sending 10 applications to each job would enable accurate detection of 7-10% of discriminators while falsely accusing fewer than 0.2% of non-discriminators. A minimax decision rule acknowledging partial identification of the joint distribution of callback rates yields higher error rates but more investigations than our baseline two-type model. Our results suggest illegal labor market discrimination can be reliably monitored with relatively small modifications to existing audit designs.

Suggested Citation

  • Patrick Kline & Christopher Walters, 2019. "Audits as Evidence: Experiments, Ensembles, and Enforcement," Papers 1907.06622, arXiv.org, revised Jul 2019.
  • Handle: RePEc:arx:papers:1907.06622
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/1907.06622
    File Function: Latest version
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Raj Chetty & Nathaniel Hendren, 2018. "The Impacts of Neighborhoods on Intergenerational Mobility II: County-Level Estimates," The Quarterly Journal of Economics, Oxford University Press, vol. 133(3), pages 1163-1228.
    2. Eva O. Arceo-Gomez & Raymundo M. Campos-Vazquez, 2014. "Race and Marriage in the Labor Market: A Discrimination Correspondence Study in a Developing Country," American Economic Review, American Economic Association, vol. 104(5), pages 376-380, May.
    3. Thomas J. Kane & Douglas O. Staiger, 2008. "Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation," NBER Working Papers 14607, National Bureau of Economic Research, Inc.
    4. James J. Heckman & Jeffrey Smith & Nancy Clements, 1997. "Making The Most Out Of Programme Evaluations and Social Experiments: Accounting For Heterogeneity in Programme Impacts," Review of Economic Studies, Oxford University Press, vol. 64(4), pages 487-535.
    5. Toru Kitagawa & Aleksey Tetenov, 2018. "Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice," Econometrica, Econometric Society, vol. 86(2), pages 591-616, March.
    6. Roland G. Fryer & Steven D. Levitt, 2004. "The Causes and Consequences of Distinctively Black Names," The Quarterly Journal of Economics, Oxford University Press, vol. 119(3), pages 767-805.
    7. Henry S. Farber & Dan Silverman & Till von Wachter, 2016. "Determinants of Callbacks to Job Applications: An Audit Study," American Economic Review, American Economic Association, vol. 106(5), pages 314-318, May.
    8. Jon Kleinberg & Jens Ludwig & Sendhil Mullainathan & Cass R. Sunstein, 2019. "Discrimination In The Age Of Algorithms," NBER Working Papers 25548, National Bureau of Economic Research, Inc.
    9. repec:hrv:faseco:30749073 is not listed on IDEAS
    10. Manski, Charles F., 2000. "Identification problems and decisions under ambiguity: Empirical analysis of treatment response and normative analysis of treatment choice," Journal of Econometrics, Elsevier, vol. 95(2), pages 415-442, April.
    11. Yusuke Narita & Shota Yasui & Kohei Yata, 2018. "Efficient Counterfactual Learning from Bandit Feedback," Cowles Foundation Discussion Papers 2155, Cowles Foundation for Research in Economics, Yale University.
    12. Raj Chetty & John N. Friedman & Jonah E. Rockoff, 2014. "Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates," American Economic Review, American Economic Association, vol. 104(9), pages 2593-2632, September.
    13. Joshua D. Angrist & Peter D. Hull & Parag A. Pathak & Christopher R. Walters, 2017. "Erratum to “Leveraging Lotteries for School Value-Added: Testing and Estimation”," The Quarterly Journal of Economics, Oxford University Press, vol. 132(4), pages 2061-2062.
    14. Timothy B. Armstrong, 2014. "Adaptive Testing on a Regression Function at a Point," Cowles Foundation Discussion Papers 1957, Cowles Foundation for Research in Economics, Yale University, revised Oct 2014.
    15. Marianne Bertrand & Sendhil Mullainathan, 2004. "Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination," American Economic Review, American Economic Association, vol. 94(4), pages 991-1013, September.
    16. Angrist, Joshua D., 2014. "The perils of peer effects," Labour Economics, Elsevier, vol. 30(C), pages 98-108.
    17. Yusuke Narita, 2018. "Experiment-as-Market: Incorporating Welfare into Randomized Controlled Trials," Cowles Foundation Discussion Papers 2127r, Cowles Foundation for Research in Economics, Yale University, revised May 2019.
    18. Joshua D. Angrist & Peter D. Hull & Parag A. Pathak & Christopher R. Walters, 2017. "Leveraging Lotteries for School Value-Added: Testing and Estimation," The Quarterly Journal of Economics, Oxford University Press, vol. 132(2), pages 871-919.
    19. Jonathan Guryan & Kerwin Kofi Charles, 2013. "Taste‐based or Statistical Discrimination: The Economics of Discrimination Returns to its Roots," Economic Journal, Royal Economic Society, vol. 123(11), pages 417-432, November.
    20. Altonji, Joseph G. & Blank, Rebecca M., 1999. "Race and gender in the labor market," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 3, chapter 48, pages 3143-3259, Elsevier.
    21. Victor Chernozhukov & Whitney K. Newey & Andres Santos, 2015. "Constrained conditional moment restriction models," CeMMAP working papers CWP59/15, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    22. Nunley John M. & Pugh Adam & Romero Nicholas & Seals R. Alan, 2015. "Racial Discrimination in the Labor Market for Recent College Graduates: Evidence from a Field Experiment," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 15(3), pages 1093-1125, July.
    23. Kerwin Kofi Charles & Jonathan Guryan, 2008. "Prejudice and Wages: An Empirical Assessment of Becker's The Economics of Discrimination," Journal of Political Economy, University of Chicago Press, vol. 116(5), pages 773-809, October.
    24. Efron, Bradley, 2004. "Large-Scale Simultaneous Hypothesis Testing: The Choice of a Null Hypothesis," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 96-104, January.
    25. Efron B. & Tibshirani R. & Storey J.D. & Tusher V., 2001. "Empirical Bayes Analysis of a Microarray Experiment," Journal of the American Statistical Association, American Statistical Association, vol. 96, pages 1151-1160, December.
    26. Raj Chetty & John N. Friedman & Nathaniel Hendren & Maggie R. Jones & Sonya R. Porter, 2018. "The Opportunity Atlas: Mapping the Childhood Roots of Social Mobility," NBER Working Papers 25147, National Bureau of Economic Research, Inc.
    27. Müller, Ulrich K. & Wang, Yulong, 2019. "Nearly weighted risk minimal unbiased estimation," Journal of Econometrics, Elsevier, vol. 209(1), pages 18-34.
    28. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, December.
    29. Pietro Tebaldi & Alexander Torgovitsky & Hanbin Yang, 2019. "Nonparametric Estimates of Demand in the California Health Insurance Exchange," NBER Working Papers 25827, National Bureau of Economic Research, Inc.
    30. John D. Storey, 2002. "A direct approach to false discovery rates," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 64(3), pages 479-498, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Peter Christensen & Ignacio Sarmiento-Barbieri & Christopher Timmins, 2022. "Housing Discrimination and the Toxics Exposure Gap in the United States: Evidence from the Rental Market," The Review of Economics and Statistics, MIT Press, vol. 104(4), pages 807-818, October.
    2. Carl Lieberman, 2020. "Variation in Racial Disparities in Police Use of Force," Working Papers 639, Princeton University, Department of Economics, Industrial Relations Section..

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Patrick Kline & Christopher Walters, 2021. "Reasonable Doubt: Experimental Detection of Job‐Level Employment Discrimination," Econometrica, Econometric Society, vol. 89(2), pages 765-792, March.
    2. Rothstein, J & von Wachter, T, 2016. "Social Experiments in the Labor Market," Department of Economics, Working Paper Series qt7957p9g6, Department of Economics, Institute for Business and Economic Research, UC Berkeley.
    3. Joanna N. Lahey & Douglas R. Oxley, 2021. "Discrimination at the Intersection of Age, Race, and Gender: Evidence from an Eye‐Tracking Experiment," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 40(4), pages 1083-1119, September.
    4. Baert, Stijn, 2015. "Hiring a Homosexual, Taking a Risk? A Lab Experiment on Employment Discrimination and Risk Aversion," IZA Discussion Papers 9536, Institute of Labor Economics (IZA).
    5. Timothy B. Armstrong & Michal Kolesár & Mikkel Plagborg-Møller, 2022. "Robust Empirical Bayes Confidence Intervals," Working Papers 2022-27, Princeton University. Economics Department..
    6. Patrick Kline & Evan K Rose & Christopher R Walters, 2022. "Systemic Discrimination Among Large U.S. Employers [“Teachers and Student Achievement in the Chicago Public High Schools,”]," The Quarterly Journal of Economics, Oxford University Press, vol. 137(4), pages 1963-2036.
    7. Timothy B. Armstrong & Michal Koles'ar & Mikkel Plagborg-M{o}ller, 2020. "Robust Empirical Bayes Confidence Intervals," Papers 2004.03448, arXiv.org, revised May 2022.
    8. Chowdhury, Shyamal & Ooi, Evarn & Slonim, Robert, 2017. "Racial discrimination and white first name adoption: a field experiment in the Australian labour market," Working Papers 2017-15, University of Sydney, School of Economics.
    9. Morten Størling Hedegaard & Jean-Robert Tyran, 2018. "The Price of Prejudice," American Economic Journal: Applied Economics, American Economic Association, vol. 10(1), pages 40-63, January.
    10. Hinnerich, Björn Tyrefors & Vlachos, Jonas, 2017. "The impact of upper-secondary voucher school attendance on student achievement. Swedish evidence using external and internal evaluations," Labour Economics, Elsevier, vol. 47(C), pages 1-14.
    11. Bertrand, Marianne & Duflo, Esther, 2016. "Field Experiments on Discrimination," CEPR Discussion Papers 11123, C.E.P.R. Discussion Papers.
    12. John M. Nunley & Mark F. Owens & R. Stephen Howard, 2010. "The Effects of Competition and Information on Racial Discrimination: Evidence from a Field Experiment," Working Papers 201007, Middle Tennessee State University, Department of Economics and Finance.
    13. Naven, Matthew, 2019. "Human-Capital Formation During Childhood and Adolescence: Evidence from School Quality and Postsecondary Success in California," MPRA Paper 97716, University Library of Munich, Germany.
    14. Gaddis, S. Michael, 2018. "An Introduction to Audit Studies in the Social Sciences," SocArXiv e5hfc, Center for Open Science.
    15. Naven, Matthew, 2020. "Within-School Heterogeneity in Quality: Do Schools Provide Equal Value Added to All Students?," MPRA Paper 100123, University Library of Munich, Germany.
    16. Islam, Asadul & Pakrashi, Debayan & Wang, Liang Choon & Zenou, Yves, 2018. "Determining the Extent of Statistical Discrimination: Evidence from a field experiment in India," CEPR Discussion Papers 12955, C.E.P.R. Discussion Papers.
    17. Bar, M.; & Bakx, P.; & Wouterse, B.; & van Doorslaer, Eddy.;, 2022. "Estimating the health value added by nursing homes," Health, Econometrics and Data Group (HEDG) Working Papers 22/12, HEDG, c/o Department of Economics, University of York.
    18. Drydakis, Nick, 2017. "Measuring labour differences between natives, non-natives, and natives with an ethnic-minority background," Economics Letters, Elsevier, vol. 161(C), pages 27-30.
    19. Manuel Arellano & Stéphane Bonhomme, 2019. "Recovering Latent Variables by Matching," Working Papers wp2019_1914, CEMFI.
    20. Hermann, Zoltán & Horváth, Hedvig, 2022. "Tanári eredményesség és tanár-diák összepárosítás az általános iskolákban. Empirikus mintázatok három magyarországi tankerület adatai alapján [Teacher effectiveness and teacher-student matching in ," Közgazdasági Szemle (Economic Review - monthly of the Hungarian Academy of Sciences), Közgazdasági Szemle Alapítvány (Economic Review Foundation), vol. 0(11), pages 1377-1406.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:1907.06622. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: http://arxiv.org/ .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.