IDEAS home Printed from
   My bibliography  Save this paper

Audits as Evidence: Experiments, Ensembles, and Enforcement


  • Patrick Kline
  • Christopher Walters


We develop tools for utilizing correspondence experiments to detect illegal discrimination by individual employers. Employers violate US employment law if their propensity to contact applicants depends on protected characteristics such as race or sex. We establish identification of higher moments of the causal effects of protected characteristics on callback rates as a function of the number of fictitious applications sent to each job ad. These moments are used to bound the fraction of jobs that illegally discriminate. Applying our results to three experimental datasets, we find evidence of significant employer heterogeneity in discriminatory behavior, with the standard deviation of gaps in job-specific callback probabilities across protected groups averaging roughly twice the mean gap. In a recent experiment manipulating racially distinctive names, we estimate that at least 85% of jobs that contact both of two white applications and neither of two black applications are engaged in illegal discrimination. To assess the tradeoff between type I and II errors presented by these patterns, we consider the performance of a series of decision rules for investigating suspicious callback behavior under a simple two-type model that rationalizes the experimental data. Though, in our preferred specification, only 17% of employers are estimated to discriminate on the basis of race, we find that an experiment sending 10 applications to each job would enable accurate detection of 7-10% of discriminators while falsely accusing fewer than 0.2% of non-discriminators. A minimax decision rule acknowledging partial identification of the joint distribution of callback rates yields higher error rates but more investigations than our baseline two-type model. Our results suggest illegal labor market discrimination can be reliably monitored with relatively small modifications to existing audit designs.

Suggested Citation

  • Patrick Kline & Christopher Walters, 2019. "Audits as Evidence: Experiments, Ensembles, and Enforcement," Papers 1907.06622,, revised Jul 2019.
  • Handle: RePEc:arx:papers:1907.06622

    Download full text from publisher

    File URL:
    File Function: Latest version
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    1. Angrist, Joshua D., 2014. "The perils of peer effects," Labour Economics, Elsevier, vol. 30(C), pages 98-108.
    2. Raj Chetty & Nathaniel Hendren, 2018. "The Impacts of Neighborhoods on Intergenerational Mobility II: County-Level Estimates," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 133(3), pages 1163-1228.
    3. Eva O. Arceo-Gomez & Raymundo M. Campos-Vazquez, 2014. "Race and Marriage in the Labor Market: A Discrimination Correspondence Study in a Developing Country," American Economic Review, American Economic Association, vol. 104(5), pages 376-380, May.
    4. Yusuke Narita, 2018. "Experiment-as-Market: Incorporating Welfare into Randomized Controlled Trials," Cowles Foundation Discussion Papers 2127r, Cowles Foundation for Research in Economics, Yale University, revised May 2019.
    5. Thomas J. Kane & Douglas O. Staiger, 2008. "Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation," NBER Working Papers 14607, National Bureau of Economic Research, Inc.
    6. Joshua D. Angrist & Peter D. Hull & Parag A. Pathak & Christopher R. Walters, 2017. "Leveraging Lotteries for School Value-Added: Testing and Estimation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 132(2), pages 871-919.
    7. Toru Kitagawa & Aleksey Tetenov, 2018. "Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice," Econometrica, Econometric Society, vol. 86(2), pages 591-616, March.
    8. Jonathan Guryan & Kerwin Kofi Charles, 2013. "Taste‐based or Statistical Discrimination: The Economics of Discrimination Returns to its Roots," Economic Journal, Royal Economic Society, vol. 123(11), pages 417-432, November.
    9. Altonji, Joseph G. & Blank, Rebecca M., 1999. "Race and gender in the labor market," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 3, chapter 48, pages 3143-3259, Elsevier.
    10. Victor Chernozhukov & Whitney K. Newey & Andres Santos, 2023. "Constrained Conditional Moment Restriction Models," Econometrica, Econometric Society, vol. 91(2), pages 709-736, March.
    11. Nunley John M. & Pugh Adam & Romero Nicholas & Seals R. Alan, 2015. "Racial Discrimination in the Labor Market for Recent College Graduates: Evidence from a Field Experiment," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 15(3), pages 1093-1125, July.
    12. James J. Heckman & Jeffrey Smith & Nancy Clements, 1997. "Making The Most Out Of Programme Evaluations and Social Experiments: Accounting For Heterogeneity in Programme Impacts," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 487-535.
    13. Roland G. Fryer & Steven D. Levitt, 2004. "The Causes and Consequences of Distinctively Black Names," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(3), pages 767-805.
    14. Henry S. Farber & Dan Silverman & Till von Wachter, 2016. "Determinants of Callbacks to Job Applications: An Audit Study," American Economic Review, American Economic Association, vol. 106(5), pages 314-318, May.
    15. Jon Kleinberg & Jens Ludwig & Sendhil Mullainathan & Cass R. Sunstein, 2019. "Discrimination In The Age Of Algorithms," NBER Working Papers 25548, National Bureau of Economic Research, Inc.
    16. Kerwin Kofi Charles & Jonathan Guryan, 2008. "Prejudice and Wages: An Empirical Assessment of Becker's The Economics of Discrimination," Journal of Political Economy, University of Chicago Press, vol. 116(5), pages 773-809, October.
    17. Raj Chetty & John N. Friedman & Jonah E. Rockoff, 2014. "Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates," American Economic Review, American Economic Association, vol. 104(9), pages 2593-2632, September.
    18. Manski, Charles F., 2000. "Identification problems and decisions under ambiguity: Empirical analysis of treatment response and normative analysis of treatment choice," Journal of Econometrics, Elsevier, vol. 95(2), pages 415-442, April.
    19. Yusuke Narita & Shota Yasui & Kohei Yata, 2018. "Efficient Counterfactual Learning from Bandit Feedback," Cowles Foundation Discussion Papers 2155, Cowles Foundation for Research in Economics, Yale University.
    20. Efron, Bradley, 2004. "Large-Scale Simultaneous Hypothesis Testing: The Choice of a Null Hypothesis," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 96-104, January.
    21. Joshua D. Angrist & Peter D. Hull & Parag A. Pathak & Christopher R. Walters, 2017. "Erratum to “Leveraging Lotteries for School Value-Added: Testing and Estimation”," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 132(4), pages 2061-2062.
    22. Marianne Bertrand & Sendhil Mullainathan, 2004. "Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination," American Economic Review, American Economic Association, vol. 94(4), pages 991-1013, September.
    23. Efron B. & Tibshirani R. & Storey J.D. & Tusher V., 2001. "Empirical Bayes Analysis of a Microarray Experiment," Journal of the American Statistical Association, American Statistical Association, vol. 96, pages 1151-1160, December.
    24. Timothy B. Armstrong, 2014. "Adaptive Testing on a Regression Function at a Point," Cowles Foundation Discussion Papers 1957, Cowles Foundation for Research in Economics, Yale University, revised Oct 2014.
    25. Pietro Tebaldi & Alexander Torgovitsky & Hanbin Yang, 2023. "Nonparametric Estimates of Demand in the California Health Insurance Exchange," Econometrica, Econometric Society, vol. 91(1), pages 107-146, January.
    26. Raj Chetty & John N. Friedman & Nathaniel Hendren & Maggie R. Jones & Sonya R. Porter, 2018. "The Opportunity Atlas: Mapping the Childhood Roots of Social Mobility," NBER Working Papers 25147, National Bureau of Economic Research, Inc.
    27. Müller, Ulrich K. & Wang, Yulong, 2019. "Nearly weighted risk minimal unbiased estimation," Journal of Econometrics, Elsevier, vol. 209(1), pages 18-34.
    28. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, November.
    29. John D. Storey, 2002. "A direct approach to false discovery rates," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 64(3), pages 479-498, August.
    Full references (including those not matched with items on IDEAS)


    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.

    Cited by:

    1. Peter Christensen & Ignacio Sarmiento-Barbieri & Christopher Timmins, 2022. "Housing Discrimination and the Toxics Exposure Gap in the United States: Evidence from the Rental Market," The Review of Economics and Statistics, MIT Press, vol. 104(4), pages 807-818, October.
    2. Carl Lieberman, 2020. "Variation in Racial Disparities in Police Use of Force," Working Papers 639, Princeton University, Department of Economics, Industrial Relations Section..
    3. Lieberman, Carl, 2024. "Variation in racial disparities in police use of force," Journal of Urban Economics, Elsevier, vol. 141(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Patrick Kline & Christopher Walters, 2021. "Reasonable Doubt: Experimental Detection of Job‐Level Employment Discrimination," Econometrica, Econometric Society, vol. 89(2), pages 765-792, March.
    2. Patrick Kline & Evan K Rose & Christopher R Walters, 2022. "Systemic Discrimination Among Large U.S. Employers [“Teachers and Student Achievement in the Chicago Public High Schools,”]," The Quarterly Journal of Economics, Oxford University Press, vol. 137(4), pages 1963-2036.
    3. Patrick Kline & Evan K Rose & Christopher R Walters, 2023. "Systemic Discrimination Among Large U.S. Employers," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 137(4), pages 1963-2036.
    4. Rothstein, Jesse & Von Wachter, Till, 2016. "Social Experiments in the Labor Market," Department of Economics, Working Paper Series qt7957p9g6, Department of Economics, Institute for Business and Economic Research, UC Berkeley.
    5. Joanna N. Lahey & Douglas R. Oxley, 2021. "Discrimination at the Intersection of Age, Race, and Gender: Evidence from an Eye‐Tracking Experiment," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 40(4), pages 1083-1119, September.
    6. Baert, Stijn, 2015. "Hiring a Homosexual, Taking a Risk? A Lab Experiment on Employment Discrimination and Risk Aversion," IZA Discussion Papers 9536, Institute of Labor Economics (IZA).
    7. Timothy B. Armstrong & Michal Kolesár & Mikkel Plagborg‐Møller, 2022. "Robust Empirical Bayes Confidence Intervals," Econometrica, Econometric Society, vol. 90(6), pages 2567-2602, November.
    8. Patrick Kline & Evan K Rose & Christopher R Walters, 2023. "Systemic Discrimination Among Large U.S. Employers," Journal of Economic Geography, Oxford University Press, vol. 137(4), pages 1963-2036.
    9. Timothy B. Armstrong & Michal Koles'ar & Mikkel Plagborg-M{o}ller, 2020. "Robust Empirical Bayes Confidence Intervals," Papers 2004.03448,, revised May 2022.
    10. Chowdhury, Shyamal & Ooi, Evarn & Slonim, Robert, 2017. "Racial discrimination and white first name adoption: a field experiment in the Australian labour market," Working Papers 2017-15, University of Sydney, School of Economics.
    11. Morten Størling Hedegaard & Jean-Robert Tyran, 2018. "The Price of Prejudice," American Economic Journal: Applied Economics, American Economic Association, vol. 10(1), pages 40-63, January.
    12. Hinnerich, Björn Tyrefors & Vlachos, Jonas, 2017. "The impact of upper-secondary voucher school attendance on student achievement. Swedish evidence using external and internal evaluations," Labour Economics, Elsevier, vol. 47(C), pages 1-14.
    13. Siddique, Abu & Vlassopoulos, Michael & Zenou, Yves, 2023. "Market competition and discrimination," European Economic Review, Elsevier, vol. 152(C).
    14. Marianne Bertrand & Esther Duflo, 2016. "Field Experiments on Discrimination," NBER Working Papers 22014, National Bureau of Economic Research, Inc.
    15. John M. Nunley & Mark F. Owens & R. Stephen Howard, 2010. "The Effects of Competition and Information on Racial Discrimination: Evidence from a Field Experiment," Working Papers 201007, Middle Tennessee State University, Department of Economics and Finance.
    16. Raffaella Giacomini & Sokbae Lee & Silvia Sarpietro, 2023. "A Robust Method for Microforecasting and Estimation of Random Effects," Working Paper Series WP 2023-26, Federal Reserve Bank of Chicago.
    17. Christine Mulhern & Isaac M. Opper, 2021. "Measuring and Summarizing the Multiple Dimensions of Teacher Effectiveness," CESifo Working Paper Series 9263, CESifo.
    18. Lenard, Matthew A. & Silliman, Mikko, 2024. "Informal Social Interactions, Academic Achievement and Behavior: Evidence from Peers on the School Bus," IZA Discussion Papers 16982, Institute of Labor Economics (IZA).
    19. Naven, Matthew, 2019. "Human-Capital Formation During Childhood and Adolescence: Evidence from School Quality and Postsecondary Success in California," MPRA Paper 97716, University Library of Munich, Germany.
    20. Gaddis, S. Michael, 2018. "An Introduction to Audit Studies in the Social Sciences," SocArXiv e5hfc, Center for Open Science.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:1907.06622. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.