IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/xz9wr_v1.html
   My bibliography  Save this paper

Improving Compliance in Experimental Studies of Discrimination

Author

Listed:
  • Kaufman, Aaron

    (New York University Abu Dhabi)

  • Grumbach, Jake
  • Celaya, Christopher

Abstract

\noindent In experimental studies of discrimination, researchers often manipulate cues of social identities such as race to isolate their discriminatory effects, holding all else constant. However, these studies cannot distinguish between \textit{noncompliance}, in which respondents do not observe, acknowledge, or update their beliefs about the social identity being signaled, and \textit{nondiscrimination}, in which respondents update their beliefs based on the signal but do not discriminate. Focusing on experiments addressing racial discrimination, we find evidence from a pre-registered, nationally representative survey experiment that racialized name cues suffer from compliance as low as 33%, especially for Black treatments. Adding pre-tested racialized pictures and resume items improves compliance to 95% on average and reduces the variance in compliance across racial treatment groups. Our simulation studies show that this noncompliance tends to attenuate the estimated effects of race by as much as 85\%, implying that racial discrimination may be much deeper than decades of experiments have suggested.

Suggested Citation

  • Kaufman, Aaron & Grumbach, Jake & Celaya, Christopher, 2024. "Improving Compliance in Experimental Studies of Discrimination," OSF Preprints xz9wr_v1, Center for Open Science.
    • Handle: RePEc:osf:osfxxx:xz9wr_v1
    DOI: 10.31219/osf.io/xz9wr_v1
    as

    Download full text from publisher

    File URL: https://osf.io/download/6661fd6e6b6c8e273104cdd2/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/xz9wr_v1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Gaddis, S. Michael, 2015. "Discrimination in the Credential Society: An Audit Study of Race and College Selectivity in the Labor Market," SocArXiv 6qjue, Center for Open Science.
    2. Landgrave, Michelangelo & Weller, Nicholas, 2022. "Do Name-Based Treatments Violate Information Equivalence? Evidence from a Correspondence Audit Experiment," Political Analysis, Cambridge University Press, vol. 30(1), pages 142-148, January.
    3. Daniel M. Butler & David E. Broockman, 2011. "Do Politicians Racially Discriminate Against Constituents? A Field Experiment on State Legislators," American Journal of Political Science, John Wiley & Sons, vol. 55(3), pages 463-477, July.
    4. Kristof Heylen & Katleen Van den Broeck, 2016. "Discrimination and selection in the Belgian private rental market," Housing Studies, Taylor & Francis Journals, vol. 31(2), pages 223-236, March.
    5. Marianne Bertrand & Sendhil Mullainathan, 2004. "Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination," American Economic Review, American Economic Association, vol. 94(4), pages 991-1013, September.
    6. Grogger, Jeffrey & Ridgeway, Greg, 2006. "Testing for Racial Profiling in Traffic Stops From Behind a Veil of Darkness," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 878-887, September.
    7. Coppock, Alexander, 2019. "Avoiding Post-Treatment Bias in Audit Experiments," Journal of Experimental Political Science, Cambridge University Press, vol. 6(1), pages 1-4, April.
    8. Kane, John V. & Velez, Yamil R. & Barabas, Jason, 2023. "Analyze the attentive and bypass bias: mock vignette checks in survey experiments," Political Science Research and Methods, Cambridge University Press, vol. 11(2), pages 293-310, April.
    9. Dafoe, Allan & Zhang, Baobao & Caughey, Devin, 2018. "Information Equivalence in Survey Experiments," Political Analysis, Cambridge University Press, vol. 26(4), pages 399-416, October.
    10. Butler, Daniel M. & Homola, Jonathan, 2017. "An Empirical Justification for the Use of Racially Distinctive Names to Signal Race in Experiments," Political Analysis, Cambridge University Press, vol. 25(1), pages 122-130, January.
    11. William C. Horrace & Shawn M. Rohlin, 2016. "How Dark Is Dark? Bright Lights, Big City, Racial Profiling," The Review of Economics and Statistics, MIT Press, vol. 98(2), pages 226-232, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:osf:socarx:e5hfc_v1 is not listed on IDEAS
    2. Gaddis, S. Michael, 2018. "An Introduction to Audit Studies in the Social Sciences," SocArXiv e5hfc, Center for Open Science.
    3. Felipe Goncalves & Steven Mello, 2021. "A Few Bad Apples? Racial Bias in Policing," American Economic Review, American Economic Association, vol. 111(5), pages 1406-1441, May.
    4. Kevin Lang & Ariella Kahn-Lang Spitzer, 2020. "Race Discrimination: An Economic Perspective," Journal of Economic Perspectives, American Economic Association, vol. 34(2), pages 68-89, Spring.
    5. Button, Patrick & Walker, Brigham, 2020. "Employment discrimination against Indigenous Peoples in the United States: Evidence from a field experiment," Labour Economics, Elsevier, vol. 65(C).
    6. repec:osf:socarx:9khds_v1 is not listed on IDEAS
    7. Madina Kurmangaliyeva & Matteo Sostero, 2022. "Walking while Black :Racial Gaps in Hit-and-Run Cases," Working Papers ECARES 2022-08, ULB -- Universite Libre de Bruxelles.
    8. Pfaff, Steven & Crabtree, Charles & Kern, Holger L. & Holbein, John B., 2018. "Does religious bias shape access to public services? A large-scale audit experiment among street-level bureaucrats," SocArXiv 9khds, Center for Open Science.
    9. Druckman, James N. & Levy, Jeremy & Sands, Natalie, 2021. "Bias in education disability accommodations," Economics of Education Review, Elsevier, vol. 85(C).
    10. Kuhn, Florian & Chanci, Luis, 2024. "Racial disparities in labor outcomes: The effects of hiring discrimination over the business cycle," Economic Analysis and Policy, Elsevier, vol. 81(C), pages 801-817.
    11. Mladen Adamovic & Andreas Leibbrandt, 2024. "Is there a glass ceiling for ethnic minorities to enter leadership positions? Evidence from a large-scale field experiment with over 12,000 job applications," Monash Economics Working Papers 2024-06, Monash University, Department of Economics.
    12. Lippens, Louis & Vermeiren, Siel & Baert, Stijn, 2023. "The state of hiring discrimination: A meta-analysis of (almost) all recent correspondence experiments," European Economic Review, Elsevier, vol. 151(C).
    13. Makofske, Matthew, 2020. "Pretextual Traffic Stops and Racial Disparities in their Use," MPRA Paper 100792, University Library of Munich, Germany.
    14. Flage, Alexandre, 2018. "Ethnic and gender discrimination in the rental housing market: Evidence from a meta-analysis of correspondence tests, 2006–2017," Journal of Housing Economics, Elsevier, vol. 41(C), pages 251-273.
    15. Melo, Vitor & Rocha, Hugo Vaca Pereira & Sigaud, Liam & Warren, Patrick L. & Gaddis, S. Michael, 2024. "Understanding Discrimination in College Admissions: A Field Experiment," SocArXiv 5ctms, Center for Open Science.
    16. Luca Fumarco & Benjamin Harrell & Patrick Button & David Schwegman & E Dils, 2020. "Gender Identity, Race, and Ethnicity-based Discrimination in Access to Mental Health Care: Evidence from an Audit Correspondence Field Experiment," NBER Working Papers 28164, National Bureau of Economic Research, Inc.
    17. Pauline Grosjean & Federico Masera & Hasin Yousaf, 2023. "Inflammatory Political Campaigns and Racial Bias in Policing," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(1), pages 413-463.
    18. Anwar Shamena & Fang Hanming, 2012. "Testing for the Role of Prejudice in Emergency Departments Using Bounceback Rates," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 13(3), pages 1-49, December.
    19. Gaddis, S. Michael & Quadlin, Natasha & Larsen, Edvard Nergård & Crabtree, Charles & Holbein, John B., 2025. "Missing Results of Discrimination: A Systematically Comparative Meta-Analysis and Re-Examinations of Racial, Gender, and Intersectional Discrimination using Correspondence Audits," SocArXiv nd2y6_v1, Center for Open Science.
    20. Matthias Stefan & Felix Holzmeister & Alexander Müllauer & Michael Kirchler, 2018. "Ethnical discrimination in Europe: Field evidence from the finance industry," PLOS ONE, Public Library of Science, vol. 13(1), pages 1-7, January.
    21. Brady P. Horn & Jill J. Mccluskey & Ron C. Mittelhammer, 2014. "Quantifying Bias In Driving-Under-The-Influence Enforcement," Economic Inquiry, Western Economic Association International, vol. 52(1), pages 269-284, January.
    22. Brent B Allred & Michael G Findley & Daniel Nielson & J C Sharman, 2017. "Anonymous shell companies: A global audit study and field experiment in 176 countries," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 48(5), pages 596-619, July.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:xz9wr_v1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.