IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/xz9wr_v1.html
   My bibliography  Save this paper

Improving Compliance in Experimental Studies of Discrimination

Author

Listed:
  • Kaufman, Aaron

    (New York University Abu Dhabi)

  • Grumbach, Jake
  • Celaya, Christopher

Abstract

\noindent In experimental studies of discrimination, researchers often manipulate cues of social identities such as race to isolate their discriminatory effects, holding all else constant. However, these studies cannot distinguish between \textit{noncompliance}, in which respondents do not observe, acknowledge, or update their beliefs about the social identity being signaled, and \textit{nondiscrimination}, in which respondents update their beliefs based on the signal but do not discriminate. Focusing on experiments addressing racial discrimination, we find evidence from a pre-registered, nationally representative survey experiment that racialized name cues suffer from compliance as low as 33%, especially for Black treatments. Adding pre-tested racialized pictures and resume items improves compliance to 95% on average and reduces the variance in compliance across racial treatment groups. Our simulation studies show that this noncompliance tends to attenuate the estimated effects of race by as much as 85\%, implying that racial discrimination may be much deeper than decades of experiments have suggested.

Suggested Citation

  • Kaufman, Aaron & Grumbach, Jake & Celaya, Christopher, 2024. "Improving Compliance in Experimental Studies of Discrimination," OSF Preprints xz9wr_v1, Center for Open Science.
    • Handle: RePEc:osf:osfxxx:xz9wr_v1
    DOI: 10.31219/osf.io/xz9wr_v1
    as

    Download full text from publisher

    File URL: https://osf.io/download/6661fd6e6b6c8e273104cdd2/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/xz9wr_v1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Gaddis, S. Michael, 2015. "Discrimination in the Credential Society: An Audit Study of Race and College Selectivity in the Labor Market," SocArXiv 6qjue, Center for Open Science.
    2. Landgrave, Michelangelo & Weller, Nicholas, 2022. "Do Name-Based Treatments Violate Information Equivalence? Evidence from a Correspondence Audit Experiment," Political Analysis, Cambridge University Press, vol. 30(1), pages 142-148, January.
    3. Coppock, Alexander, 2019. "Avoiding Post-Treatment Bias in Audit Experiments," Journal of Experimental Political Science, Cambridge University Press, vol. 6(1), pages 1-4, April.
    4. Kane, John V. & Velez, Yamil R. & Barabas, Jason, 2023. "Analyze the attentive and bypass bias: mock vignette checks in survey experiments," Political Science Research and Methods, Cambridge University Press, vol. 11(2), pages 293-310, April.
    5. Marianne Bertrand & Sendhil Mullainathan, 2004. "Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination," American Economic Review, American Economic Association, vol. 94(4), pages 991-1013, September.
    6. Grogger, Jeffrey & Ridgeway, Greg, 2006. "Testing for Racial Profiling in Traffic Stops From Behind a Veil of Darkness," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 878-887, September.
    7. Dafoe, Allan & Zhang, Baobao & Caughey, Devin, 2018. "Information Equivalence in Survey Experiments," Political Analysis, Cambridge University Press, vol. 26(4), pages 399-416, October.
    8. Daniel M. Butler & David E. Broockman, 2011. "Do Politicians Racially Discriminate Against Constituents? A Field Experiment on State Legislators," American Journal of Political Science, John Wiley & Sons, vol. 55(3), pages 463-477, July.
    9. Kristof Heylen & Katleen Van den Broeck, 2016. "Discrimination and selection in the Belgian private rental market," Housing Studies, Taylor & Francis Journals, vol. 31(2), pages 223-236, March.
    10. Butler, Daniel M. & Homola, Jonathan, 2017. "An Empirical Justification for the Use of Racially Distinctive Names to Signal Race in Experiments," Political Analysis, Cambridge University Press, vol. 25(1), pages 122-130, January.
    11. William C. Horrace & Shawn M. Rohlin, 2016. "How Dark Is Dark? Bright Lights, Big City, Racial Profiling," The Review of Economics and Statistics, MIT Press, vol. 98(2), pages 226-232, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gaddis, S. Michael, 2018. "An Introduction to Audit Studies in the Social Sciences," SocArXiv e5hfc_v1, Center for Open Science.
    2. Gaddis, S. Michael, 2018. "An Introduction to Audit Studies in the Social Sciences," SocArXiv e5hfc, Center for Open Science.
    3. Kevin Lang & Ariella Kahn-Lang Spitzer, 2020. "Race Discrimination: An Economic Perspective," Journal of Economic Perspectives, American Economic Association, vol. 34(2), pages 68-89, Spring.
    4. Button, Patrick & Walker, Brigham, 2020. "Employment discrimination against Indigenous Peoples in the United States: Evidence from a field experiment," Labour Economics, Elsevier, vol. 65(C).
    5. Pfaff, Steven & Crabtree, Charles & Kern, Holger L. & Holbein, John B., 2018. "Does religious bias shape access to public services? A large-scale audit experiment among street-level bureaucrats," SocArXiv 9khds_v1, Center for Open Science.
    6. Felipe Goncalves & Steven Mello, 2017. "A Few Bad Apples? Racial Bias in Policing," Working Papers 608, Princeton University, Department of Economics, Industrial Relations Section..
    7. Madina Kurmangaliyeva & Matteo Sostero, 2022. "Walking while Black :Racial Gaps in Hit-and-Run Cases," Working Papers ECARES 2022-08, ULB -- Universite Libre de Bruxelles.
    8. Felipe Goncalves & Steven Mello, 2021. "A Few Bad Apples? Racial Bias in Policing," American Economic Review, American Economic Association, vol. 111(5), pages 1406-1441, May.
    9. Pfaff, Steven & Crabtree, Charles & Kern, Holger L. & Holbein, John B., 2018. "Does religious bias shape access to public services? A large-scale audit experiment among street-level bureaucrats," SocArXiv 9khds, Center for Open Science.
    10. Ingrid Gould Ellen & Stephen L. Ross, 2018. "Race and the City," Working Papers 2018-022, Human Capital and Economic Opportunity Working Group.
    11. Debopam Bhattacharya & Shin Kanaya & Margaret Stevens, 2017. "Are University Admissions Academically Fair?," The Review of Economics and Statistics, MIT Press, vol. 99(3), pages 449-464, July.
    12. Judd B. Kessler & Corinne Low & Colin D. Sullivan, 2019. "Incentivized Resume Rating: Eliciting Employer Preferences without Deception," American Economic Review, American Economic Association, vol. 109(11), pages 3713-3744, November.
    13. Baert, Stijn, 2017. "Hiring Discrimination: An Overview of (Almost) All Correspondence Experiments Since 2005," GLO Discussion Paper Series 61, Global Labor Organization (GLO).
    14. Druckman, James N. & Levy, Jeremy & Sands, Natalie, 2021. "Bias in education disability accommodations," Economics of Education Review, Elsevier, vol. 85(C).
    15. Michael Rochlitz & Evgeniya Mitrokhina & Irina Nizovkina, 2020. "Bureaucratic Discrimination in Electoral Authoritarian Regimes: Experimental Evidence from Russia," Bremen Papers on Economics & Innovation 2010, University of Bremen, Faculty of Business Studies and Economics.
    16. Marina Mileo Gorzig & Deborah Rho, 2022. "The effect of the 2016 United States presidential election on employment discrimination," Journal of Population Economics, Springer;European Society for Population Economics, vol. 35(1), pages 45-88, January.
    17. Kuhn, Florian & Chanci, Luis, 2024. "Racial disparities in labor outcomes: The effects of hiring discrimination over the business cycle," Economic Analysis and Policy, Elsevier, vol. 81(C), pages 801-817.
    18. Mladen Adamovic & Andreas Leibbrandt, 2024. "Is there a glass ceiling for ethnic minorities to enter leadership positions? Evidence from a large-scale field experiment with over 12,000 job applications," Monash Economics Working Papers 2024-06, Monash University, Department of Economics.
    19. Lippens, Louis & Vermeiren, Siel & Baert, Stijn, 2023. "The state of hiring discrimination: A meta-analysis of (almost) all recent correspondence experiments," European Economic Review, Elsevier, vol. 151(C).
    20. Makofske, Matthew, 2020. "Pretextual Traffic Stops and Racial Disparities in their Use," MPRA Paper 100792, University Library of Munich, Germany.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:xz9wr_v1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.