IDEAS home Printed from https://ideas.repec.org/a/nas/journl/v117y2020p7684-7689.html
   My bibliography  Save this article

Racial disparities in automated speech recognition

Author

Listed:
  • Allison Koenecke

    (Institute for Computational & Mathematical Engineering, Stanford University, Stanford, CA 94305)

  • Andrew Nam

    (Department of Psychology, Stanford University, Stanford, CA 94305)

  • Emily Lake

    (Department of Linguistics, Stanford University, Stanford, CA 94305)

  • Joe Nudell

    (Department of Management Science & Engineering, Stanford University, Stanford, CA 94305)

  • Minnie Quartey

    (Department of Linguistics, Georgetown University, Washington, DC 20057)

  • Zion Mengesha

    (Department of Linguistics, Stanford University, Stanford, CA 94305)

  • Connor Toups

    (Department of Linguistics, Stanford University, Stanford, CA 94305)

  • John R. Rickford

    (Department of Linguistics, Stanford University, Stanford, CA 94305)

  • Dan Jurafsky

    (Department of Linguistics, Stanford University, Stanford, CA 94305; Department of Computer Science, Stanford University, Stanford, CA 94305)

  • Sharad Goel

    (Department of Management Science & Engineering, Stanford University, Stanford, CA 94305)

Abstract

Automated speech recognition (ASR) systems, which use sophisticated machine-learning algorithms to convert spoken language to text, have become increasingly widespread, powering popular virtual assistants, facilitating automated closed captioning, and enabling digital dictation platforms for health care. Over the last several years, the quality of these systems has dramatically improved, due both to advances in deep learning and to the collection of large-scale datasets used to train the systems. There is concern, however, that these tools do not work equally well for all subgroups of the population. Here, we examine the ability of five state-of-the-art ASR systems—developed by Amazon, Apple, Google, IBM, and Microsoft—to transcribe structured interviews conducted with 42 white speakers and 73 black speakers. In total, this corpus spans five US cities and consists of 19.8 h of audio matched on the age and gender of the speaker. We found that all five ASR systems exhibited substantial racial disparities, with an average word error rate (WER) of 0.35 for black speakers compared with 0.19 for white speakers. We trace these disparities to the underlying acoustic models used by the ASR systems as the race gap was equally large on a subset of identical phrases spoken by black and white individuals in our corpus. We conclude by proposing strategies—such as using more diverse training datasets that include African American Vernacular English—to reduce these performance differences and ensure speech recognition technology is inclusive.

Suggested Citation

  • Allison Koenecke & Andrew Nam & Emily Lake & Joe Nudell & Minnie Quartey & Zion Mengesha & Connor Toups & John R. Rickford & Dan Jurafsky & Sharad Goel, 2020. "Racial disparities in automated speech recognition," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 117(14), pages 7684-7689, April.
  • Handle: RePEc:nas:journl:v:117:y:2020:p:7684-7689
    as

    Download full text from publisher

    File URL: http://www.pnas.org/content/117/14/7684.full
    Download Restriction: no
    ---><---

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Nils Köbis & Jean-François Bonnefon & Iyad Rahwan, 2021. "Bad machines corrupt good morals," Nature Human Behaviour, Nature, vol. 5(6), pages 679-685, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nas:journl:v:117:y:2020:p:7684-7689. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Eric Cain (email available below). General contact details of provider: http://www.pnas.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.