Identifying bias in models that detect vocal fold paralysis from audio recordings using explainable machine learning and clinician ratings

Detecting voice disorders from voice recordings could allow for frequent, remote, and low-cost screening before costly clinical visits and a more invasive laryngoscopy examination. Our goals were to detect unilateral vocal fold paralysis (UVFP) from voice recordings using machine learning, to identify which acoustic variables were important for prediction to increase trust, and to determine model performance relative to clinician performance. Patients with confirmed UVFP through endoscopic examination (N = 77) and controls with normal voices matched for age and sex (N = 77) were included. Voice samples were elicited by reading the Rainbow Passage and sustaining phonation of the vowel "a". Four machine learning models of differing complexity were used. SHapley Additive exPlanations (SHAP) was used to identify important features. The highest median bootstrapped ROC AUC score was 0.87 and beat clinician’s performance (range: 0.74–0.81) based on the recordings. Recording durations were different between UVFP recordings and controls due to how that data was originally processed when storing, which we can show can classify both groups. And counterintuitively, many UVFP recordings had higher intensity than controls, when UVFP patients tend to have weaker voices, revealing a dataset-specific bias which we mitigate in an additional analysis. We demonstrate that recording biases in audio duration and intensity created dataset-specific differences between patients and controls, which models used to improve classification. Furthermore, clinician’s ratings provide further evidence that patients were over-projecting their voices and being recorded at a higher amplitude signal than controls. Interestingly, after matching audio duration and removing variables associated with intensity in order to mitigate the biases, the models were able to achieve a similar high performance. We provide a set of recommendations to avoid bias when building and evaluating machine learning models for screening in laryngology.Author summary: The diagnosis of certain voice disorders can involve costly and time-consuming methods such as video laryngoscopy. An alternative is to screen using machine learning models that predict risk given just a short audio recording from a mobile device. However, these models can be biased if they detect recording idiosyncrasies of a given dataset that would not generalize to new samples with a different recording protocol, making the model unusable. These types of biases are not always evaluated in clinical machine learning studies. We found that a model we trained to detect unilateral vocal fold paralysis from healthy voices from brief audio recordings was biased: patients with a softer voice may have been induced to over-project their voice to obtain clearer recordings or the gain on the microphone may have been increased only for these participants, creating a bias that is unlikely to generalize. We demonstrate how to detect such biases using explainable machine learning and clinician ratings as well as how to potentially mitigate the effect of the bias. We also provide recommendations for identifying and mitigating bias in machine learning models that use audio recordings for screening in laryngology in general.