Author
Listed:
- Aakash Gupta
- Umme Salma M. Pirzada
Abstract
Conventional automated systems for screening for depression leverage speech/text-based features, which makes such systems sensitive to external noises, potential mistakes in automatic speech recognition (ASR), and other modality-related limitations. Besides, most current approaches fail to provide any form of uncertainty estimates, properly calibrated output probabilities, and explanation capabilities for their predictions, limiting the use of these tools within a clinical environment. In this work, we present an approach for building trustworthy depression screening systems based on a fusion of acoustic and linguistic features using a novel cross-attention-based method. Specifically, self-supervised learning techniques like wav2vec 2.0 and HuBERT models are used for extracting acoustic features from raw audio recordings. For text processing, our framework leverages DistilBERT and RoBERTa language representation models. By employing a multi-head cross-attention module, we allow our model to effectively exploit interactions between linguistic content and acoustic features. Predictive uncertainty estimates are produced by incorporating Monte Carlo dropout into the model architecture. Temperature scaling is applied for proper calibration of output probabilities. Token-level attributions are used for explaining predictions made for linguistic input, while attention coefficients for segments of audio signal correspond to explanation. Experiments conducted on a dataset of clinical interviews from the DAIC-WOZ corpus show that our method significantly outperforms audio-only, text-only, and fusion baselines, reaching an accuracy, Macro-F1, Weighted-F1, AUROC, and ECE of 0.82, 0.80, 0.81, 0.87, and 0.034 respectively. Our system also shows increased robustness against noisy audio conditions, ASR-based transcripts, and missing data.
Suggested Citation
Handle:
RePEc:gvu:tcsst0:v:8:y:2026:i:3:id:2287
DOI: 10.36548/jtcsst.2026.3.004
Download full text from publisher
Corrections
All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gvu:tcsst0:v:8:y:2026:i:3:id:2287. See general information about how to correct material in RePEc.
If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.
We have no bibliographic references for this item. You can help adding them by using this form .
If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Journal Manager (email available below). General contact details of provider: https://irojournals.com/tcsst .
Please note that corrections may take a couple of weeks to filter through
the various RePEc services.