Multimodal Deception Detection Using Linguistic and Acoustic Features

Recently, there has been a growing interest among researchers in the automatic detection of deceptive behavior, actions, and contents. This surge in attention is driven by the wide-ranging applications of deception detection, particularly in criminology and cybersecurity. To advance this line of research, this study investigates both text and audio data derived from speeches in natural languages. We evaluate traditional linguistic models alongside deep models and advanced Large Language Models (LLMs), utilizing Natural Language Processing (NLP) techniques to model deception detection. Furthermore, we employ various feature selection methods to determine the significance of linguistic features. Through extensive experimentation, we assess the effectiveness of both conventional and advanced deep models on transcribed data while also applying deep models to audio data, thus leveraging both types of data to build a multimodal model for deception and lie detection. Our findings indicate that the Bidirectional Long Short-Term Memory (BiLSTM) model excels in processing textual data. On the other hand, the ResNet50 model performs best with audio data. By combining these models in a late fusion approach, we achieve a model that outperforms individual text and audio models.