CLAAF: Multimodal fake information detection based on contrastive learning and adaptive Agg-modality fusion

The widespread disinformation on social media platforms has created significant challenges in verifying the authenticity of content, especially in multimodal contexts. However, simple modality fusion can introduce much noise due to the differences in feature representations among various modalities, ultimately impacting the accuracy of detection results. Thus, this paper proposes the Contrastive Learning and Adaptive Agg-modality Fusion (CLAAF) model for multimodal fake information detection. Firstly, a contrastive learning strategy is designed to align text and image modalities, preserving essential features while minimizing redundant noise. Secondly, an adaptive agg-modality fusion module is proposed to facilitate deep interaction and integration between modalities, enhancing the model’s capability to process complex multimodal information. Finally, a comprehensive multimodal dataset is constructed through web crawling from authoritative news sources and multiple fact-checking platforms, establishing a solid foundation for training and validating the model. The experimental results demonstrate that the CLAAF model achieves a 3.45% improvement in accuracy compared to the best-performing baseline models, observably advancing the precision and robustness of multimodal fake information detection.