Two-Stage Dynamic Fusion Framework for Multimodal Classification Tasks

Multimodal learning has provided an opportunity to better analyze a system or phenomenon. Numerous classification studies have developed advanced dynamic fusion methods to fuse information from different modalities. However, few works have considered a reliable design of dynamic fusion methods based on theoretical insights. In this context, we address the research gaps as follows. From a theoretical perspective, we initially establish the performance range for the accuracy of a multimodal classifier. Subsequently, we derive a condition based on the upper limit of the range to indicate how to improve the accuracy of the model. From a technical perspective, we propose a two-stage dynamic fusion framework according to this condition. In the first stage, we design an uncertainty-aware dynamic fusion method. In the second stage, we propose a regression-based method to adaptively generate the learned fusion weight for each modality. In the experiment, we use seven existing models for comparisons and exploit four public data sets to examine the effectiveness of the two-stage framework. The results indicate that our proposed framework generally outperforms existing methods in terms of accuracy and robustness. Additionally, we conduct a comprehensive discussion from several aspects to further illustrate the merits of the proposed framework.