Multimodal Recognition of Users States at Human-AI Interaction Adaptation

The study investigates advancements in multimodal emotion recognition (MMER) within the evolving landscape of human-AI interaction. By synthesizing insights from psychology, computer science, and cognitive neuroscience, this paper examines the integration of multiple modalities—such as facial expressions, vocal tone, textual sentiment, and physiological signals—to achieve adaptive and emotionally intelligent AI systems. Leveraging a systematic literature review, it identifies state-of-the-art methodologies, including deep learning-based data fusion techniques, and their applications across sectors such as healthcare, education, and transportation. The review provides practical tools to support the implementation of MMER systems, such as a decision tree for selecting the most appropriate theoretical approach based on specific application needs and another for choosing the optimal fusion method. The proposed simplifications can help system designers and researchers solve key challenges, such asintegrating different data types and real-time processing. The article will also address ethical issues such as privacy and anti-bias