Attention-Based Multimodal Emotion Recognition for Fine-Grained Visual Ad Engagement Prediction on Instagram

This paper presents a novel Attention-Based Multimodal Framework (ABMF) for emotion recognition and fine-grained engagement prediction in Instagram advertisements. Traditional approaches to advertisement assessment rely primarily on unimodal analysis and fail to capture the nuanced relationship between emotional content and engagement behaviors. The proposed framework integrates visual, textual, and metadata features through cross-modal attention mechanisms that dynamically identify emotionally salient components across modalities. We construct and annotate the Instagram Advertisement Emotion Dataset (IAED) containing 10,000 sponsored posts with valence-arousal ratings and engagement metrics. Experimental results demonstrate that ABMF achieves significant improvements over state-of-the-art baselines, with 12.1% reduction in valence MAE and 7.1% improvement in engagement prediction MAP. The research reveals distinct relationships between emotional dimensions and specific engagement behaviors: high arousal content generates 78.6% higher share rates while positive valence drives 62.7% more likes compared to negative content. The findings provide quantifiable insights for optimizing emotional content in advertisements based on campaign objectives. The cross-modal attention mechanism enables precise identification of engagement-driving features, offering Instagram advertisers a computational approach to predict and enhance user engagement through targeted emotional content design.