A multifactor model using large language models and multimodal investor sentiment

This study constructs multimodal investor sentiment indices using news and image data from the China News Service, covering the period from January 1, 2017, to December 31, 2024. We employ the RoBERTa model for text-based sentiment measurement and the Google Inception(v3) model for image-based sentiment measurement. We use a multimodal semantic correlation fusion model to integrate textual and visual sentiment features. These sentiment indices are categorised as industry-specific and market-wide investor sentiment, enabling separate analyses of their effects on stock markets. Furthermore, we develop a multifactor stock selection model that incorporates these sentiment indices with other microeconomic factors. Our findings demonstrate that multimodal sentiment analysis yields superior predictive accuracy. Industry-specific investor sentiment influences stock market returns, which in turn exacerbates changes in market-wide investor sentiment. Incorporating industry-specific sentiment into the multifactor stock selection model enhances portfolio returns, and combining market-wide sentiment with timing strategies further improves performance.