Machine learning for stock return prediction: Transformers or simple neural networks

We study whether a transformer improves stock return forecasts over simple neural networks. Using all U.S. stocks from 1957 to 2021, we train an Autoformer with 920 features that cover firm characteristics, industry dummies, and macro variables. We compare it with DLinear and simple neural networks. We predict excess returns at 1-, 3-, and 12-month horizons. We evaluate probabilistic forecasts using mean squared error (MSE). Autoformer consistently outperforms the simple neural networks advocated by Gu et al. (2020), markedly at longer horizons. Feature-importance tests repeatedly highlight net equity expansion and a technology-industry dummy. These tests also emphasize fundamentals such as return on assets, sales-to-price, earnings volatility, and sales-to-receivables. Overall, transformer architectures appear better at encoding fundamental information than shallow networks.