AI-driven forecasting and portfolio optimization under ESG and uncertainty regimes: evidence from global equity indices

PurposeThis study investigates how advanced machine learning (ML) and deep reinforcement learning (DRL) models can enhance forecasting accuracy and portfolio decision-making under environmental, social, and governance (ESG)-driven and uncertainty-sensitive market conditions. The objective is to compare the predictive efficiency of state-of-the-art ML architectures with policy-based portfolio strategies to uncover the role of sustainability indicators in shaping market dynamics.Design/methodology/approachWe analyze daily data for major ESG and conventional market indices across the USA, Europe, and China, integrating the Standard Statistics Company and Poor's Publishing Company (S&P) Green Bond Index, Geopolitical Risk (GPR), the US Financial Stress Index (FSI), Physical Risk Index (PRI), and Transition Risk Index (TRI). The forecasting stage combines a 20-day long short-term memory (LSTM) temporal representation with Categorical Boosting (CatBoost), Light Gradient Boosting Machine (LightGBM), and Extreme Gradient Boosting (XGBoost) under a strict 70/15/15 chronological train-validation-test design and train-only scaling. The portfolio stage compares Twin Delayed Deep Deterministic Policy Gradient (TD3), Soft Actor-Critic (SAC), and Deep Deterministic Policy Gradient (DDPG) in a weekly self-financing environment with long-only weights, a 40% position cap, and proportional transaction costs. Random-Walk/Ridge/LSTM forecasting benchmarks and Equal-Weight/Minimum-Variance portfolio benchmarks are included, together with multi-seed and transaction-cost robustness checks. Local interpretable model-agnostic explanations (LIME) is retained for local model interpretation.FindingsForecasting performance is market-dependent rather than governed by a universally dominant learner. Validation selects LSTM-LightGBM for the China and US ESG indices, LSTM-XGBoost for Europe ESG, and LSTM-CatBoost for the three conventional indices. Multi-seed robustness yields positive average root mean square error (RMSE) gains over the Random Walk in five of six markets, with the strongest stability in China and statistically significant main-seed improvement for US ESG. In portfolio allocation, TD3 on the conventional opportunity set provides the strongest ex-post DRL performance (11.10% annualized return; Sharpe ratio 0.776), while SAC is the strongest ESG DRL strategy. Equal Weight remains a demanding benchmark, particularly for ESG portfolios, and the DDPG policies selected ex ante on validation remain profitable across three seeds and under transaction-cost scenarios up to 25 basis points (bps).Originality/valueThe study extends the emerging literature by integrating explainable hybrid forecasting and continuous-action DRL within a leakage-controlled, benchmark-oriented framework. Its contribution lies in combining ESG-sensitive uncertainty variables, chronological validation, realistic self-financing portfolio constraints, explicit transaction-cost analysis, and reproducibility checks rather than assuming universal AI or ESG superiority.