IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2505.07537.html
   My bibliography  Save this paper

The Exploratory Multi-Asset Mean-Variance Portfolio Selection using Reinforcement Learning

Author

Listed:
  • Yu Li
  • Yuhan Wu
  • Shuhua Zhang

Abstract

In this paper, we study the continuous-time multi-asset mean-variance (MV) portfolio selection using a reinforcement learning (RL) algorithm, specifically the soft actor-critic (SAC) algorithm, in the time-varying financial market. A family of Gaussian portfolio selections is derived, and a policy iteration process is crafted to learn the optimal exploratory portfolio selection. We prove the convergence of the policy iteration process theoretically, based on which the SAC algorithm is developed. To improve the algorithm's stability and the learning accuracy in the multi-asset scenario, we divide the model parameters that influence the optimal portfolio selection into three parts, and learn each part progressively. Numerical studies in the simulated and real financial markets confirm the superior performance of the proposed SAC algorithm under various criteria.

Suggested Citation

  • Yu Li & Yuhan Wu & Shuhua Zhang, 2025. "The Exploratory Multi-Asset Mean-Variance Portfolio Selection using Reinforcement Learning," Papers 2505.07537, arXiv.org.
    • Handle: RePEc:arx:papers:2505.07537
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2505.07537
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Ledoit, Olivier & Wolf, Michael, 2004. "A well-conditioned estimator for large-dimensional covariance matrices," Journal of Multivariate Analysis, Elsevier, vol. 88(2), pages 365-411, February.
    2. Jorion, Philippe, 1986. "Bayes-Stein Estimation for Portfolio Analysis," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 21(3), pages 279-292, September.
    3. Pedro Barroso & Konark Saxena, 2022. "Lest We Forget: Learn from Out-of-Sample Forecast Errors When Optimizing Portfolios," The Review of Financial Studies, Society for Financial Studies, vol. 35(3), pages 1222-1278.
    4. Victor Duarte & Diogo Duarte & Dejanir H Silva, 2024. "Machine Learning for Continuous-Time Finance," The Review of Financial Studies, Society for Financial Studies, vol. 37(11), pages 3217-3271.
    5. De Gennaro Aquino, Luca & Sornette, Didier & Strub, Moris S., 2023. "Portfolio selection with exploration of new investment assets," European Journal of Operational Research, Elsevier, vol. 310(2), pages 773-792.
    6. Hutchinson, James M & Lo, Andrew W & Poggio, Tomaso, 1994. "A Nonparametric Approach to Pricing and Hedging Derivative Securities via Learning Networks," Journal of Finance, American Finance Association, vol. 49(3), pages 851-889, July.
    7. Olivier Ledoit & Michael Wolf, 2017. "Nonlinear Shrinkage of the Covariance Matrix for Portfolio Selection: Markowitz Meets Goldilocks," The Review of Financial Studies, Society for Financial Studies, vol. 30(12), pages 4349-4388.
    8. Ledoit, Olivier & Wolf, Michael, 2003. "Improved estimation of the covariance matrix of stock returns with an application to portfolio selection," Journal of Empirical Finance, Elsevier, vol. 10(5), pages 603-621, December.
    9. Duan Li & Wan‐Lung Ng, 2000. "Optimal Dynamic Portfolio Selection: Multiperiod Mean‐Variance Formulation," Mathematical Finance, Wiley Blackwell, vol. 10(3), pages 387-406, July.
    10. Shi, Fangquan & Shu, Lianjie & Yang, Aijun & He, Fangyi, 2020. "Improving Minimum-Variance Portfolios by Alleviating Overdispersion of Eigenvalues," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 55(8), pages 2700-2731, December.
    11. Victor Duarte & Diogo Duarte & Dejanir H. Silva, 2024. "Machine Learning for Continuous-Time Finance," CESifo Working Paper Series 10909, CESifo.
    12. Wang, J. & Forsyth, P.A., 2010. "Numerical solution of the Hamilton-Jacobi-Bellman formulation for continuous time mean variance asset allocation," Journal of Economic Dynamics and Control, Elsevier, vol. 34(2), pages 207-230, February.
    13. Haoran Wang & Xun Yu Zhou, 2020. "Continuous‐time mean–variance portfolio selection: A reinforcement learning framework," Mathematical Finance, Wiley Blackwell, vol. 30(4), pages 1273-1308, October.
    14. Hiraki, Kazuhiro & Sun, Chuanping, 2022. "A toolkit for exploiting contemporaneous stock correlations," Journal of Empirical Finance, Elsevier, vol. 65(C), pages 99-124.
    15. Joong-Ho Won & Johan Lim & Seung-Jean Kim & Bala Rajaratnam, 2013. "Condition-number-regularized covariance estimation," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 75(3), pages 427-450, June.
    16. Kydland, Finn E & Prescott, Edward C, 1982. "Time to Build and Aggregate Fluctuations," Econometrica, Econometric Society, vol. 50(6), pages 1345-1370, November.
    17. R. Jiang & D. Saunders & C. Weng, 2022. "The reinforcement learning Kelly strategy," Quantitative Finance, Taylor & Francis Journals, vol. 22(8), pages 1445-1464, August.
    18. Merton, Robert C, 1969. "Lifetime Portfolio Selection under Uncertainty: The Continuous-Time Case," The Review of Economics and Statistics, MIT Press, vol. 51(3), pages 247-257, August.
    19. Xin Guo & Renyuan Xu & Thaleia Zariphopoulou, 2022. "Entropy Regularization for Mean Field Games with Learning," Mathematics of Operations Research, INFORMS, vol. 47(4), pages 3239-3260, November.
    20. Candelon, B. & Hurlin, C. & Tokpavi, S., 2012. "Sampling error and double shrinkage estimation of minimum variance portfolios," Journal of Empirical Finance, Elsevier, vol. 19(4), pages 511-527.
    21. Balduzzi, Pierluigi & Lynch, Anthony W., 1999. "Transaction costs and predictability: some utility cost calculations," Journal of Financial Economics, Elsevier, vol. 52(1), pages 47-78, April.
    22. Merton, Robert C., 1972. "An Analytic Derivation of the Efficient Portfolio Frontier," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 7(4), pages 1851-1872, September.
    23. Min Dai & Yuchao Dong & Yanwei Jia, 2023. "Learning equilibrium mean‐variance strategy," Mathematical Finance, Wiley Blackwell, vol. 33(4), pages 1166-1212, October.
    24. Elena Vigna, 2020. "On Time Consistency For Mean-Variance Portfolio Selection," International Journal of Theoretical and Applied Finance (IJTAF), World Scientific Publishing Co. Pte. Ltd., vol. 23(06), pages 1-22, September.
    25. Christoph Czichowsky, 2013. "Time-consistent mean-variance portfolio selection in discrete and continuous time," Finance and Stochastics, Springer, vol. 17(2), pages 227-271, April.
    26. Tomas Björk & Agatha Murgoci & Xun Yu Zhou, 2014. "Mean–Variance Portfolio Optimization With State-Dependent Risk Aversion," Mathematical Finance, Wiley Blackwell, vol. 24(1), pages 1-24, January.
    27. Kirby, Chris & Ostdiek, Barbara, 2012. "It’s All in the Timing: Simple Active Portfolio Strategies that Outperform Naïve Diversification," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 47(2), pages 437-467, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yuanyuan Zhang & Xiang Li & Sini Guo, 2018. "Portfolio selection problems with Markowitz’s mean–variance framework: a review of literature," Fuzzy Optimization and Decision Making, Springer, vol. 17(2), pages 125-158, June.
    2. Tu, Xueyong & Li, Bin, 2024. "Robust portfolio selection with smart return prediction," Economic Modelling, Elsevier, vol. 135(C).
    3. Zhou, Zhongbao & Xiao, Helu & Yin, Jialing & Zeng, Ximei & Lin, Ling, 2016. "Pre-commitment vs. time-consistent strategies for the generalized multi-period portfolio optimization with stochastic cash flows," Insurance: Mathematics and Economics, Elsevier, vol. 68(C), pages 187-202.
    4. Ruili Sun & Tiefeng Ma & Shuangzhe Liu & Milind Sathye, 2019. "Improved Covariance Matrix Estimation for Portfolio Risk Measurement: A Review," JRFM, MDPI, vol. 12(1), pages 1-34, March.
    5. Lim Hao Shen Keith, 2024. "Covariance Matrix Analysis for Optimal Portfolio Selection," Papers 2407.08748, arXiv.org.
    6. Johannes Bock, 2018. "An updated review of (sub-)optimal diversification models," Papers 1811.08255, arXiv.org.
    7. De Gennaro Aquino, Luca & Sornette, Didier & Strub, Moris S., 2023. "Portfolio selection with exploration of new investment assets," European Journal of Operational Research, Elsevier, vol. 310(2), pages 773-792.
    8. Shi, Fangquan & Shu, Lianjie & He, Fangyi & Huang, Wenpo, 2025. "Improving minimum-variance portfolio through shrinkage of large covariance matrices," Economic Modelling, Elsevier, vol. 144(C).
    9. Luca De Gennaro Aquino & Sascha Desmettre & Yevhen Havrylenko & Mogens Steffensen, 2024. "Equilibrium control theory for Kihlstrom-Mirman preferences in continuous time," Papers 2407.16525, arXiv.org, revised Oct 2024.
    10. Istvan Varga-Haszonits & Fabio Caccioli & Imre Kondor, 2016. "Replica approach to mean-variance portfolio optimization," Papers 1606.08679, arXiv.org.
    11. Penaranda, Francisco, 2007. "Portfolio choice beyond the traditional approach," LSE Research Online Documents on Economics 24481, London School of Economics and Political Science, LSE Library.
    12. Chen, Jia & Li, Degui & Linton, Oliver, 2019. "A new semiparametric estimation approach for large dynamic covariance matrices with multiple conditioning variables," Journal of Econometrics, Elsevier, vol. 212(1), pages 155-176.
    13. Min Dai & Yuchao Dong & Yanwei Jia & Xun Yu Zhou, 2023. "Data-Driven Merton's Strategies via Policy Randomization," Papers 2312.11797, arXiv.org, revised May 2025.
    14. Yan, Cheng & Zhang, Huazhu, 2017. "Mean-variance versus naïve diversification: The role of mispricing," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 48(C), pages 61-81.
    15. Alex Shkolnik & Alec Kercheval & Hubeyb Gurdogan & Lisa R. Goldberg & Haim Bar, 2025. "Portfolio selection revisited," Annals of Operations Research, Springer, vol. 346(1), pages 137-155, March.
    16. Nathan Lassance & Frédéric Vrins, 2021. "Minimum Rényi entropy portfolios," Annals of Operations Research, Springer, vol. 299(1), pages 23-46, April.
    17. Long Zhao & Deepayan Chakrabarti & Kumar Muthuraman, 2019. "Portfolio Construction by Mitigating Error Amplification: The Bounded-Noise Portfolio," Operations Research, INFORMS, vol. 67(4), pages 965-983, July.
    18. Qiao, W. & Bu, D. & Gibberd, A. & Liao, Y. & Wen, T. & Li, E., 2023. "When “time varying” volatility meets “transaction cost” in portfolio selection," Journal of Empirical Finance, Elsevier, vol. 73(C), pages 220-237.
    19. Raymond Kan & Xiaolu Wang & Guofu Zhou, 2022. "Optimal Portfolio Choice with Estimation Risk: No Risk-Free Asset Case," Management Science, INFORMS, vol. 68(3), pages 2047-2068, March.
    20. Min Dai & Hanqing Jin & Steven Kou & Yuhong Xu, 2021. "A Dynamic Mean-Variance Analysis for Log Returns," Management Science, INFORMS, vol. 67(2), pages 1093-1108, February.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2505.07537. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.