IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2307.07694.html
   My bibliography  Save this paper

Evaluation of Deep Reinforcement Learning Algorithms for Portfolio Optimisation

Author

Listed:
  • Chung I Lu

Abstract

We evaluate benchmark deep reinforcement learning (DRL) algorithms on the task of portfolio optimisation under a simulator. The simulator is based on correlated geometric Brownian motion (GBM) with the Bertsimas-Lo (BL) market impact model. Using the Kelly criterion (log utility) as the objective, we can analytically derive the optimal policy without market impact and use it as an upper bound to measure performance when including market impact. We found that the off-policy algorithms DDPG, TD3 and SAC were unable to learn the right Q function due to the noisy rewards and therefore perform poorly. The on-policy algorithms PPO and A2C, with the use of generalised advantage estimation (GAE), were able to deal with the noise and derive a close to optimal policy. The clipping variant of PPO was found to be important in preventing the policy from deviating from the optimal once converged. In a more challenging environment where we have regime changes in the GBM parameters, we found that PPO, combined with a hidden Markov model (HMM) to learn and predict the regime context, is able to learn different policies adapted to each regime. Overall, we find that the sample complexity of these algorithms is too high, requiring more than 2m steps to learn a good policy in the simplest setting, which is equivalent to almost 8,000 years of daily prices.

Suggested Citation

  • Chung I Lu, 2023. "Evaluation of Deep Reinforcement Learning Algorithms for Portfolio Optimisation," Papers 2307.07694, arXiv.org, revised Jul 2023.
  • Handle: RePEc:arx:papers:2307.07694
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2307.07694
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Andrew Ang & Geert Bekaert, 2002. "International Asset Allocation With Regime Shifts," The Review of Financial Studies, Society for Financial Studies, vol. 15(4), pages 1137-1187.
    2. Adriano Koshiyama & Nick Firoozye & Philip Treleaven, 2021. "Generative adversarial networks for financial trading strategies fine-tuning and combination," Quantitative Finance, Taylor & Francis Journals, vol. 21(5), pages 797-813, May.
    3. Huntley Schaller & Simon Van Norden, 1997. "Regime switching in stock market returns," Applied Financial Economics, Taylor & Francis Journals, vol. 7(2), pages 177-191.
    4. Matthew Dixon & Igor Halperin, 2020. "G-Learner and GIRL: Goal Based Wealth Management with Reinforcement Learning," Papers 2002.10990, arXiv.org.
    5. Hans Buhler & Blanka Horvath & Terry Lyons & Imanol Perez Arribas & Ben Wood, 2020. "A Data-driven Market Simulator for Small Data Environments," Papers 2006.14498, arXiv.org.
    6. Magnus Wiese & Robert Knobloch & Ralf Korn & Peter Kretschmer, 2020. "Quant GANs: deep generation of financial time series," Quantitative Finance, Taylor & Francis Journals, vol. 20(9), pages 1419-1440, September.
    7. Mark Davis & Sébastien Lleo, 2011. "Fractional Kelly Strategies for Benchmarked Asset Management," World Scientific Book Chapters, in: Leonard C MacLean & Edward O Thorp & William T Ziemba (ed.), THE KELLY CAPITAL GROWTH INVESTMENT CRITERION THEORY and PRACTICE, chapter 27, pages 385-407, World Scientific Publishing Co. Pte. Ltd..
    8. Yunan Ye & Hengzhi Pei & Boxin Wang & Pin-Yu Chen & Yada Zhu & Jun Xiao & Bo Li, 2020. "Reinforcement-Learning based Portfolio Management with Augmented Asset Movement Prediction States," Papers 2002.05780, arXiv.org.
    9. Bertsimas, Dimitris & Lo, Andrew W., 1998. "Optimal control of execution costs," Journal of Financial Markets, Elsevier, vol. 1(1), pages 1-50, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Daniil Karzanov & Rub'en Garz'on & Mikhail Terekhov & Caglar Gulcehre & Thomas Raffinot & Marcin Detyniecki, 2025. "Regret-Optimized Portfolio Enhancement through Deep Reinforcement Learning and Future Looking Rewards," Papers 2502.02619, arXiv.org.
    2. Kansuda Pankwaen & Sukrit Thongkairat & Worrawat Saijai, 2025. "Global Cross-Market Trading Optimization Using Iterative Combined Algorithm: A Multi-Asset Approach with Stocks and Cryptocurrencies," Mathematics, MDPI, vol. 13(8), pages 1-27, April.
    3. Chung I Lu & Julian Sester, 2024. "Generative model for financial time series trained with MMD using a signature kernel," Papers 2407.19848, arXiv.org, revised Dec 2024.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alexandre Miot, 2020. "Adversarial trading," Papers 2101.03128, arXiv.org.
    2. Francesca Biagini & Lukas Gonon & Niklas Walter, 2024. "Universal randomised signatures for generative time series modelling," Papers 2406.10214, arXiv.org, revised Sep 2024.
    3. Chung I Lu & Julian Sester, 2024. "Generative model for financial time series trained with MMD using a signature kernel," Papers 2407.19848, arXiv.org, revised Dec 2024.
    4. Emiel Lemahieu & Kris Boudt & Maarten Wyns, 2023. "Generating drawdown-realistic financial price paths using path signatures," Papers 2309.04507, arXiv.org.
    5. Chatziantoniou, Ioannis & Filis, George & Floros, Christos, 2017. "Asset prices regime-switching and the role of inflation targeting monetary policy," Global Finance Journal, Elsevier, vol. 32(C), pages 97-112.
    6. Mohamed Hamdouche & Pierre Henry-Labordere & Huy^en Pham, 2023. "Generative modeling for time series via Schr{\"o}dinger bridge," Papers 2304.05093, arXiv.org.
    7. Shuo Sun & Rundong Wang & Bo An, 2021. "Reinforcement Learning for Quantitative Trading," Papers 2109.13851, arXiv.org.
    8. Guidolin, Massimo & Hyde, Stuart & McMillan, David & Ono, Sadayuki, 2009. "Non-linear predictability in stock and bond returns: When and where is it exploitable?," International Journal of Forecasting, Elsevier, vol. 25(2), pages 373-399.
    9. John M. Maheu & Thomas H. McCurdy & Yong Song, 2012. "Components of Bull and Bear Markets: Bull Corrections and Bear Rallies," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(3), pages 391-403, February.
    10. Sohyeon Kwon & Yongjae Lee, 2024. "Can GANs Learn the Stylized Facts of Financial Time Series?," Papers 2410.09850, arXiv.org.
    11. Chang, Kuang-Liang, 2009. "Do macroeconomic variables have regime-dependent effects on stock return dynamics? Evidence from the Markov regime switching model," Economic Modelling, Elsevier, vol. 26(6), pages 1283-1299, November.
    12. Martin Hess, 2006. "Timing and diversification: A state-dependent asset allocation approach," The European Journal of Finance, Taylor & Francis Journals, vol. 12(3), pages 189-204.
    13. Martín Solá & Zacharias Psaradakis & Fabio Spagnolo & Nicola Spagnolo, 2010. "Some Cautionary Results Concerning Markov-Switching Models with Time-Varying Transition Probabilities," Department of Economics Working Papers 2010-12, Universidad Torcuato Di Tella.
    14. Wasim Ahmad & N. Bhanumurthy & Sanjay Sehgal, 2015. "Regime dependent dynamics and European stock markets: Is asset allocation really possible?," Empirica, Springer;Austrian Institute for Economic Research;Austrian Economic Association, vol. 42(1), pages 77-107, February.
    15. Song Wei & Andrea Coletta & Svitlana Vyetrenko & Tucker Balch, 2023. "INTAGS: Interactive Agent-Guided Simulation," Papers 2309.01784, arXiv.org, revised Nov 2023.
    16. Rama Cont & Mihai Cucuringu & Renyuan Xu & Chao Zhang, 2022. "Tail-GAN: Learning to Simulate Tail Risk Scenarios," Papers 2203.01664, arXiv.org, revised May 2025.
    17. Magnus Wiese & Phillip Murray & Ralf Korn, 2023. "Sig-Splines: universal approximation and convex calibration of time series generative models," Papers 2307.09767, arXiv.org.
    18. Massimo Guidolin & Allan Timmermann, 2006. "An econometric model of nonlinear dynamics in the joint distribution of stock and bond returns," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 21(1), pages 1-22, January.
    19. Samuel N. Cohen & Christoph Reisinger & Sheng Wang, 2022. "Estimating risks of option books using neural-SDE market models," Papers 2202.07148, arXiv.org.
    20. Ben Hambly & Renyuan Xu & Huining Yang, 2023. "Recent advances in reinforcement learning in finance," Mathematical Finance, Wiley Blackwell, vol. 33(3), pages 437-503, July.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2307.07694. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.