IDEAS home Printed from https://ideas.repec.org/p/arx/papers/1904.04973.html
   My bibliography  Save this paper

Model-Free Reinforcement Learning for Financial Portfolios: A Brief Survey

Author

Listed:
  • Yoshiharu Sato

Abstract

Financial portfolio management is one of the problems that are most frequently encountered in the investment industry. Nevertheless, it is not widely recognized that both Kelly Criterion and Risk Parity collapse into Mean Variance under some conditions, which implies that a universal solution to the portfolio optimization problem could potentially exist. In fact, the process of sequential computation of optimal component weights that maximize the portfolio's expected return subject to a certain risk budget can be reformulated as a discrete-time Markov Decision Process (MDP) and hence as a stochastic optimal control, where the system being controlled is a portfolio consisting of multiple investment components, and the control is its component weights. Consequently, the problem could be solved using model-free Reinforcement Learning (RL) without knowing specific component dynamics. By examining existing methods of both value-based and policy-based model-free RL for the portfolio optimization problem, we identify some of the key unresolved questions and difficulties facing today's portfolio managers of applying model-free RL to their investment portfolios.

Suggested Citation

  • Yoshiharu Sato, 2019. "Model-Free Reinforcement Learning for Financial Portfolios: A Brief Survey," Papers 1904.04973, arXiv.org, revised May 2019.
    • Handle: RePEc:arx:papers:1904.04973
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/1904.04973
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Harry M. Markowitz, 2011. "Investment for the Long Run: New Evidence for an Old Rule," World Scientific Book Chapters, in: Leonard C MacLean & Edward O Thorp & William T Ziemba (ed.), THE KELLY CAPITAL GROWTH INVESTMENT CRITERION THEORY and PRACTICE, chapter 35, pages 495-508, World Scientific Publishing Co. Pte. Ltd..
    2. Yifeng Guo & Xingyu Fu & Yuyan Shi & Mingwen Liu, 2018. "Robust Log-Optimal Strategy with Reinforcement Learning," Papers 1805.00205, arXiv.org.
    3. Igor Halperin, 2017. "QLBS: Q-Learner in the Black-Scholes(-Merton) Worlds," Papers 1712.04609, arXiv.org, revised Sep 2019.
    4. Paolo Laureti & Matus Medo & Yi-Cheng Zhang, 2010. "Analysis of Kelly-optimal portfolios," Quantitative Finance, Taylor & Francis Journals, vol. 10(7), pages 689-697.
    5. Black, Fischer & Scholes, Myron S, 1973. "The Pricing of Options and Corporate Liabilities," Journal of Political Economy, University of Chicago Press, vol. 81(3), pages 637-654, May-June.
    6. MOSSIN, Jan, 1968. "Optimal multiperiod portfolio policies," LIDAM Reprints CORE 19, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    7. Merton, Robert C., 1980. "On estimating the expected return on the market : An exploratory investigation," Journal of Financial Economics, Elsevier, vol. 8(4), pages 323-361, December.
    8. Zhipeng Liang & Hao Chen & Junhao Zhu & Kangkang Jiang & Yanran Li, 2018. "Adversarial Deep Reinforcement Learning in Portfolio Management," Papers 1808.09940, arXiv.org, revised Nov 2018.
    9. Harry Markowitz, 1956. "The optimization of a quadratic function subject to linear constraints," Naval Research Logistics Quarterly, John Wiley & Sons, vol. 3(1‐2), pages 111-133, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ben Hambly & Renyuan Xu & Huining Yang, 2021. "Recent Advances in Reinforcement Learning in Finance," Papers 2112.04553, arXiv.org, revised Feb 2023.
    2. Adrian Millea, 2021. "Deep Reinforcement Learning for Trading—A Critical Survey," Data, MDPI, vol. 6(11), pages 1-25, November.
    3. Eric Andr'e & Guillaume Coqueret, 2020. "Dirichlet policies for reinforced factor portfolios," Papers 2011.05381, arXiv.org, revised Jun 2021.
    4. Longbing Cao, 2021. "AI in Finance: Challenges, Techniques and Opportunities," Papers 2107.09051, arXiv.org.
    5. Reilly Pickard & Yuri Lawryshyn, 2023. "Deep Reinforcement Learning for Dynamic Stock Option Hedging: A Review," Mathematics, MDPI, vol. 11(24), pages 1-19, December.
    6. Ali Al-Ameer & Khaled Alshehri, 2021. "Conditional Value-at-Risk for Quantitative Trading: A Direct Reinforcement Learning Approach," Papers 2109.14438, arXiv.org.
    7. Hanwen Zhang & Duy-Minh Dang, 2023. "A monotone numerical integration method for mean-variance portfolio optimization under jump-diffusion models," Papers 2309.05977, arXiv.org.
    8. Xiangyu Cui & Xun Li & Yun Shi & Si Zhao, 2023. "Discrete-Time Mean-Variance Strategy Based on Reinforcement Learning," Papers 2312.15385, arXiv.org.
    9. van Staden, Pieter M. & Dang, Duy-Minh & Forsyth, Peter A., 2021. "The surprising robustness of dynamic Mean-Variance portfolio optimization to model misspecification errors," European Journal of Operational Research, Elsevier, vol. 289(2), pages 774-792.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ben Hambly & Renyuan Xu & Huining Yang, 2021. "Recent Advances in Reinforcement Learning in Finance," Papers 2112.04553, arXiv.org, revised Feb 2023.
    2. Sonntag, Dominik, 2018. "Die Theorie der fairen geometrischen Rendite [The Theory of Fair Geometric Returns]," MPRA Paper 87082, University Library of Munich, Germany.
    3. Ben Hambly & Renyuan Xu & Huining Yang, 2023. "Recent advances in reinforcement learning in finance," Mathematical Finance, Wiley Blackwell, vol. 33(3), pages 437-503, July.
    4. Peter Nystrup & Stephen Boyd & Erik Lindström & Henrik Madsen, 2019. "Multi-period portfolio selection with drawdown control," Annals of Operations Research, Springer, vol. 282(1), pages 245-271, November.
    5. Robert C. Merton, 2006. "Paul Samuelson and Financial Economics," The American Economist, Sage Publications, vol. 50(2), pages 9-31, October.
    6. Diego Amaya & Jean-François Bégin & Geneviève Gauthier, 2022. "The Informational Content of High-Frequency Option Prices," Management Science, INFORMS, vol. 68(3), pages 2166-2201, March.
    7. Carmine De Franco & Johann Nicolle & Huyên Pham, 2019. "Dealing with Drift Uncertainty: A Bayesian Learning Approach," Risks, MDPI, vol. 7(1), pages 1-18, January.
    8. Matus Medo & Chi Ho Yeung & Yi-Cheng Zhang, 2008. "How to quantify the influence of correlations on investment diversification," Papers 0805.3397, arXiv.org, revised Feb 2009.
    9. Yu, Jun, 2014. "Econometric Analysis Of Continuous Time Models: A Survey Of Peter Phillips’S Work And Some New Results," Econometric Theory, Cambridge University Press, vol. 30(4), pages 737-774, August.
    10. Ian Martin, 2017. "What is the Expected Return on the Market?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 132(1), pages 367-433.
    11. Amirhosein Mosavi & Yaser Faghan & Pedram Ghamisi & Puhong Duan & Sina Faizollahzadeh Ardabili & Ely Salwana & Shahab S. Band, 2020. "Comprehensive Review of Deep Reinforcement Learning Methods and Applications in Economics," Mathematics, MDPI, vol. 8(10), pages 1-42, September.
    12. Sassan Alizadeh & Michael W. Brandt & Francis X. Diebold, 2002. "Range‐Based Estimation of Stochastic Volatility Models," Journal of Finance, American Finance Association, vol. 57(3), pages 1047-1091, June.
    13. Dicle, Mehmet F. & Levendis, John, 2020. "Historic risk and implied volatility," Global Finance Journal, Elsevier, vol. 45(C).
    14. Tim Bollerslev & Ray Y. Chou & Narayanan Jayaraman & Kenneth F. Kroner - L, 1991. "es modéles ARCH en finance : un point sur la théorie et les résultats empiriques," Annals of Economics and Statistics, GENES, issue 24, pages 1-59.
    15. Elyas Elyasiani & Luca Gambarelli & Silvia Muzzioli, 2015. "Towards a skewness index for the Italian stock market," Department of Economics 0064, University of Modena and Reggio E., Faculty of Economics "Marco Biagi".
    16. Renata Rendek, 2013. "Modeling Diversified Equity Indices," PhD Thesis, Finance Discipline Group, UTS Business School, University of Technology, Sydney, number 23, July-Dece.
    17. Blomvall, Jorgen & Lindberg, Per Olov, 2003. "Back-testing the performance of an actively managed option portfolio at the Swedish Stock Market, 1990-1999," Journal of Economic Dynamics and Control, Elsevier, vol. 27(6), pages 1099-1112, April.
    18. Gonçalo Faria & João Correia-da-Silva, 2014. "A closed-form solution for options with ambiguity about stochastic volatility," Review of Derivatives Research, Springer, vol. 17(2), pages 125-159, July.
    19. repec:adr:anecst:y:1991:i:24:p:01 is not listed on IDEAS
    20. Merton, Robert, 1990. "Capital market theory and the pricing of financial securities," Handbook of Monetary Economics, in: B. M. Friedman & F. H. Hahn (ed.), Handbook of Monetary Economics, edition 1, volume 1, chapter 11, pages 497-581, Elsevier.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:1904.04973. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.