IDEAS home Printed from https://ideas.repec.org/a/gam/jdataj/v6y2021i11p119-d680602.html
   My bibliography  Save this article

Deep Reinforcement Learning for Trading—A Critical Survey

Author

Listed:
  • Adrian Millea

    (Imperial College London, London SW7 2AZ, UK)

Abstract

Deep reinforcement learning (DRL) has achieved significant results in many machine learning (ML) benchmarks. In this short survey, we provide an overview of DRL applied to trading on financial markets with the purpose of unravelling common structures used in the trading community using DRL, as well as discovering common issues and limitations of such approaches. We include also a short corpus summarization using Google Scholar. Moreover, we discuss how one can use hierarchy for dividing the problem space, as well as using model-based RL to learn a world model of the trading environment which can be used for prediction. In addition, multiple risk measures are defined and discussed, which not only provide a way of quantifying the performance of various algorithms, but they can also act as (dense) reward-shaping mechanisms for the agent. We discuss in detail the various state representations used for financial markets, which we consider critical for the success and efficiency of such DRL agents. The market in focus for this survey is the cryptocurrency market; the results of this survey are two-fold: firstly, to find the most promising directions for further research and secondly, to show how a lack of consistency in the community can significantly impede research and the development of DRL agents for trading.

Suggested Citation

  • Adrian Millea, 2021. "Deep Reinforcement Learning for Trading—A Critical Survey," Data, MDPI, vol. 6(11), pages 1-25, November.
    • Handle: RePEc:gam:jdataj:v:6:y:2021:i:11:p:119-:d:680602
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2306-5729/6/11/119/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2306-5729/6/11/119/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Ioannis Boukas & Damien Ernst & Thibaut Th'eate & Adrien Bolland & Alexandre Huynen & Martin Buchwald & Christelle Wynants & Bertrand Corn'elusse, 2020. "A Deep Reinforcement Learning Framework for Continuous Intraday Market Bidding," Papers 2004.05940, arXiv.org.
    2. JoonBum Leem & Ha Young Kim, 2020. "Action-specialized expert ensemble trading system with extended discrete action space using deep reinforcement learning," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-39, July.
    3. Yue-Gang Song & Yu-Long Zhou & Ren-Jie Han, 2018. "Neural networks for stock price prediction," Papers 1805.11317, arXiv.org.
    4. Michael L. Littman, 2015. "Reinforcement learning improves behaviour from evaluative feedback," Nature, Nature, vol. 521(7553), pages 445-451, May.
    5. Karush Suri & Xiao Qi Shi & Konstantinos Plataniotis & Yuri Lawryshyn, 2021. "TradeR: Practical Deep Hierarchical Reinforcement Learning for Trade Execution," Papers 2104.00620, arXiv.org.
    6. Tan, Shay-Kee & Chan, Jennifer So-Kuen & Ng, Kok-Haur, 2020. "On the speculative nature of cryptocurrencies: A study on Garman and Klass volatility measure," Finance Research Letters, Elsevier, vol. 32(C).
    7. Bin Li & Steven C. H. Hoi, 2012. "On-Line Portfolio Selection with Moving Average Reversion," Papers 1206.4626, arXiv.org.
    8. Zexin Hu & Yiqi Zhao & Matloob Khushi, 2021. "A Survey of Forex and Stock Price Prediction Using Deep Learning," Papers 2103.09750, arXiv.org.
    9. Terry Lingze Meng & Matloob Khushi, 2019. "Reinforcement Learning in Financial Markets," Data, MDPI, vol. 4(3), pages 1-17, July.
    10. Zhengyao Jiang & Dixing Xu & Jinjun Liang, 2017. "A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem," Papers 1706.10059, arXiv.org, revised Jul 2017.
    11. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    12. Sidra Mehtab & Jaydip Sen, 2020. "Stock Price Prediction Using Convolutional Neural Networks on a Multivariate Timeseries," Papers 2001.09769, arXiv.org.
    13. Angelos Filos, 2019. "Reinforcement Learning for Portfolio Management," Papers 1909.09571, arXiv.org.
    14. Helder Sebastião & Pedro Godinho, 2021. "Forecasting and trading cryptocurrencies with machine learning under changing market conditions," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 7(1), pages 1-30, December.
    15. Kumar Yashaswi, 2021. "Deep Reinforcement Learning for Portfolio Optimization using Latent Feature State Space (LFSS) Module," Papers 2102.06233, arXiv.org.
    16. Souradeep Chakraborty, 2019. "Capturing Financial markets to apply Deep Reinforcement Learning," Papers 1907.04373, arXiv.org, revised Dec 2019.
    17. Seung-Hyun Moon & Yong-Hyuk Kim & Byung-Ro Moon, 2019. "Empirical investigation of state-of-the-art mean reversion strategies for equity markets," Papers 1909.04327, arXiv.org.
    18. Jonathan Sadighian, 2020. "Extending Deep Reinforcement Learning Frameworks in Cryptocurrency Market Making," Papers 2004.06985, arXiv.org.
    19. Nosratabadi, Saeed & Mosavi, Amir & Duan, Puhong & Ghamisi, Pedram & Filip, Ferdinand & Band, Shahab S. & Reuter, Uwe & Gama, Joao & Gandomi, Amir H., 2020. "Data science in economics: comprehensive review of advanced machine learning and deep learning methods," Thesis Commons auyvc, Center for Open Science.
    20. Yoshiharu Sato, 2019. "Model-Free Reinforcement Learning for Financial Portfolios: A Brief Survey," Papers 1904.04973, arXiv.org, revised May 2019.
    21. Amir Mosavi & Pedram Ghamisi & Yaser Faghan & Puhong Duan, 2020. "Comprehensive Review of Deep Reinforcement Learning Methods and Applications in Economics," Papers 2004.01509, arXiv.org.
    22. Antonio Briola & Jeremy Turiel & Riccardo Marcaccioli & Alvaro Cauderan & Tomaso Aste, 2021. "Deep Reinforcement Learning for Active High Frequency Trading," Papers 2101.07107, arXiv.org, revised Aug 2023.
    23. Fischer, Thomas G., 2018. "Reinforcement learning in financial markets - a survey," FAU Discussion Papers in Economics 12/2018, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics.
    24. Mosavi, Amir & Faghan, Yaser & Ghamisi, Pedram & Duan, Puhong & Ardabili, Sina Faizollahzadeh & Hassan, Salwana & Band, Shahab S., 2020. "Comprehensive Review of Deep Reinforcement Learning Methods and Applications in Economics," OSF Preprints jrc58, Center for Open Science.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Boian Lazov, 2023. "A Deep Reinforcement Learning Trader without Offline Training," Papers 2303.00356, arXiv.org.
    2. Charl Maree & Christian W. Omlin, 2022. "Reinforcement learning with intrinsic affinity for personalized prosperity management," Digital Finance, Springer, vol. 4(2), pages 241-262, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ben Hambly & Renyuan Xu & Huining Yang, 2021. "Recent Advances in Reinforcement Learning in Finance," Papers 2112.04553, arXiv.org, revised Feb 2023.
    2. Charl Maree & Christian W. Omlin, 2022. "Balancing Profit, Risk, and Sustainability for Portfolio Management," Papers 2207.02134, arXiv.org.
    3. Mei-Li Shen & Cheng-Feng Lee & Hsiou-Hsiang Liu & Po-Yin Chang & Cheng-Hong Yang, 2021. "An Effective Hybrid Approach for Forecasting Currency Exchange Rates," Sustainability, MDPI, vol. 13(5), pages 1-29, March.
    4. Brini, Alessio & Tedeschi, Gabriele & Tantari, Daniele, 2023. "Reinforcement learning policy recommendation for interbank network stability," Journal of Financial Stability, Elsevier, vol. 67(C).
    5. Petr Suler & Zuzana Rowland & Tomas Krulicky, 2021. "Evaluation of the Accuracy of Machine Learning Predictions of the Czech Republic’s Exports to the China," JRFM, MDPI, vol. 14(2), pages 1-30, February.
    6. Ben Hambly & Renyuan Xu & Huining Yang, 2023. "Recent advances in reinforcement learning in finance," Mathematical Finance, Wiley Blackwell, vol. 33(3), pages 437-503, July.
    7. Bruno Gašperov & Stjepan Begušić & Petra Posedel Šimović & Zvonko Kostanjčar, 2021. "Reinforcement Learning Approaches to Optimal Market Making," Mathematics, MDPI, vol. 9(21), pages 1-22, October.
    8. Longbing Cao, 2021. "AI in Finance: Challenges, Techniques and Opportunities," Papers 2107.09051, arXiv.org.
    9. Shuo Sun & Rundong Wang & Bo An, 2021. "Reinforcement Learning for Quantitative Trading," Papers 2109.13851, arXiv.org.
    10. Dimitrios Vamvakas & Panagiotis Michailidis & Christos Korkas & Elias Kosmatopoulos, 2023. "Review and Evaluation of Reinforcement Learning Frameworks on Smart Grid Applications," Energies, MDPI, vol. 16(14), pages 1-38, July.
    11. Tian Zhu & Wei Zhu, 2022. "Quantitative Trading through Random Perturbation Q-Network with Nonlinear Transaction Costs," Stats, MDPI, vol. 5(2), pages 1-15, June.
    12. Ruan Pretorius & Terence van Zyl, 2022. "Deep Reinforcement Learning and Convex Mean-Variance Optimisation for Portfolio Management," Papers 2203.11318, arXiv.org.
    13. MohammadAmin Fazli & Mahdi Lashkari & Hamed Taherkhani & Jafar Habibi, 2022. "A Novel Experts Advice Aggregation Framework Using Deep Reinforcement Learning for Portfolio Management," Papers 2212.14477, arXiv.org.
    14. Gang Huang & Xiaohua Zhou & Qingyang Song, 2020. "Deep reinforcement learning for portfolio management," Papers 2012.13773, arXiv.org, revised Apr 2022.
    15. Xiao-Yang Liu & Hongyang Yang & Qian Chen & Runjia Zhang & Liuqing Yang & Bowen Xiao & Christina Dan Wang, 2020. "FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance," Papers 2011.09607, arXiv.org, revised Mar 2022.
    16. Reilly Pickard & Yuri Lawryshyn, 2023. "Deep Reinforcement Learning for Dynamic Stock Option Hedging: A Review," Mathematics, MDPI, vol. 11(24), pages 1-19, December.
    17. Jinan Zou & Qingying Zhao & Yang Jiao & Haiyao Cao & Yanxi Liu & Qingsen Yan & Ehsan Abbasnejad & Lingqiao Liu & Javen Qinfeng Shi, 2022. "Stock Market Prediction via Deep Learning Techniques: A Survey," Papers 2212.12717, arXiv.org, revised Feb 2023.
    18. Amir Masoud Rahmani & Efat Yousefpoor & Mohammad Sadegh Yousefpoor & Zahid Mehmood & Amir Haider & Mehdi Hosseinzadeh & Rizwan Ali Naqvi, 2021. "Machine Learning (ML) in Medicine: Review, Applications, and Challenges," Mathematics, MDPI, vol. 9(22), pages 1-52, November.
    19. Yuhong Wang & Lei Chen & Hong Zhou & Xu Zhou & Zongsheng Zheng & Qi Zeng & Li Jiang & Liang Lu, 2021. "Flexible Transmission Network Expansion Planning Based on DQN Algorithm," Energies, MDPI, vol. 14(7), pages 1-21, April.
    20. Amirhosein Mosavi & Yaser Faghan & Pedram Ghamisi & Puhong Duan & Sina Faizollahzadeh Ardabili & Ely Salwana & Shahab S. Band, 2020. "Comprehensive Review of Deep Reinforcement Learning Methods and Applications in Economics," Mathematics, MDPI, vol. 8(10), pages 1-42, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jdataj:v:6:y:2021:i:11:p:119-:d:680602. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.