IDEAS home Printed from https://ideas.repec.org/p/arx/papers/1911.10107.html
   My bibliography  Save this paper

Deep Reinforcement Learning for Trading

Author

Listed:
  • Zihao Zhang
  • Stefan Zohren
  • Stephen Roberts

Abstract

We adopt Deep Reinforcement Learning algorithms to design trading strategies for continuous futures contracts. Both discrete and continuous action spaces are considered and volatility scaling is incorporated to create reward functions which scale trade positions based on market volatility. We test our algorithms on the 50 most liquid futures contracts from 2011 to 2019, and investigate how performance varies across different asset classes including commodities, equity indices, fixed income and FX markets. We compare our algorithms against classical time series momentum strategies, and show that our method outperforms such baseline models, delivering positive profits despite heavy transaction costs. The experiments show that the proposed algorithms can follow large market trends without changing positions and can also scale down, or hold, through consolidation periods.

Suggested Citation

  • Zihao Zhang & Stefan Zohren & Stephen Roberts, 2019. "Deep Reinforcement Learning for Trading," Papers 1911.10107, arXiv.org.
    • Handle: RePEc:arx:papers:1911.10107
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/1911.10107
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Zihao Zhang & Stefan Zohren & Stephen Roberts, 2018. "BDLOB: Bayesian Deep Convolutional Neural Networks for Limit Order Books," Papers 1811.10041, arXiv.org.
    2. Xiao-Yang Liu & Zhuoran Xiong & Shan Zhong & Hongyang Yang & Anwar Walid, 2018. "Practical Deep Reinforcement Learning Approach for Stock Trading," Papers 1811.07522, arXiv.org, revised Jul 2022.
    3. Merton, Robert C, 1969. "Lifetime Portfolio Selection under Uncertainty: The Continuous-Time Case," The Review of Economics and Statistics, MIT Press, vol. 51(3), pages 247-257, August.
    4. Wei Bao & Jun Yue & Yulei Rao, 2017. "A deep learning framework for financial time series using stacked autoencoders and long-short term memory," PLOS ONE, Public Library of Science, vol. 12(7), pages 1-24, July.
    5. Justin Sirignano & Rama Cont, 2019. "Universal features of price formation in financial markets: perspectives from deep learning," Quantitative Finance, Taylor & Francis Journals, vol. 19(9), pages 1449-1459, September.
    6. Bekiros, Stelios D., 2010. "Heterogeneous trading strategies with adaptive fuzzy Actor-Critic reinforcement learning: A behavioral approach," Journal of Economic Dynamics and Control, Elsevier, vol. 34(6), pages 1153-1170, June.
    7. Zihao Zhang & Stefan Zohren & Stephen Roberts, 2019. "Extending Deep Learning Models for Limit Order Books to Quantile Regression," Papers 1906.04404, arXiv.org.
    8. Chien Yi Huang, 2018. "Financial Trading as a Game: A Deep Reinforcement Learning Approach," Papers 1807.02787, arXiv.org.
    9. Francesco Bertoluzzo & Marco Corazza, 2012. "Reinforcement Learning for automatic financial trading: Introduction and some applications," Working Papers 2012:33, Department of Economics, University of Venice "Ca' Foscari", revised 2012.
    10. Fischer, Thomas G., 2018. "Reinforcement learning in financial markets - a survey," FAU Discussion Papers in Economics 12/2018, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics.
    11. Bryan Lim & Stefan Zohren & Stephen Roberts, 2019. "Enhancing Time Series Momentum Strategies Using Deep Neural Networks," Papers 1904.04912, arXiv.org, revised Sep 2020.
    12. Zihao Zhang & Stefan Zohren & Stephen Roberts, 2018. "DeepLOB: Deep Convolutional Neural Networks for Limit Order Books," Papers 1808.03668, arXiv.org, revised Jan 2020.
    13. Moskowitz, Tobias J. & Ooi, Yao Hua & Pedersen, Lasse Heje, 2012. "Time series momentum," Journal of Financial Economics, Elsevier, vol. 104(2), pages 228-250.
    14. Fischer, Thomas & Krauss, Christopher, 2018. "Deep learning with long short-term memory networks for financial market predictions," European Journal of Operational Research, Elsevier, vol. 270(2), pages 654-669.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Eric Benhamou & David Saltiel & Sandrine Ungari & Abhishek Mukhopadhyay, 2020. "Bridging the gap between Markowitz planning and deep reinforcement learning," Papers 2010.09108, arXiv.org.
    2. Yuanrong Wang & Yinsen Miao & Alexander CY Wong & Nikita P Granger & Christian Michler, 2023. "Domain-adapted Learning and Interpretability: DRL for Gas Trading," Papers 2301.08359, arXiv.org, revised Sep 2023.
    3. Sihang Chen & Weiqi Luo & Chao Yu, 2021. "Reinforcement Learning with Expert Trajectory For Quantitative Trading," Papers 2105.03844, arXiv.org.
    4. Tessa Bauman & Bruno Gav{s}perov & Stjepan Beguv{s}i'c & Zvonko Kostanjv{c}ar, 2023. "Deep Reinforcement Learning for Robust Goal-Based Wealth Management," Papers 2307.13501, arXiv.org.
    5. Gabriel Borrageiro & Nick Firoozye & Paolo Barucca, 2021. "Reinforcement Learning for Systematic FX Trading," Papers 2110.04745, arXiv.org, revised May 2022.
    6. Ali Raheman & Anton Kolonin & Alexey Glushchenko & Arseniy Fokin & Ikram Ansari, 2022. "Adaptive Multi-Strategy Market-Making Agent For Volatile Markets," Papers 2204.13265, arXiv.org.
    7. Eric Andr'e & Guillaume Coqueret, 2020. "Dirichlet policies for reinforced factor portfolios," Papers 2011.05381, arXiv.org, revised Jun 2021.
    8. Miquel Noguer i Alonso & Sonam Srivastava, 2020. "Deep Reinforcement Learning for Asset Allocation in US Equities," Papers 2010.04404, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zihao Zhang & Stefan Zohren & Stephen Roberts, 2020. "Deep Learning for Portfolio Optimization," Papers 2005.13665, arXiv.org, revised Jan 2021.
    2. Tidor-Vlad Pricope, 2021. "Deep Reinforcement Learning in Quantitative Algorithmic Trading: A Review," Papers 2106.00123, arXiv.org.
    3. Antonio Briola & Jeremy Turiel & Riccardo Marcaccioli & Alvaro Cauderan & Tomaso Aste, 2021. "Deep Reinforcement Learning for Active High Frequency Trading," Papers 2101.07107, arXiv.org, revised Aug 2023.
    4. Nosratabadi, Saeed & Mosavi, Amir & Duan, Puhong & Ghamisi, Pedram & Filip, Ferdinand & Band, Shahab S. & Reuter, Uwe & Gama, Joao & Gandomi, Amir H., 2020. "Data science in economics: comprehensive review of advanced machine learning and deep learning methods," LawArXiv kczj5, Center for Open Science.
    5. Adamantios Ntakaris & Moncef Gabbouj & Juho Kanniainen, 2023. "Optimum Output Long Short-Term Memory Cell for High-Frequency Trading Forecasting," Papers 2304.09840, arXiv.org, revised May 2023.
    6. Shuo Sun & Rundong Wang & Bo An, 2021. "Reinforcement Learning for Quantitative Trading," Papers 2109.13851, arXiv.org.
    7. Saeed Nosratabadi & Amir Mosavi & Puhong Duan & Pedram Ghamisi, 2020. "Data Science in Economics," Papers 2003.13422, arXiv.org.
    8. Bryan Lim & Stefan Zohren & Stephen Roberts, 2019. "Enhancing Time Series Momentum Strategies Using Deep Neural Networks," Papers 1904.04912, arXiv.org, revised Sep 2020.
    9. Saeed Nosratabadi & Amirhosein Mosavi & Puhong Duan & Pedram Ghamisi & Ferdinand Filip & Shahab S. Band & Uwe Reuter & Joao Gama & Amir H. Gandomi, 2020. "Data Science in Economics: Comprehensive Review of Advanced Machine Learning and Deep Learning Methods," Mathematics, MDPI, vol. 8(10), pages 1-25, October.
    10. Nosratabadi, Saeed & Mosavi, Amir & Duan, Puhong & Ghamisi, Pedram & Filip, Ferdinand & Band, Shahab S. & Reuter, Uwe & Gama, Joao & Gandomi, Amir H., 2020. "Data science in economics: comprehensive review of advanced machine learning and deep learning methods," MetaArXiv haf2v, Center for Open Science.
    11. Xiao-Yang Liu & Hongyang Yang & Jiechao Gao & Christina Dan Wang, 2021. "FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance," Papers 2111.09395, arXiv.org.
    12. Jie Zou & Jiashu Lou & Baohua Wang & Sixue Liu, 2022. "A Novel Deep Reinforcement Learning Based Automated Stock Trading System Using Cascaded LSTM Networks," Papers 2212.02721, arXiv.org, revised Jul 2023.
    13. Nosratabadi, Saeed & Mosavi, Amir & Duan, Puhong & Ghamisi, Pedram & Filip, Ferdinand & Band, Shahab S. & Reuter, Uwe & Gama, Joao & Gandomi, Amir H., 2020. "Data science in economics: comprehensive review of advanced machine learning and deep learning methods," OSF Preprints yc6e2, Center for Open Science.
    14. Nosratabadi, Saeed & Mosavi, Amir & Duan, Puhong & Ghamisi, Pedram & Filip, Ferdinand & Band, Shahab S. & Reuter, Uwe & Gama, Joao & Gandomi, Amir H., 2020. "Data science in economics: comprehensive review of advanced machine learning and deep learning methods," EdArXiv 5dwrt, Center for Open Science.
    15. Nosratabadi, Saeed & Mosavi, Amir & Duan, Puhong & Ghamisi, Pedram & Filip, Ferdinand & Band, Shahab S. & Reuter, Uwe & Gama, Joao & Gandomi, Amir H., 2020. "Data science in economics: comprehensive review of advanced machine learning and deep learning methods," SocArXiv 9vdwf, Center for Open Science.
    16. Nosratabadi, Saeed & Mosavi, Amir & Duan, Puhong & Ghamisi, Pedram & Filip, Ferdinand & Band, Shahab S. & Reuter, Uwe & Gama, Joao & Gandomi, Amir H., 2020. "Data science in economics: comprehensive review of advanced machine learning and deep learning methods," Thesis Commons auyvc, Center for Open Science.
    17. Zihao Zhang & Stefan Zohren, 2021. "Multi-Horizon Forecasting for Limit Order Books: Novel Deep Learning Approaches and Hardware Acceleration using Intelligent Processing Units," Papers 2105.10430, arXiv.org, revised Aug 2021.
    18. Zihao Zhang & Bryan Lim & Stefan Zohren, 2021. "Deep Learning for Market by Order Data," Papers 2102.08811, arXiv.org, revised Jul 2021.
    19. Xiao-Yang Liu & Hongyang Yang & Qian Chen & Runjia Zhang & Liuqing Yang & Bowen Xiao & Christina Dan Wang, 2020. "FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance," Papers 2011.09607, arXiv.org, revised Mar 2022.
    20. Antonio Briola & Jeremy Turiel & Tomaso Aste, 2020. "Deep Learning modeling of Limit Order Book: a comparative perspective," Papers 2007.07319, arXiv.org, revised Oct 2020.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:1911.10107. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.