IDEAS home Printed from https://ideas.repec.org/p/zbw/iwqwdp/052020.html
   My bibliography  Save this paper

Deep reinforcement learning for the optimal placement of cryptocurrency limit orders

Author

Listed:
  • Schnaubelt, Matthias

Abstract

This paper presents the first large-scale application of deep reinforcement learning to optimize the placement of limit orders at cryptocurrency exchanges. For training and out-of-sample evaluation, we use a virtual limit order exchange to reward agents according to the realized shortfall over a series of time steps. Based on the literature, we generate features that inform the agent about the current market state. Leveraging 18 months of high-frequency data with 300 million historic trades and more than 3.5 million order book states from major exchanges and currency pairs, we empirically compare state-of-the-art deep reinforcement learning algorithms to several benchmarks. We find proximal policy optimization to reliably learn superior order placement strategies when compared to deep double Q-networks and other benchmarks. Further analyses shed light into the black box of the learned execution strategy. Important features are current liquidity costs and queue imbalances, where the latter can be interpreted as predictors of short-term mid-price returns. To preferably execute volume in limit orders to avoid additional market order exchange fees, order placement tends to be more aggressive in expectation of unfavorable price movements.

Suggested Citation

  • Schnaubelt, Matthias, 2020. "Deep reinforcement learning for the optimal placement of cryptocurrency limit orders," FAU Discussion Papers in Economics 05/2020, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics.
  • Handle: RePEc:zbw:iwqwdp:052020
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/216206/1/1696077540.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Erhan Bayraktar & Michael Ludkovski, 2014. "Liquidation In Limit Order Books With Controlled Intensity," Mathematical Finance, Wiley Blackwell, vol. 24(4), pages 627-650, October.
    2. �lvaro Cartea & Sebastian Jaimungal, 2015. "Optimal execution with limit and market orders," Quantitative Finance, Taylor & Francis Journals, vol. 15(8), pages 1279-1291, August.
    3. Martin D. Gould & Mason A. Porter & Stacy Williams & Mark McDonald & Daniel J. Fenn & Sam D. Howison, 2013. "Limit order books," Quantitative Finance, Taylor & Francis Journals, vol. 13(11), pages 1709-1742, November.
    4. Thomas Günter Fischer & Christopher Krauss & Alexander Deinert, 2019. "Statistical Arbitrage in Cryptocurrency Markets," JRFM, MDPI, vol. 12(1), pages 1-15, February.
    5. Peter Gomber & Uwe Schweickert & Erik Theissen, 2015. "Liquidity Dynamics in an Electronic Open Limit Order Book: an Event Study Approach," European Financial Management, European Financial Management Association, vol. 21(1), pages 52-78, January.
    6. Brian Ning & Franco Ho Ting Lin & Sebastian Jaimungal, 2018. "Double Deep Q-Learning for Optimal Execution," Papers 1812.06600, arXiv.org, revised Jun 2020.
    7. Martin D. Gould & Mason A. Porter & Stacy Williams & Mark McDonald & Daniel J. Fenn & Sam D. Howison, 2010. "Limit Order Books," Papers 1012.0349, arXiv.org, revised Apr 2013.
    8. Justin A. Sirignano, 2019. "Deep learning for limit order books," Quantitative Finance, Taylor & Francis Journals, vol. 19(4), pages 549-570, April.
    9. Charles Cao & Oliver Hansch & Xiaoxin Wang, 2009. "The information content of an open limit‐order book," Journal of Futures Markets, John Wiley & Sons, Ltd., vol. 29(1), pages 16-41, January.
    10. Ranaldo, Angelo, 2004. "Order aggressiveness in limit order book markets," Journal of Financial Markets, Elsevier, vol. 7(1), pages 53-74, January.
    11. Obizhaeva, Anna A. & Wang, Jiang, 2013. "Optimal trading strategy and supply/demand dynamics," Journal of Financial Markets, Elsevier, vol. 16(1), pages 1-32.
    12. Parameswaran Gopikrishnan & Vasiliki Plerou & Xavier Gabaix & H. Eugene Stanley, 2000. "Statistical Properties of Share Volume Traded in Financial Markets," Papers cond-mat/0008113, arXiv.org.
    13. Potters, Marc & Bouchaud, Jean-Philippe, 2003. "More statistical properties of order books and price impact," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 324(1), pages 133-140.
    14. Hans Degryse & Frank De Jong & Maarten Van Ravenswaaij & Gunther Wuyts, 2005. "Aggressive Orders and the Resiliency of a Limit Order Market," Review of Finance, European Finance Association, vol. 9(2), pages 201-242.
    15. Matthias Schnaubelt & Jonas Rende & Christopher Krauss, 2019. "Testing Stylized Facts of Bitcoin Limit Order Books," JRFM, MDPI, vol. 12(1), pages 1-30, February.
    16. Rama Cont & Arseniy Kukanov & Sasha Stoikov, 2014. "The Price Impact of Order Book Events," The Journal of Financial Econometrics, Society for Financial Econometrics, vol. 12(1), pages 47-88.
    17. Biais, Bruno & Hillion, Pierre & Spatt, Chester, 1995. "An Empirical Analysis of the Limit Order Book and the Order Flow in the Paris Bourse," Journal of Finance, American Finance Association, vol. 50(5), pages 1655-1689, December.
    18. Tashman, Leonard J., 2000. "Out-of-sample tests of forecasting accuracy: an analysis and review," International Journal of Forecasting, Elsevier, vol. 16(4), pages 437-450.
    19. Wenhang Bao & Xiao-yang Liu, 2019. "Multi-Agent Deep Reinforcement Learning for Liquidation Strategy Analysis," Papers 1906.11046, arXiv.org.
    20. Rama Cont & Arseniy Kukanov, 2017. "Optimal order placement in limit order markets," Quantitative Finance, Taylor & Francis Journals, vol. 17(1), pages 21-39, January.
    21. Zhang, Guoqiang & Eddy Patuwo, B. & Y. Hu, Michael, 1998. "Forecasting with artificial neural networks:: The state of the art," International Journal of Forecasting, Elsevier, vol. 14(1), pages 35-62, March.
    22. Yagna Patel, 2018. "Optimizing Market Making using Multi-Agent Reinforcement Learning," Papers 1812.10252, arXiv.org.
    23. Dieter Hendricks & Diane Wilcox, 2014. "A reinforcement learning extension to the Almgren-Chriss model for optimal trade execution," Papers 1403.2229, arXiv.org.
    24. Bertsimas, Dimitris & Lo, Andrew W., 1998. "Optimal control of execution costs," Journal of Financial Markets, Elsevier, vol. 1(1), pages 1-50, April.
    25. James Richard Cummings & Alex Frino, 2010. "Further analysis of the speed of response to large trades in interest rate futures," Journal of Futures Markets, John Wiley & Sons, Ltd., vol. 30(8), pages 705-724, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Schnaubelt, Matthias, 2022. "Deep reinforcement learning for the optimal placement of cryptocurrency limit orders," European Journal of Operational Research, Elsevier, vol. 296(3), pages 993-1006.
    2. Matthias Schnaubelt & Jonas Rende & Christopher Krauss, 2019. "Testing Stylized Facts of Bitcoin Limit Order Books," JRFM, MDPI, vol. 12(1), pages 1-30, February.
    3. Hai-Chuan Xu & Wei Chen & Xiong Xiong & Wei Zhang & Wei-Xing Zhou & H Eugene Stanley, 2016. "Limit-order book resiliency after effective market orders: Spread, depth and intensity," Papers 1602.00731, arXiv.org, revised Feb 2017.
    4. Yamamoto, Ryuichi, 2019. "Dynamic Predictor Selection And Order Splitting In A Limit Order Market," Macroeconomic Dynamics, Cambridge University Press, vol. 23(5), pages 1757-1792, July.
    5. Martin D. Gould & Mason A. Porter & Stacy Williams & Mark McDonald & Daniel J. Fenn & Sam D. Howison, 2010. "Limit Order Books," Papers 1012.0349, arXiv.org, revised Apr 2013.
    6. Ningyuan Chen & Steven Kou & Chun Wang, 2018. "A Partitioning Algorithm for Markov Decision Processes with Applications to Market Microstructure," Management Science, INFORMS, vol. 64(2), pages 784-803, February.
    7. Olivier Guéant, 2016. "The Financial Mathematics of Market Liquidity: From Optimal Execution to Market Making," Post-Print hal-01393136, HAL.
    8. Saran Ahuja & George Papanicolaou & Weiluo Ren & Tzu-Wei Yang, 2016. "Limit order trading with a mean reverting reference price," Papers 1607.00454, arXiv.org, revised Nov 2016.
    9. S. C. P. Yam & W. Zhou, 2017. "Optimal Liquidation of Child Limit Orders," Mathematics of Operations Research, INFORMS, vol. 42(2), pages 517-545, May.
    10. Kashyap, Ravi, 2020. "David vs Goliath (You against the Markets), A dynamic programming approach to separate the impact and timing of trading costs," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 545(C).
    11. Martin D. Gould & Mason A. Porter & Stacy Williams & Mark McDonald & Daniel J. Fenn & Sam D. Howison, 2013. "Limit order books," Quantitative Finance, Taylor & Francis Journals, vol. 13(11), pages 1709-1742, November.
    12. Chen, Yuanyuan & Gao, Xuefeng & Li, Duan, 2018. "Optimal order execution using hidden orders," Journal of Economic Dynamics and Control, Elsevier, vol. 94(C), pages 89-116.
    13. repec:zbw:cfswop:wp200918 is not listed on IDEAS
    14. Xuefeng Gao & S. J. Deng, 2014. "Hydrodynamic limit of order book dynamics," Papers 1411.7502, arXiv.org, revised Feb 2016.
    15. Danny Lo, 2015. "Essays in Market Microstructure and Investor Trading," PhD Thesis, Finance Discipline Group, UTS Business School, University of Technology, Sydney, number 4-2015.
    16. Ben Hambly & Renyuan Xu & Huining Yang, 2021. "Recent Advances in Reinforcement Learning in Finance," Papers 2112.04553, arXiv.org, revised Aug 2022.
    17. Siu, Chi Chung & Guo, Ivan & Zhu, Song-Ping & Elliott, Robert J., 2019. "Optimal execution with regime-switching market resilience," Journal of Economic Dynamics and Control, Elsevier, vol. 101(C), pages 17-40.
    18. Härdle, Wolfgang Karl & Hautsch, Nikolaus & Mihoci, Andrija, 2012. "Modelling and forecasting liquidity supply using semiparametric factor dynamics," Journal of Empirical Finance, Elsevier, vol. 19(4), pages 610-625.
    19. Cebiroğlu, Gökhan & Horst, Ulrich, 2015. "Optimal order display in limit order markets with liquidity competition," Journal of Economic Dynamics and Control, Elsevier, vol. 58(C), pages 81-100.
    20. Danny Lo, 2015. "Essays in Market Microstructure and Investor Trading," PhD Thesis, Finance Discipline Group, UTS Business School, University of Technology, Sydney, number 22, July-Dece.
    21. Cebiroglu, Gökhan & Hautsch, Nikolaus & Horst, Ulrich, 2014. "Order exposure and liquidity coordination: Does hidden liquidity harm price efficiency?," CFS Working Paper Series 468, Center for Financial Studies (CFS).

    More about this item

    Keywords

    Finance; Optimal Execution; Limit Order Markets; Machine learning; Deep Reinforcement Learning;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:iwqwdp:052020. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/vierlde.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/vierlde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.