IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2006.05574.html
   My bibliography  Save this paper

Multi-Agent Reinforcement Learning in a Realistic Limit Order Book Market Simulation

Author

Listed:
  • Michael Karpe
  • Jin Fang
  • Zhongyao Ma
  • Chen Wang

Abstract

Optimal order execution is widely studied by industry practitioners and academic researchers because it determines the profitability of investment decisions and high-level trading strategies, particularly those involving large volumes of orders. However, complex and unknown market dynamics pose significant challenges for the development and validation of optimal execution strategies. In this paper, we propose a model-free approach by training Reinforcement Learning (RL) agents in a realistic market simulation environment with multiple agents. First, we configure a multi-agent historical order book simulation environment for execution tasks built on an Agent-Based Interactive Discrete Event Simulation (ABIDES) [arXiv:1904.12066]. Second, we formulate the problem of optimal execution in an RL setting where an intelligent agent can make order execution and placement decisions based on market microstructure trading signals in High Frequency Trading (HFT). Third, we develop and train an RL execution agent using the Double Deep Q-Learning (DDQL) algorithm in the ABIDES environment. In some scenarios, our RL agent converges towards a Time-Weighted Average Price (TWAP) strategy. Finally, we evaluate the simulation with our RL agent by comparing it with a market replay simulation using real market Limit Order Book (LOB) data.

Suggested Citation

  • Michael Karpe & Jin Fang & Zhongyao Ma & Chen Wang, 2020. "Multi-Agent Reinforcement Learning in a Realistic Limit Order Book Market Simulation," Papers 2006.05574, arXiv.org, revised Sep 2020.
    • Handle: RePEc:arx:papers:2006.05574
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2006.05574
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Svitlana Vyetrenko & David Byrd & Nick Petosa & Mahmoud Mahfouz & Danial Dervovic & Manuela Veloso & Tucker Hybinette Balch, 2019. "Get Real: Realism Metrics for Robust Limit Order Book Market Simulations," Papers 1912.04941, arXiv.org.
    2. Wenhang Bao, 2019. "Fairness in Multi-agent Reinforcement Learning for Stock Trading," Papers 2001.00918, arXiv.org.
    3. Wenhang Bao & Xiao-yang Liu, 2019. "Multi-Agent Deep Reinforcement Learning for Liquidation Strategy Analysis," Papers 1906.11046, arXiv.org.
    4. Yagna Patel, 2018. "Optimizing Market Making using Multi-Agent Reinforcement Learning," Papers 1812.10252, arXiv.org.
    5. Abergel,Frédéric & Anane,Marouane & Chakraborti,Anirban & Jedidi,Aymen & Muni Toke,Ioane, 2016. "Limit Order Books," Cambridge Books, Cambridge University Press, number 9781107163980.
    6. Thomas Spooner & John Fearnley & Rahul Savani & Andreas Koukorinis, 2018. "Market Making via Reinforcement Learning," Papers 1804.04216, arXiv.org.
    7. Frédéric Abergel & Anirban Chakraborti & Aymen Jedidi & Ioane Muni Toke & Marouane Anane, 2016. "Limit Order Books," Post-Print hal-02177394, HAL.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Adam Bouland & Wim van Dam & Hamed Joorati & Iordanis Kerenidis & Anupam Prakash, 2020. "Prospects and challenges of quantum finance," Papers 2011.06492, arXiv.org.
    2. Xiao-Yang Liu & Jingyang Rui & Jiechao Gao & Liuqing Yang & Hongyang Yang & Zhaoran Wang & Christina Dan Wang & Jian Guo, 2021. "FinRL-Meta: A Universe of Near-Real Market Environments for Data-Driven Deep Reinforcement Learning in Quantitative Finance," Papers 2112.06753, arXiv.org, revised Mar 2022.
    3. Jin Fang & Jiacheng Weng & Yi Xiang & Xinwen Zhang, 2022. "Imitate then Transcend: Multi-Agent Optimal Execution with Dual-Window Denoise PPO," Papers 2206.10736, arXiv.org.
    4. Ben Hambly & Renyuan Xu & Huining Yang, 2021. "Recent Advances in Reinforcement Learning in Finance," Papers 2112.04553, arXiv.org, revised Feb 2023.
    5. Antonio Briola & Jeremy Turiel & Riccardo Marcaccioli & Alvaro Cauderan & Tomaso Aste, 2021. "Deep Reinforcement Learning for Active High Frequency Trading," Papers 2101.07107, arXiv.org, revised Aug 2023.
    6. Zijian Shi & John Cartlidge, 2023. "Neural Stochastic Agent-Based Limit Order Book Simulation: A Hybrid Methodology," Papers 2303.00080, arXiv.org.
    7. Peer Nagy & Jan-Peter Calliess & Stefan Zohren, 2023. "Asynchronous Deep Double Duelling Q-Learning for Trading-Signal Execution in Limit Order Book Markets," Papers 2301.08688, arXiv.org, revised Sep 2023.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Thomas Spooner & Rahul Savani, 2020. "Robust Market Making via Adversarial Reinforcement Learning," Papers 2003.01820, arXiv.org, revised Jul 2020.
    2. Svitlana Vyetrenko & David Byrd & Nick Petosa & Mahmoud Mahfouz & Danial Dervovic & Manuela Veloso & Tucker Hybinette Balch, 2019. "Get Real: Realism Metrics for Robust Limit Order Book Market Simulations," Papers 1912.04941, arXiv.org.
    3. Ben Hambly & Renyuan Xu & Huining Yang, 2021. "Recent Advances in Reinforcement Learning in Finance," Papers 2112.04553, arXiv.org, revised Feb 2023.
    4. Bruno Gašperov & Stjepan Begušić & Petra Posedel Šimović & Zvonko Kostanjčar, 2021. "Reinforcement Learning Approaches to Optimal Market Making," Mathematics, MDPI, vol. 9(21), pages 1-22, October.
    5. Peng Wu & Marcello Rambaldi & Jean-Franc{c}ois Muzy & Emmanuel Bacry, 2019. "Queue-reactive Hawkes models for the order flow," Papers 1901.08938, arXiv.org.
    6. Michael Karpe, 2020. "An overall view of key problems in algorithmic trading and recent progress," Papers 2006.05515, arXiv.org.
    7. Junyi Li & Xitong Wang & Yaoyang Lin & Arunesh Sinha & Micheal P. Wellman, 2020. "Generating Realistic Stock Market Order Streams," Papers 2006.04212, arXiv.org.
    8. Charles-Albert Lehalle & Eyal Neuman, 2019. "Incorporating signals into optimal trading," Finance and Stochastics, Springer, vol. 23(2), pages 275-311, April.
    9. Yufei Wu & Mahmoud Mahfouz & Daniele Magazzeni & Manuela Veloso, 2021. "How Robust are Limit Order Book Representations under Data Perturbation?," Papers 2110.04752, arXiv.org.
    10. Xiaofei Lu & Frédéric Abergel, 2017. "Limit order book modelling with high dimensional Hawkes processes," Working Papers hal-01512430, HAL.
    11. Nik Alexandrov & Dave Cliff & Charlie Figuero, 2021. "Exploring Coevolutionary Dynamics of Competitive Arms-Races Between Infinitely Diverse Heterogenous Adaptive Automated Trader-Agents," Papers 2109.10429, arXiv.org.
    12. Shuo Sun & Rundong Wang & Bo An, 2021. "Reinforcement Learning for Quantitative Trading," Papers 2109.13851, arXiv.org.
    13. Ioane Muni Toke, 2017. "Stationary Distribution Of The Volume At The Best Quote In A Poisson Order Book Model," Post-Print hal-01705085, HAL.
    14. Federico Gonzalez & Mark Schervish, 2017. "Instantaneous order impact and high-frequency strategy optimization in limit order books," Papers 1707.01167, arXiv.org, revised Oct 2017.
    15. Emmanouil Sfendourakis & Ioane Muni Toke, 2021. "LOB modeling using Hawkes processes with a state-dependent factor," Papers 2107.12872, arXiv.org, revised Dec 2021.
    16. Peng Wu & Marcello Rambaldi & Jean-François Muzy & Emmanuel Bacry, 2021. "Queue-reactive Hawkes models for the order flow," Working Papers hal-02409073, HAL.
    17. Xiaofei Lu & Fr'ed'eric Abergel, 2018. "Order-book modelling and market making strategies," Papers 1806.05101, arXiv.org.
    18. Masatoshi Goda, 2021. "Hawkes process and Edgeworth expansion with application to maximum likelihood estimator," Statistical Inference for Stochastic Processes, Springer, vol. 24(2), pages 277-325, July.
    19. Tommaso Mariotti & Fabrizio Lillo & Giacomo Toscano, 2022. "From Zero-Intelligence to Queue-Reactive: Limit Order Book modeling for high-frequency volatility estimation and optimal execution," Papers 2202.12137, arXiv.org, revised Sep 2022.
    20. Restocchi, Valerio & McGroarty, Frank & Gerding, Enrico, 2019. "Statistical properties of volume and calendar effects in prediction markets," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 523(C), pages 1150-1160.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2006.05574. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.