IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2003.01820.html
   My bibliography  Save this paper

Robust Market Making via Adversarial Reinforcement Learning

Author

Listed:
  • Thomas Spooner
  • Rahul Savani

Abstract

We show that adversarial reinforcement learning (ARL) can be used to produce market marking agents that are robust to adversarial and adaptively-chosen market conditions. To apply ARL, we turn the well-studied single-agent model of Avellaneda and Stoikov [2008] into a discrete-time zero-sum game between a market maker and adversary. The adversary acts as a proxy for other market participants that would like to profit at the market maker's expense. We empirically compare two conventional single-agent RL agents with ARL, and show that our ARL approach leads to: 1) the emergence of risk-averse behaviour without constraints or domain-specific penalties; 2) significant improvements in performance across a set of standard metrics, evaluated with or without an adversary in the test environment, and; 3) improved robustness to model uncertainty. We empirically demonstrate that our ARL method consistently converges, and we prove for several special cases that the profiles that we converge to correspond to Nash equilibria in a simplified single-stage game.

Suggested Citation

  • Thomas Spooner & Rahul Savani, 2020. "Robust Market Making via Adversarial Reinforcement Learning," Papers 2003.01820, arXiv.org, revised Jul 2020.
  • Handle: RePEc:arx:papers:2003.01820
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2003.01820
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Olivier Guéant & Iuliia Manziuk, 2019. "Deep Reinforcement Learning for Market Making in Corporate Bonds: Beating the Curse of Dimensionality," Applied Mathematical Finance, Taylor & Francis Journals, vol. 26(5), pages 387-452, September.
    2. Sanford J. Grossman & Merton H. Miller, 1988. "Liquidity and Market Structure," NBER Working Papers 2641, National Bureau of Economic Research, Inc.
    3. Martin D. Gould & Mason A. Porter & Stacy Williams & Mark McDonald & Daniel J. Fenn & Sam D. Howison, 2010. "Limit Order Books," Papers 1012.0349, arXiv.org, revised Apr 2013.
    4. Pietro Fodra & Mauricio Labadie, 2012. "High-frequency market-making with inventory constraints and directional bets," Papers 1206.4810, arXiv.org.
    5. Martin D. Gould & Mason A. Porter & Stacy Williams & Mark McDonald & Daniel J. Fenn & Sam D. Howison, 2013. "Limit order books," Quantitative Finance, Taylor & Francis Journals, vol. 13(11), pages 1709-1742, November.
    6. repec:bla:jfinan:v:43:y:1988:i:3:p:617-37 is not listed on IDEAS
    7. Olivier Gu'eant & Iuliia Manziuk, 2019. "Deep reinforcement learning for market making in corporate bonds: beating the curse of dimensionality," Papers 1910.13205, arXiv.org.
    8. Ho, Thomas & Stoll, Hans R., 1981. "Optimal dealer pricing under transactions and return uncertainty," Journal of Financial Economics, Elsevier, vol. 9(1), pages 47-73, March.
    9. Marco Avellaneda & Sasha Stoikov, 2008. "High-frequency trading in a limit order book," Quantitative Finance, Taylor & Francis Journals, vol. 8(3), pages 217-224.
    10. Nicholas T. Chan and Christian Shelton, 2001. "An Adaptive Electronic Market-Maker," Computing in Economics and Finance 2001 146, Society for Computational Economics.
    11. Glosten, Lawrence R. & Milgrom, Paul R., 1985. "Bid, ask and transaction prices in a specialist market with heterogeneously informed traders," Journal of Financial Economics, Elsevier, vol. 14(1), pages 71-100, March.
    12. Abergel,Frédéric & Anane,Marouane & Chakraborti,Anirban & Jedidi,Aymen & Muni Toke,Ioane, 2016. "Limit Order Books," Cambridge Books, Cambridge University Press, number 9781107163980, October.
    13. Pietro Fodra & Mauricio Labadie, 2012. "High-frequency market-making with inventory constraints and directional bets," Working Papers hal-00675925, HAL.
    14. Thomas Spooner & John Fearnley & Rahul Savani & Andreas Koukorinis, 2018. "Market Making via Reinforcement Learning," Papers 1804.04216, arXiv.org.
    15. Frédéric Abergel & Anirban Chakraborti & Aymen Jedidi & Ioane Muni Toke & Marouane Anane, 2016. "Limit Order Books," Post-Print hal-02177394, HAL.
    16. Olivier Gu'eant, 2016. "Optimal market making," Papers 1605.01862, arXiv.org, revised May 2017.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ben Hambly & Renyuan Xu & Huining Yang, 2021. "Recent Advances in Reinforcement Learning in Finance," Papers 2112.04553, arXiv.org, revised Feb 2023.
    2. Ben Hambly & Renyuan Xu & Huining Yang, 2023. "Recent advances in reinforcement learning in finance," Mathematical Finance, Wiley Blackwell, vol. 33(3), pages 437-503, July.
    3. Zhou Fang & Haiqing Xu, 2023. "Market Making of Options via Reinforcement Learning," Papers 2307.01814, arXiv.org.
    4. Joseph Jerome & Gregory Palmer & Rahul Savani, 2022. "Market Making with Scaled Beta Policies," Papers 2207.03352, arXiv.org, revised Sep 2022.
    5. Shuo Sun & Rundong Wang & Bo An, 2021. "Reinforcement Learning for Quantitative Trading," Papers 2109.13851, arXiv.org.
    6. Leo Ardon & Nelson Vadori & Thomas Spooner & Mengda Xu & Jared Vann & Sumitra Ganesh, 2021. "Towards a fully RL-based Market Simulator," Papers 2110.06829, arXiv.org, revised Nov 2021.
    7. Nelson Vadori & Leo Ardon & Sumitra Ganesh & Thomas Spooner & Selim Amrouni & Jared Vann & Mengda Xu & Zeyu Zheng & Tucker Balch & Manuela Veloso, 2022. "Towards Multi-Agent Reinforcement Learning driven Over-The-Counter Market Simulations," Papers 2210.07184, arXiv.org, revised Aug 2023.
    8. Thomas Spooner & Rahul Savani, 2020. "A Natural Actor-Critic Algorithm with Downside Risk Constraints," Papers 2007.04203, arXiv.org.
    9. Bruno Gašperov & Stjepan Begušić & Petra Posedel Šimović & Zvonko Kostanjčar, 2021. "Reinforcement Learning Approaches to Optimal Market Making," Mathematics, MDPI, vol. 9(21), pages 1-22, October.
    10. Tessa Bauman & Bruno Gav{s}perov & Stjepan Beguv{s}i'c & Zvonko Kostanjv{c}ar, 2023. "Deep Reinforcement Learning for Robust Goal-Based Wealth Management," Papers 2307.13501, arXiv.org.
    11. Shuo Sun & Molei Qin & Xinrun Wang & Bo An, 2023. "PRUDEX-Compass: Towards Systematic Evaluation of Reinforcement Learning in Financial Markets," Papers 2302.00586, arXiv.org, revised Mar 2023.
    12. Joseph Jerome & Leandro Sanchez-Betancourt & Rahul Savani & Martin Herdegen, 2022. "Model-based gym environments for limit order book trading," Papers 2209.07823, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alexander Barzykin & Philippe Bergault & Olivier Gu'eant, 2021. "Algorithmic market making in dealer markets with hedging and market impact," Papers 2106.06974, arXiv.org, revised Dec 2022.
    2. Pankaj Kumar, 2021. "Deep Hawkes Process for High-Frequency Market Making," Papers 2109.15110, arXiv.org.
    3. Thomas Spooner & John Fearnley & Rahul Savani & Andreas Koukorinis, 2018. "Market Making via Reinforcement Learning," Papers 1804.04216, arXiv.org.
    4. Bruno Gašperov & Stjepan Begušić & Petra Posedel Šimović & Zvonko Kostanjčar, 2021. "Reinforcement Learning Approaches to Optimal Market Making," Mathematics, MDPI, vol. 9(21), pages 1-22, October.
    5. Qing-Qing Yang & Wai-Ki Ching & Jiawen Gu & Tak-Kuen Siu, 2020. "Trading strategy with stochastic volatility in a limit order book market," Decisions in Economics and Finance, Springer;Associazione per la Matematica, vol. 43(1), pages 277-301, June.
    6. Philippe Bergault & Olivier Guéant, 2021. "Size matters for OTC market makers: General results and dimensionality reduction techniques," Mathematical Finance, Wiley Blackwell, vol. 31(1), pages 279-322, January.
    7. Philippe Bergault & David Evangelista & Olivier Gu'eant & Douglas Vieira, 2018. "Closed-form approximations in multi-asset market making," Papers 1810.04383, arXiv.org, revised Sep 2022.
    8. Alexander Barzykin & Philippe Bergault & Olivier Guéant, 2023. "Algorithmic market making in dealer markets with hedging and market impact," Mathematical Finance, Wiley Blackwell, vol. 33(1), pages 41-79, January.
    9. Philippe Bergault & Louis Bertucci & David Bouba & Olivier Gu'eant & Julien Guilbert, 2024. "Price-Aware Automated Market Makers: Models Beyond Brownian Prices and Static Liquidity," Papers 2405.03496, arXiv.org, revised May 2024.
    10. Nelson Vadori & Leo Ardon & Sumitra Ganesh & Thomas Spooner & Selim Amrouni & Jared Vann & Mengda Xu & Zeyu Zheng & Tucker Balch & Manuela Veloso, 2022. "Towards Multi-Agent Reinforcement Learning driven Over-The-Counter Market Simulations," Papers 2210.07184, arXiv.org, revised Aug 2023.
    11. Marcello Monga, 2024. "Automated Market Making and Decentralized Finance," Papers 2407.16885, arXiv.org.
    12. Bastien Baldacci & Philippe Bergault & Olivier Gu'eant, 2019. "Algorithmic market making for options," Papers 1907.12433, arXiv.org, revised Jul 2020.
    13. Burcu Aydoğan & Ömür Uğur & Ümit Aksoy, 2023. "Optimal Limit Order Book Trading Strategies with Stochastic Volatility in the Underlying Asset," Computational Economics, Springer;Society for Computational Economics, vol. 62(1), pages 289-324, June.
    14. Philippe Bergault & Olivier Gu'eant, 2019. "Size matters for OTC market makers: general results and dimensionality reduction techniques," Papers 1907.01225, arXiv.org, revised Sep 2022.
    15. Sumitra Ganesh & Nelson Vadori & Mengda Xu & Hua Zheng & Prashant Reddy & Manuela Veloso, 2019. "Reinforcement Learning for Market Making in a Multi-agent Dealer Market," Papers 1911.05892, arXiv.org.
    16. Campi, Luciano & Zabaljauregui, Diego, 2020. "Optimal market making under partial information with general intensities," LSE Research Online Documents on Economics 104612, London School of Economics and Political Science, LSE Library.
    17. Bastien Baldacci & Joffrey Derchu & Iuliia Manziuk, 2020. "An approximate solution for options market-making in high dimension," Papers 2009.00907, arXiv.org.
    18. Bastien Baldacci & Jerome Benveniste & Gordon Ritter, 2020. "Optimal trading without optimal control," Papers 2012.12945, arXiv.org.
    19. Hui Niu & Siyuan Li & Jiahao Zheng & Zhouchi Lin & Jian Li & Jian Guo & Bo An, 2023. "IMM: An Imitative Reinforcement Learning Approach with Predictive Representation Learning for Automatic Market Making," Papers 2308.08918, arXiv.org.
    20. Abbas Haider & Hui Wang & Bryan Scotney & Glenn Hawe, 2022. "Predictive Market Making via Machine Learning," SN Operations Research Forum, Springer, vol. 3(1), pages 1-21, March.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2003.01820. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.