IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2408.11773.html

Deviations from the Nash equilibrium in a two-player optimal execution game with reinforcement learning

Author

Listed:
  • Fabrizio Lillo
  • Andrea Macr`i

Abstract

The use of reinforcement learning algorithms in financial trading is becoming increasingly prevalent. However, the autonomous nature of these algorithms can lead to unexpected outcomes that deviate from traditional game-theoretical predictions and may even destabilize markets. In this study, we examine a scenario in which two autonomous agents, modelled with Double Deep Q-Learning, learn to liquidate the same asset optimally in the presence of market impact, under the Almgren-Chriss (2000) framework. We show that the strategies learned by the agents deviate significantly from the Nash equilibrium of the corresponding market impact game. Notably, the learned strategies exhibit supra-competitive solution, {which might be compatible with a tacit collusive behaviour}, closely aligning with the Pareto-optimal solution. We further explore how different levels of market volatility influence the agents' performance and the equilibria they discover, including scenarios where volatility differs between the training and testing phases.

Suggested Citation

  • Fabrizio Lillo & Andrea Macr`i, 2024. "Deviations from the Nash equilibrium in a two-player optimal execution game with reinforcement learning," Papers 2408.11773, arXiv.org, revised Feb 2026.
    • Handle: RePEc:arx:papers:2408.11773
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2408.11773
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Alexander Schied & Tao Zhang, 2019. "A Market Impact Game Under Transient Price Impact," Mathematics of Operations Research, INFORMS, vol. 44(1), pages 102-121, February.
    2. Schnaubelt, Matthias, 2022. "Deep reinforcement learning for the optimal placement of cryptocurrency limit orders," European Journal of Operational Research, Elsevier, vol. 296(3), pages 993-1006.
    3. Alexander Schied & Tao Zhang, 2017. "A State-Constrained Differential Game Arising In Optimal Portfolio Liquidation," Mathematical Finance, Wiley Blackwell, vol. 27(3), pages 779-802, July.
    4. Matthias Hettich, 2021. "Algorithmic Collusion: Insights from Deep Learning," CQE Working Papers 9421, Center for Quantitative Economics (CQE), University of Muenster.
    5. Ben Hambly & Renyuan Xu & Huining Yang, 2023. "Recent advances in reinforcement learning in finance," Mathematical Finance, Wiley Blackwell, vol. 33(3), pages 437-503, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Moritz Voß, 2022. "A two-player portfolio tracking game," Mathematics and Financial Economics, Springer, volume 16, number 6, March.
    2. Tonkin, Isaac & Gepp, Adrian & Harris, Geoff & Vanstone, Bruce, 2025. "Benchmarking deep reinforcement learning approaches to trade execution," Pacific-Basin Finance Journal, Elsevier, vol. 94(C).
    3. Moritz Vo{ss}, 2019. "A two-player portfolio tracking game," Papers 1911.05122, arXiv.org, revised Jul 2022.
    4. Samuel Drapeau & Peng Luo & Alexander Schied & Dewen Xiong, 2019. "An FBSDE approach to market impact games with stochastic parameters," Papers 2001.00622, arXiv.org.
    5. Francesco Cordoni & Fabrizio Lillo, 2024. "Instabilities in multi-asset and multi-agent market impact games," Annals of Operations Research, Springer, vol. 336(1), pages 505-539, May.
    6. Steven Campbell & Marcel Nutz, 2025. "Optimal Execution among $N$ Traders with Transient Price Impact," Papers 2501.09638, arXiv.org.
    7. Chau, Huy & Nguyen, Duy & Nguyen, Thai, 2026. "Continuous-time optimal investment with portfolio constraints: A reinforcement learning approach," European Journal of Operational Research, Elsevier, vol. 328(3), pages 1068-1092.
    8. Nicole Bäuerle & Tamara Göll, 2024. "Nash equilibria for relative investors with (non)linear price impact," Mathematics and Financial Economics, Springer, volume 18, number 2, March.
    9. Bouyaddou, Youssef & Jebabli, Ikram, 2025. "Integration of investor behavioral perspective and climate change in reinforcement learning for portfolio optimization," Research in International Business and Finance, Elsevier, vol. 73(PB).
    10. Yinhong Dong & Donglei Du & Qiaoming Han & Jianfeng Ren & Dachuan Xu, 2024. "A Stackelberg order execution game," Annals of Operations Research, Springer, vol. 336(1), pages 571-604, May.
    11. Guanxing Fu & Ulrich Horst & Xiaonyu Xia, 2022. "Portfolio liquidation games with self‐exciting order flow," Mathematical Finance, Wiley Blackwell, vol. 32(4), pages 1020-1065, October.
    12. Steven Campbell & Marcel Nutz, 2025. "Randomization in Optimal Execution Games," Papers 2503.08833, arXiv.org.
    13. Francesco Cordoni & Fabrizio Lillo, 2022. "Transient impact from the Nash equilibrium of a permanent market impact game," Papers 2205.00494, arXiv.org, revised Mar 2023.
    14. Xianfeng Jiao & Zizhong Li & Chang Xu & Yang Liu & Weiqing Liu & Jiang Bian, 2023. "Microstructure-Empowered Stock Factor Extraction and Utilization," Papers 2308.08135, arXiv.org.
    15. Eyal Neuman & Moritz Voß, 2023. "Trading with the crowd," Mathematical Finance, Wiley Blackwell, vol. 33(3), pages 548-617, July.
    16. Rzayev, Khaladdin & Sakkas, Athanasios & Urquhart, Andrew, 2025. "An adoption model of cryptocurrencies," European Journal of Operational Research, Elsevier, vol. 323(1), pages 253-266.
    17. François, Pascal & Gauthier, Geneviève & Godin, Frédéric & Mendoza, Carlos Octavio Pérez, 2025. "Is the difference between deep hedging and delta hedging a statistical arbitrage?," Finance Research Letters, Elsevier, vol. 73(C).
    18. Alejandra de-la-Rica-Escudero & Eduardo C Garrido-Merchán & María Coronado-Vaca, 2025. "Explainable post hoc portfolio management financial policy of a Deep Reinforcement Learning agent," PLOS ONE, Public Library of Science, vol. 20(1), pages 1-19, January.
    19. Yan Dolinsky & Shir Moshe, 2021. "Utility Indifference Pricing with High Risk Aversion and Small Linear Price Impact," Papers 2111.00451, arXiv.org, revised Jan 2022.
    20. Wu, Bo & Li, Lingfei, 2024. "Reinforcement learning for continuous-time mean-variance portfolio selection in a regime-switching market," Journal of Economic Dynamics and Control, Elsevier, vol. 158(C).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2408.11773. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.