IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2212.14670.html
   My bibliography  Save this paper

Hierarchical Deep Reinforcement Learning for VWAP Strategy Optimization

Author

Listed:
  • Xiaodong Li
  • Pangjing Wu
  • Chenxin Zou
  • Qing Li

Abstract

Designing an intelligent volume-weighted average price (VWAP) strategy is a critical concern for brokers, since traditional rule-based strategies are relatively static that cannot achieve a lower transaction cost in a dynamic market. Many studies have tried to minimize the cost via reinforcement learning, but there are bottlenecks in improvement, especially for long-duration strategies such as the VWAP strategy. To address this issue, we propose a deep learning and hierarchical reinforcement learning jointed architecture termed Macro-Meta-Micro Trader (M3T) to capture market patterns and execute orders from different temporal scales. The Macro Trader first allocates a parent order into tranches based on volume profiles as the traditional VWAP strategy does, but a long short-term memory neural network is used to improve the forecasting accuracy. Then the Meta Trader selects a short-term subgoal appropriate to instant liquidity within each tranche to form a mini-tranche. The Micro Trader consequently extracts the instant market state and fulfils the subgoal with the lowest transaction cost. Our experiments over stocks listed on the Shanghai stock exchange demonstrate that our approach outperforms baselines in terms of VWAP slippage, with an average cost saving of 1.16 base points compared to the optimal baseline.

Suggested Citation

  • Xiaodong Li & Pangjing Wu & Chenxin Zou & Qing Li, 2022. "Hierarchical Deep Reinforcement Learning for VWAP Strategy Optimization," Papers 2212.14670, arXiv.org.
  • Handle: RePEc:arx:papers:2212.14670
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2212.14670
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Bialkowski, Jedrzej & Darolles, Serge & Le Fol, Gaëlle, 2008. "Improving VWAP strategies: A dynamic volume approach," Journal of Banking & Finance, Elsevier, vol. 32(9), pages 1709-1722, September.
    2. Yuchen Fang & Kan Ren & Weiqing Liu & Dong Zhou & Weinan Zhang & Jiang Bian & Yong Yu & Tie-Yan Liu, 2021. "Universal Trading for Order Execution with Oracle Policy Distillation," Papers 2103.10860, arXiv.org.
    3. Gur Huberman & Werner Stanzl, 2005. "Optimal Liquidity Trading," Review of Finance, Springer, vol. 9(2), pages 165-200, June.
    4. Dieter Hendricks & Diane Wilcox, 2014. "A reinforcement learning extension to the Almgren-Chriss model for optimal trade execution," Papers 1403.2229, arXiv.org.
    5. Konishi, Hizuru, 2002. "Optimal slice of a VWAP trade," Journal of Financial Markets, Elsevier, vol. 5(2), pages 197-221, April.
    6. Bertsimas, Dimitris & Lo, Andrew W., 1998. "Optimal control of execution costs," Journal of Financial Markets, Elsevier, vol. 1(1), pages 1-50, April.
    7. Joel Hasbrouck, 1999. "The Dynamics of Discrete Bid and Ask Quotes," Journal of Finance, American Finance Association, vol. 54(6), pages 2109-2142, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Woo Jae Byun & Bumkyu Choi & Seongmin Kim & Joohyun Jo, 2023. "Practical Application of Deep Reinforcement Learning to Optimal Trade Execution," FinTech, MDPI, vol. 2(3), pages 1-16, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Christopher Kath & Florian Ziel, 2020. "Optimal Order Execution in Intraday Markets: Minimizing Costs in Trade Trajectories," Papers 2009.07892, arXiv.org, revised Oct 2020.
    2. Olivier Guéant, 2016. "The Financial Mathematics of Market Liquidity: From Optimal Execution to Market Making," Post-Print hal-01393136, HAL.
    3. Shuo Sun & Rundong Wang & Bo An, 2021. "Reinforcement Learning for Quantitative Trading," Papers 2109.13851, arXiv.org.
    4. Feiyang Pan & Tongzhe Zhang & Ling Luo & Jia He & Shuoling Liu, 2022. "Learn Continuously, Act Discretely: Hybrid Action-Space Reinforcement Learning For Optimal Execution," Papers 2207.11152, arXiv.org.
    5. Soohan Kim & Jimyeong Kim & Hong Kee Sul & Youngjoon Hong, 2023. "An Adaptive Dual-level Reinforcement Learning Approach for Optimal Trade Execution," Papers 2307.10649, arXiv.org.
    6. Enzo Busseti & Stephen Boyd, 2015. "Volume Weighted Average Price Optimal Execution," Papers 1509.08503, arXiv.org.
    7. Yuchen Fang & Kan Ren & Weiqing Liu & Dong Zhou & Weinan Zhang & Jiang Bian & Yong Yu & Tie-Yan Liu, 2021. "Universal Trading for Order Execution with Oracle Policy Distillation," Papers 2103.10860, arXiv.org.
    8. Curatola, Giuliano, 2022. "Price impact, strategic interaction and portfolio choice," The North American Journal of Economics and Finance, Elsevier, vol. 59(C).
    9. Olivier Guéant & Charles-Albert Lehalle, 2015. "General Intensity Shapes In Optimal Liquidation," Mathematical Finance, Wiley Blackwell, vol. 25(3), pages 457-495, July.
    10. Samuel N. Cohen & Lukasz Szpruch, 2011. "A limit order book model for latency arbitrage," Papers 1110.4811, arXiv.org.
    11. Somayeh Moazeni & Thomas F. Coleman & Yuying Li, 2016. "Smoothing and parametric rules for stochastic mean-CVaR optimal execution strategy," Annals of Operations Research, Springer, vol. 237(1), pages 99-120, February.
    12. Woo Jae Byun & Bumkyu Choi & Seongmin Kim & Joohyun Jo, 2023. "Practical Application of Deep Reinforcement Learning to Optimal Trade Execution," FinTech, MDPI, vol. 2(3), pages 1-16, June.
    13. Steven L. Heston & Robert A. Korajczyk & Ronnie Sadka, 2010. "Intraday Patterns in the Cross‐section of Stock Returns," Journal of Finance, American Finance Association, vol. 65(4), pages 1369-1407, August.
    14. Oehmke, Martin, 2014. "Liquidating illiquid collateral," Journal of Economic Theory, Elsevier, vol. 149(C), pages 183-210.
    15. Olivier Gu'eant & Guillaume Royer, 2013. "VWAP execution and guaranteed VWAP," Papers 1306.2832, arXiv.org, revised May 2014.
    16. Griese, Knut & Kempf, Alexander, 2005. "Liquiditätsdynamik am deutschen Aktienmarkt," CFR Working Papers 05-12, University of Cologne, Centre for Financial Research (CFR).
    17. Frey, Stefan & Sandås, Patrik, 2009. "The impact of iceberg orders in limit order books," CFR Working Papers 09-06, University of Cologne, Centre for Financial Research (CFR).
    18. Takashi Kato, 2014. "VWAP Execution as an Optimal Strategy," Papers 1408.6118, arXiv.org, revised Jan 2017.
    19. Ye Xunyu & Yan Rui & Li Handong, 2014. "Forecasting trading volume in the Chinese stock market based on the dynamic VWAP," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 18(2), pages 125-144, April.
    20. Olivier Guéant & Royer Guillaume, 2014. "VWAP execution and guaranteed VWAP," Post-Print hal-01393121, HAL.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2212.14670. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.