IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2301.08360.html
   My bibliography  Save this paper

Domain-adapted Learning and Imitation: DRL for Power Arbitrage

Author

Listed:
  • Yuanrong Wang
  • Vignesh Raja Swaminathan
  • Nikita P. Granger
  • Carlos Ros Perez
  • Christian Michler

Abstract

In this paper, we discuss the Dutch power market, which is comprised of a day-ahead market and an intraday balancing market that operates like an auction. Due to fluctuations in power supply and demand, there is often an imbalance that leads to different prices in the two markets, providing an opportunity for arbitrage. To address this issue, we restructure the problem and propose a collaborative dual-agent reinforcement learning approach for this bi-level simulation and optimization of European power arbitrage trading. We also introduce two new implementations designed to incorporate domain-specific knowledge by imitating the trading behaviours of power traders. By utilizing reward engineering to imitate domain expertise, we are able to reform the reward system for the RL agent, which improves convergence during training and enhances overall performance. Additionally, the tranching of orders increases bidding success rates and significantly boosts profit and loss (P&L). Our study demonstrates that by leveraging domain expertise in a general learning problem, the performance can be improved substantially, and the final integrated approach leads to a three-fold improvement in cumulative P&L compared to the original agent. Furthermore, our methodology outperforms the highest benchmark policy by around 50% while maintaining efficient computational performance.

Suggested Citation

  • Yuanrong Wang & Vignesh Raja Swaminathan & Nikita P. Granger & Carlos Ros Perez & Christian Michler, 2023. "Domain-adapted Learning and Imitation: DRL for Power Arbitrage," Papers 2301.08360, arXiv.org, revised Sep 2023.
  • Handle: RePEc:arx:papers:2301.08360
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2301.08360
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Yuanrong Wang & Tomaso Aste, 2022. "Sparsification and Filtering for Spatial-temporal GNN in Multivariate Time-series," Papers 2203.03991, arXiv.org.
    2. Zou, Peng & Chen, Qixin & Xia, Qing & He, Guannan & Kang, Chongqing & Conejo, Antonio J., 2016. "Pool equilibria including strategic storage," Applied Energy, Elsevier, vol. 177(C), pages 260-270.
    3. Rui Albuquerque, 2012. "Skewness in Stock Returns: Reconciling the Evidence on Firm Versus Aggregate Returns," The Review of Financial Studies, Society for Financial Studies, vol. 25(5), pages 1630-1673.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Huang, Qisheng & Xu, Yunjian & Courcoubetis, Costas, 2020. "Stackelberg competition between merchant and regulated storage investment in wholesale electricity markets," Applied Energy, Elsevier, vol. 264(C).
    2. Chang, Bo Young & Christoffersen, Peter & Jacobs, Kris, 2013. "Market skewness risk and the cross section of stock returns," Journal of Financial Economics, Elsevier, vol. 107(1), pages 46-68.
    3. Robert Brooks & Robert Faff & Sirimon Treepongkaruna & Eliza Wu, 2015. "Do Sovereign Re-Ratings Destabilize Equity Markets during Financial Crises? New Evidence from Higher Return Moments," Journal of Business Finance & Accounting, Wiley Blackwell, vol. 42(5-6), pages 777-799, June.
    4. Jondeau, Eric & Zhang, Qunzi & Zhu, Xiaoneng, 2019. "Average skewness matters," Journal of Financial Economics, Elsevier, vol. 134(1), pages 29-47.
    5. Aryani, Morteza & Ahmadian, Mohammad & Sheikh-El-Eslami, Mohammad-Kazem, 2020. "Designing a regulatory tool for coordinated investment in renewable and conventional generation capacities considering market equilibria," Applied Energy, Elsevier, vol. 279(C).
    6. Annaert, Jan & De Ceuster, Marc & Van Cappellen, Jef, 2023. "Can average skewness really predict financial returns? The euro area case," Finance Research Letters, Elsevier, vol. 52(C).
    7. Wu, Qi & Yan, Xing, 2019. "Capturing deep tail risk via sequential learning of quantile dynamics," Journal of Economic Dynamics and Control, Elsevier, vol. 109(C).
    8. Stephen G Dimmock & Roy Kouwenberg & Olivia S Mitchell & Kim Peijnenburg, 2021. "Household Portfolio Underdiversification and Probability Weighting: Evidence from the Field," The Review of Financial Studies, Society for Financial Studies, vol. 34(9), pages 4524-4563.
    9. Pfeifer, Antun & Feijoo, Felipe & Duić, Neven, 2023. "Fast energy transition as a best strategy for all? The nash equilibrium of long-term energy planning strategies in coupled power markets," Energy, Elsevier, vol. 284(C).
    10. Yuanrong Wang & Yinsen Miao & Alexander CY Wong & Nikita P Granger & Christian Michler, 2023. "Domain-adapted Learning and Interpretability: DRL for Gas Trading," Papers 2301.08359, arXiv.org, revised Sep 2023.
    11. Ayadi, Mohamed A. & Cao, Xu & Lazrak, Skander & Wang, Yan, 2019. "Do idiosyncratic skewness and kurtosis really matter?," The North American Journal of Economics and Finance, Elsevier, vol. 50(C).
    12. Bae, Kwangil & Kang, Jangkoo & Lee, Soonhee, 2016. "Bullish/bearish/neutral strategies under short sale restrictions," Journal of Banking & Finance, Elsevier, vol. 71(C), pages 227-239.
    13. Patrick Roger & Marie-Hélène Broihanne & Maxime Merli, 2012. "In search of positive skewness: the case of individual investors," Working Papers of LaRGE Research Center 2012-04, Laboratoire de Recherche en Gestion et Economie (LaRGE), Université de Strasbourg.
    14. Andrea Rigamonti, 2020. "Mean-Variance Optimization Is a Good Choice, But for Other Reasons than You Might Think," Risks, MDPI, vol. 8(1), pages 1-16, March.
    15. Viral V. Acharya & Peter DeMarzo & Ilan Kremer, 2011. "Endogenous Information Flows and the Clustering of Announcements," American Economic Review, American Economic Association, vol. 101(7), pages 2955-2979, December.
    16. Ahadzie, Richard Mawulawoe & Jeyasreedharan, Nagaratnam, 2020. "Trading volume and realized higher-order moments in the Australian stock market," Journal of Behavioral and Experimental Finance, Elsevier, vol. 28(C).
    17. Yigit Atilgan & K. Ozgur Demirtas & A. Doruk Gunaydin & Imra Kirli, 2023. "Average skewness in global equity markets," International Review of Finance, International Review of Finance Ltd., vol. 23(2), pages 245-271, June.
    18. Antonio Ciccone & Felix Rusche, 2025. "Reporting Big News, Missing the Big Picture? Stock Market Performance in the Media," CESifo Working Paper Series 11793, CESifo.
    19. Chen Zhao & Jiaqi Sun & Ping He & Shaohua Zhang & Yuqi Ji, 2023. "Integrating Risk Preferences into Game Analysis of Price-Making Retailers in Power Market," Energies, MDPI, vol. 16(8), pages 1-18, April.
    20. Liu, Qingfu & Hua, Renhai & An, Yunbi, 2016. "Determinants and information content of intraday bid-ask spreads: Evidence from Chinese commodity futures markets," Pacific-Basin Finance Journal, Elsevier, vol. 38(C), pages 135-148.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2301.08360. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.