IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v388y2025ics0306261925002910.html
   My bibliography  Save this article

Deep reinforcement learning-based strategy for maximizing returns from renewable energy and energy storage systems in multi-electricity markets

Author

Listed:
  • Cardo-Miota, Javier
  • Beltran, Hector
  • Pérez, Emilio
  • Khadem, Shafi
  • Bahloul, Mohamed

Abstract

The integration of Renewable Energy Sources (RES) with Energy Storage Systems (ESS) presents challenges and opportunities in optimizing their participation in electricity markets. This study introduces a novel approach that leverages Deep Reinforcement Learning (RL) algorithms to develop optimal bidding strategies for collocated RES with Battery ESS (BESS) configurations, enabling multi-market participation in both energy and ancillary services (AS) markets. The proposed method uses a Markov Decision Process (MDP) framework to manage BESS utilization dynamically, considering market conditions and technical constraints. As an RL agent, the actor–critic approach known as the Twin Delayed Deep Deterministic (TD3) Policy Gradient algorithm is implemented. A data-driven training process facilitates model learning while minimizing the required training dataset and time. Focused on the Irish context, the case study involves participation in both the day-ahead energy market and reserve services for frequency droop curve response of the DS3 Programme. Historical data from a 7 MW solar PV plant and a 1 MWh BESS are utilized to evaluate the performance. The RL agent dynamically adapts to market dynamics and system constraints, achieving substantial economic benefits compared to benchmark strategies, with an additional 8271€, 166,738€, and 11,369€, respectively.

Suggested Citation

  • Cardo-Miota, Javier & Beltran, Hector & Pérez, Emilio & Khadem, Shafi & Bahloul, Mohamed, 2025. "Deep reinforcement learning-based strategy for maximizing returns from renewable energy and energy storage systems in multi-electricity markets," Applied Energy, Elsevier, vol. 388(C).
    • Handle: RePEc:eee:appene:v:388:y:2025:i:c:s0306261925002910
    DOI: 10.1016/j.apenergy.2025.125561
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261925002910
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2025.125561?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Konstantinos Oureilidis & Kyriaki-Nefeli Malamaki & Konstantinos Gallos & Achilleas Tsitsimelis & Christos Dikaiakos & Spyros Gkavanoudis & Milos Cvetkovic & Juan Manuel Mauricio & Jose Maria Maza Ort, 2020. "Ancillary Services Market Design in Distribution Networks: Review and Identification of Barriers," Energies, MDPI, vol. 13(4), pages 1-44, February.
    2. Cardo-Miota, Javier & Trivedi, Rohit & Patra, Sandipan & Khadem, Shafi & Bahloul, Mohamed, 2024. "Data-driven approach for day-ahead System Non-Synchronous Penetration forecasting: A comprehensive framework, model development and analysis," Applied Energy, Elsevier, vol. 362(C).
    3. Groppi, Daniele & Pfeifer, Antun & Garcia, Davide Astiaso & Krajačić, Goran & Duić, Neven, 2021. "A review on energy storage and demand side management solutions in smart energy islands," Renewable and Sustainable Energy Reviews, Elsevier, vol. 135(C).
    4. Johnson, Samuel C. & Rhodes, Joshua D. & Webber, Michael E., 2020. "Understanding the impact of non-synchronous wind and solar generation on grid stability and identifying mitigation pathways," Applied Energy, Elsevier, vol. 262(C).
    5. Liu, Xuezhi & Yan, Zheng & Wu, Jianzhong, 2019. "Optimal coordinated operation of a multi-energy community considering interactions between energy storage and conversion devices," Applied Energy, Elsevier, vol. 248(C), pages 256-273.
    6. Bahloul, Mohamed & Daoud, Mohamed & Khadem, Shafi K., 2024. "Optimal dispatch of battery energy storage for multi-service provision in a collocated PV power plant considering battery ageing," Energy, Elsevier, vol. 293(C).
    7. Bahloul, Mohamed & Daoud, Mohamed & Khadem, Shafiuzzaman K., 2022. "A bottom-up approach for techno-economic analysis of battery energy storage system for Irish grid DS3 service provision," Energy, Elsevier, vol. 245(C).
    8. Fernández-Muñoz, Daniel & Pérez-Díaz, Juan I. & Guisández, Ignacio & Chazarra, Manuel & Fernández-Espina, Álvaro, 2020. "Fast frequency control ancillary services: An international review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 120(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rancilio, G. & Rossi, A. & Falabretti, D. & Galliani, A. & Merlo, M., 2022. "Ancillary services markets in europe: Evolution and regulatory trade-offs," Renewable and Sustainable Energy Reviews, Elsevier, vol. 154(C).
    2. Cardo-Miota, Javier & Trivedi, Rohit & Patra, Sandipan & Khadem, Shafi & Bahloul, Mohamed, 2024. "Data-driven approach for day-ahead System Non-Synchronous Penetration forecasting: A comprehensive framework, model development and analysis," Applied Energy, Elsevier, vol. 362(C).
    3. Evyatar Littwitz & Ofira Ayalon, 2025. "Feasibility Analysis of Storage and Renewable Energy Ancillary Services for Grid Operations," Energies, MDPI, vol. 18(11), pages 1-35, May.
    4. Pablo Carrasco Ortega & Pablo Durán Gómez & Julio César Mérida Sánchez & Fernando Echevarría Camarero & Ángel Á. Pardiñas, 2023. "Battery Energy Storage Systems for the New Electricity Market Landscape: Modeling, State Diagnostics, Management, and Viability—A Review," Energies, MDPI, vol. 16(17), pages 1-51, August.
    5. Sevdari, Kristian & Calearo, Lisa & Andersen, Peter Bach & Marinelli, Mattia, 2022. "Ancillary services and electric vehicles: An overview from charging clusters and chargers technology perspectives," Renewable and Sustainable Energy Reviews, Elsevier, vol. 167(C).
    6. Gržanić, M. & Capuder, T. & Zhang, N. & Huang, W., 2022. "Prosumers as active market participants: A systematic review of evolution of opportunities, models and challenges," Renewable and Sustainable Energy Reviews, Elsevier, vol. 154(C).
    7. Østergaard, P.A. & Lund, H. & Thellufsen, J.Z. & Sorknæs, P. & Mathiesen, B.V., 2022. "Review and validation of EnergyPLAN," Renewable and Sustainable Energy Reviews, Elsevier, vol. 168(C).
    8. Theofilos A. Papadopoulos & Kalliopi D. Pippi & Georgios A. Barzegkar-Ntovom & Eleftherios O. Kontis & Angelos I. Nousdilis & Christos L. Athanasiadis & Georgios C. Kryonidis, 2023. "Validation of a Holistic System for Operational Analysis and Provision of Ancillary Services in Active Distribution Networks," Energies, MDPI, vol. 16(6), pages 1-27, March.
    9. Clarke, Fiona & Dorneanu, Bogdan & Mechleri, Evgenia & Arellano-Garcia, Harvey, 2021. "Optimal design of heating and cooling pipeline networks for residential distributed energy resource systems," Energy, Elsevier, vol. 235(C).
    10. Cheng, Yi & Azizipanah-Abarghooee, Rasoul & Azizi, Sadegh & Ding, Lei & Terzija, Vladimir, 2020. "Smart frequency control in low inertia energy systems based on frequency response techniques: A review," Applied Energy, Elsevier, vol. 279(C).
    11. Oleksandr Miroshnyk & Oleksandr Moroz & Taras Shchur & Andrii Chepizhnyi & Mohamed Qawaqzeh & Sławomir Kocira, 2023. "Investigation of Smart Grid Operation Modes with Electrical Energy Storage System," Energies, MDPI, vol. 16(6), pages 1-13, March.
    12. Kalyani Makarand Kurundkar & Geetanjali Abhijit Vaidya, 2023. "Stochastic Security-Constrained Economic Dispatch of Load-Following and Contingency Reserves Ancillary Service Using a Grid-Connected Microgrid during Uncertainty," Energies, MDPI, vol. 16(6), pages 1-25, March.
    13. Lund, H. & Østergaard, P.A. & Yuan, M. & Sorknæs, P. & Thellufsen, J.Z., 2025. "Energy balancing and storage in climate-neutral smart energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 209(C).
    14. Pacheco, A. & Monteiro, J. & Santos, J. & Sequeira, C. & Nunes, J., 2022. "Energy transition process and community engagement on geographic islands: The case of Culatra Island (Ria Formosa, Portugal)," Renewable Energy, Elsevier, vol. 184(C), pages 700-711.
    15. Psarros, Georgios N. & Papathanassiou, Stavros A., 2023. "Generation scheduling in island systems with variable renewable energy sources: A literature review," Renewable Energy, Elsevier, vol. 205(C), pages 1105-1124.
    16. Garcet, J. & De Meulenaere, R. & Blondeau, J., 2022. "Enabling flexible CHP operation for grid support by exploiting the DHN thermal inertia," Applied Energy, Elsevier, vol. 316(C).
    17. Herc, Luka & Pfeifer, Antun & Duić, Neven & Wang, Fei, 2022. "Economic viability of flexibility options for smart energy systems with high penetration of renewable energy," Energy, Elsevier, vol. 252(C).
    18. Liu, Hong & Zhao, Yue & Gu, Chenghong & Ge, Shaoyun & Yang, Zan, 2021. "Adjustable capability of the distributed energy system: Definition, framework, and evaluation model," Energy, Elsevier, vol. 222(C).
    19. Park, Joungho & Kang, Sungho & Kim, Sunwoo & Kim, Hana & Kim, Sang-Kyung & Lee, Jay H., 2024. "Optimizing green hydrogen systems: Balancing economic viability and reliability in the face of supply-demand volatility," Applied Energy, Elsevier, vol. 368(C).
    20. Hu, Qian & Zhu, Ziqing & Bu, Siqi & Wing Chan, Ka & Li, Fangxing, 2021. "A multi-market nanogrid P2P energy and ancillary service trading paradigm: Mechanisms and implementations," Applied Energy, Elsevier, vol. 293(C).

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:388:y:2025:i:c:s0306261925002910. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.