IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v14y2021i24p8365-d700404.html
   My bibliography  Save this article

Co-Optimizing Battery Storage for Energy Arbitrage and Frequency Regulation in Real-Time Markets Using Deep Reinforcement Learning

Author

Listed:
  • Yushen Miao

    (James Watt School of Engineering, University of Glasgow, Glasgow G12 8QQ, UK)

  • Tianyi Chen

    (James Watt School of Engineering, University of Glasgow, Glasgow G12 8QQ, UK)

  • Shengrong Bu

    (Department of Engineering, Brock University, St. Catharines, ON L2S 3A1, Canada)

  • Hao Liang

    (Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T6G 2R3, Canada)

  • Zhu Han

    (Department of Electrical and Computer Engineering, University of Houston, Houston, TX 77204, USA)

Abstract

Battery energy storage systems (BESSs) play a critical role in eliminating uncertainties associated with renewable energy generation, to maintain stability and improve flexibility of power networks. In this paper, a BESS is used to provide energy arbitrage (EA) and frequency regulation (FR) services simultaneously to maximize its total revenue within the physical constraints. The EA and FR actions are taken at different timescales. The multitimescale problem is formulated as two nested Markov decision process (MDP) submodels. The problem is a complex decision-making problem with enormous high-dimensional data and uncertainty (e.g., the price of the electricity). Therefore, a novel co-optimization scheme is proposed to handle the multitimescale problem, and also coordinate EA and FR services. A triplet deep deterministic policy gradient with exploration noise decay (TDD–ND) approach is used to obtain the optimal policy at each timescale. Simulations are conducted with real-time electricity prices and regulation signals data from the American PJM regulation market. The simulation results show that the proposed approach performs better than other studied policies in literature.

Suggested Citation

  • Yushen Miao & Tianyi Chen & Shengrong Bu & Hao Liang & Zhu Han, 2021. "Co-Optimizing Battery Storage for Energy Arbitrage and Frequency Regulation in Real-Time Markets Using Deep Reinforcement Learning," Energies, MDPI, vol. 14(24), pages 1-17, December.
    • Handle: RePEc:gam:jeners:v:14:y:2021:i:24:p:8365-:d:700404
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/14/24/8365/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/14/24/8365/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Walawalkar, Rahul & Apt, Jay & Mancini, Rick, 2007. "Economics of electric energy storage for energy arbitrage and regulation in New York," Energy Policy, Elsevier, vol. 35(4), pages 2558-2568, April.
    2. Sioshansi, Ramteen & Denholm, Paul & Jenkin, Thomas & Weiss, Jurgen, 2009. "Estimating the value of electricity storage in PJM: Arbitrage and some welfare effects," Energy Economics, Elsevier, vol. 31(2), pages 269-277, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Alba Leduchowicz-Municio & Miguel Edgar Morales Udaeta & André Luiz Veiga Gimenes & Tuo Ji & Victor Baiochi Riboldi, 2022. "Socio-Environmental Evaluation of MV Commercial Time-Shift Application Based on Battery Energy Storage Systems," Energies, MDPI, vol. 15(14), pages 1-21, July.
    2. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. McConnell, Dylan & Forcey, Tim & Sandiford, Mike, 2015. "Estimating the value of electricity storage in an energy-only wholesale market," Applied Energy, Elsevier, vol. 159(C), pages 422-432.
    2. Sioshansi, Ramteen & Denholm, Paul & Jenkin, Thomas, 2011. "A comparative analysis of the value of pure and hybrid electricity storage," Energy Economics, Elsevier, vol. 33(1), pages 56-66, January.
    3. Telaretti, E. & Dusonchet, L., 2017. "Stationary battery technologies in the U.S.: Development Trends and prospects," Renewable and Sustainable Energy Reviews, Elsevier, vol. 75(C), pages 380-392.
    4. Telaretti, E. & Graditi, G. & Ippolito, M.G. & Zizzo, G., 2016. "Economic feasibility of stationary electrochemical storages for electric bill management applications: The Italian scenario," Energy Policy, Elsevier, vol. 94(C), pages 126-137.
    5. Dufo-López, Rodolfo, 2015. "Optimisation of size and control of grid-connected storage under real time electricity pricing conditions," Applied Energy, Elsevier, vol. 140(C), pages 395-408.
    6. Núñez, Fernando & Canca, David & Arcos-Vargas, Ángel, 2022. "An assessment of European electricity arbitrage using storage systems," Energy, Elsevier, vol. 242(C).
    7. Carson, Richard T. & Novan, Kevin, 2013. "The private and social economics of bulk electricity storage," Journal of Environmental Economics and Management, Elsevier, vol. 66(3), pages 404-423.
    8. Daniel R. Jiang & Warren B. Powell, 2015. "Optimal Hour-Ahead Bidding in the Real-Time Electricity Market with Battery Storage Using Approximate Dynamic Programming," INFORMS Journal on Computing, INFORMS, vol. 27(3), pages 525-543, August.
    9. Shafiee, Soroush & Zamani-Dehkordi, Payam & Zareipour, Hamidreza & Knight, Andrew M., 2016. "Economic assessment of a price-maker energy storage facility in the Alberta electricity market," Energy, Elsevier, vol. 111(C), pages 537-547.
    10. Ramteen Sioshansi & Paul Denholm & Thomas Jenkin, 2012. "Market and Policy Barriers to Deployment of Energy Storage," Economics of Energy & Environmental Policy, International Association for Energy Economics, vol. 0(Number 2).
    11. Shcherbakova, Anastasia & Kleit, Andrew & Cho, Joohyun, 2014. "The value of energy storage in South Korea’s electricity market: A Hotelling approach," Applied Energy, Elsevier, vol. 125(C), pages 93-102.
    12. Antweiler, Werner, 2021. "Microeconomic models of electricity storage: Price Forecasting, arbitrage limits, curtailment insurance, and transmission line utilization," Energy Economics, Elsevier, vol. 101(C).
    13. Biggins, F.A.V. & Travers, D. & Ejeh, J.O. & Lee, R. & Buckley, A. & Brown, S., 2023. "The economic impact of location on a solar farm co-located with energy storage," Energy, Elsevier, vol. 278(C).
    14. McPherson, Madeleine & Tahseen, Samiha, 2018. "Deploying storage assets to facilitate variable renewable energy integration: The impacts of grid flexibility, renewable penetration, and market structure," Energy, Elsevier, vol. 145(C), pages 856-870.
    15. Lin, Boqiang & Wu, Wei, 2017. "Economic viability of battery energy storage and grid strategy: A special case of China electricity market," Energy, Elsevier, vol. 124(C), pages 423-434.
    16. Denholm, Paul & Sioshansi, Ramteen, 2009. "The value of compressed air energy storage with wind in transmission-constrained electric power systems," Energy Policy, Elsevier, vol. 37(8), pages 3149-3158, August.
    17. Zafirakis, Dimitrios & Chalvatzis, Konstantinos J. & Baiocchi, Giovanni & Daskalakis, Georgios, 2016. "The value of arbitrage for energy storage: Evidence from European electricity markets," Applied Energy, Elsevier, vol. 184(C), pages 971-986.
    18. Linn, Joshua & Shih, Jhih-Shyang, 2019. "Do lower electricity storage costs reduce greenhouse gas emissions?," Journal of Environmental Economics and Management, Elsevier, vol. 96(C), pages 130-158.
    19. Dowling, Alexander W. & Kumar, Ranjeet & Zavala, Victor M., 2017. "A multi-scale optimization framework for electricity market participation," Applied Energy, Elsevier, vol. 190(C), pages 147-164.
    20. Jeon, Wooyoung & Mo, Jung Youn, 2018. "The true economic value of supply-side energy storage in the smart grid environment – The case of Korea," Energy Policy, Elsevier, vol. 121(C), pages 101-111.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:14:y:2021:i:24:p:8365-:d:700404. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.