IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v15y2022i14p4960-d857302.html
   My bibliography  Save this article

Bidding a Battery on Electricity Markets and Minimizing Battery Aging Costs: A Reinforcement Learning Approach

Author

Listed:
  • Harri Aaltonen

    (Department of Electrical Engineering and Automation, School of Electrical Engineering, Aalto University, FI-00076 Espoo, Finland)

  • Seppo Sierla

    (Department of Electrical Engineering and Automation, School of Electrical Engineering, Aalto University, FI-00076 Espoo, Finland)

  • Ville Kyrki

    (Department of Electrical Engineering and Automation, School of Electrical Engineering, Aalto University, FI-00076 Espoo, Finland)

  • Mahdi Pourakbari-Kasmaei

    (Department of Electrical Engineering and Automation, School of Electrical Engineering, Aalto University, FI-00076 Espoo, Finland)

  • Valeriy Vyatkin

    (Department of Electrical Engineering and Automation, School of Electrical Engineering, Aalto University, FI-00076 Espoo, Finland
    Department of Computer Science, Electrical and Space Engineering, Luleå University of Technology, 97187 Luleå, Sweden)

Abstract

Battery storage is emerging as a key component of intelligent green electricitiy systems. The battery is monetized through market participation, which usually involves bidding. Bidding is a multi-objective optimization problem, involving targets such as maximizing market compensation and minimizing penalties for failing to provide the service and costs for battery aging. In this article, battery participation is investigated on primary frequency reserve markets. Reinforcement learning is applied for the optimization. In previous research, only simplified formulations of battery aging have been used in the reinforcement learning formulation, so it is unclear how the optimizer would perform with a real battery. In this article, a physics-based battery aging model is used to assess the aging. The contribution of this article is a methodology involving a realistic battery simulation to assess the performance of the trained RL agent with respect to battery aging in order to inform the selection of the weighting of the aging term in the RL reward formula. The RL agent performs day-ahead bidding on the Finnish Frequency Containment Reserves for Normal Operation market, with the objective of maximizing market compensation, minimizing market penalties and minimizing aging costs.

Suggested Citation

  • Harri Aaltonen & Seppo Sierla & Ville Kyrki & Mahdi Pourakbari-Kasmaei & Valeriy Vyatkin, 2022. "Bidding a Battery on Electricity Markets and Minimizing Battery Aging Costs: A Reinforcement Learning Approach," Energies, MDPI, vol. 15(14), pages 1-19, July.
    • Handle: RePEc:gam:jeners:v:15:y:2022:i:14:p:4960-:d:857302
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/15/14/4960/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/15/14/4960/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Niko Karhula & Seppo Sierla & Valeriy Vyatkin, 2021. "Validating the Real-Time Performance of Distributed Energy Resources Participating on Primary Frequency Reserves," Energies, MDPI, vol. 14(21), pages 1-19, October.
    2. Kempitiya, Thimal & Sierla, Seppo & De Silva, Daswin & Yli-Ojanperä, Matti & Alahakoon, Damminda & Vyatkin, Valeriy, 2020. "An Artificial Intelligence framework for bidding optimization with uncertainty in multiple frequency reserve markets," Applied Energy, Elsevier, vol. 280(C).
    3. Beltrán, Sergio & Castro, Alain & Irizar, Ion & Naveran, Gorka & Yeregui, Imanol, 2022. "Framework for collaborative intelligence in forecasting day-ahead electricity price," Applied Energy, Elsevier, vol. 306(PA).
    4. Evgeny Nefedov & Seppo Sierla & Valeriy Vyatkin, 2018. "Internet of Energy Approach for Sustainable Use of Electric Vehicles as Energy Storage of Prosumer Buildings," Energies, MDPI, vol. 11(8), pages 1-18, August.
    5. Hua, Haochen & Qin, Yuchao & Hao, Chuantong & Cao, Junwei, 2019. "Optimal energy management strategies for energy Internet via deep reinforcement learning approach," Applied Energy, Elsevier, vol. 239(C), pages 598-609.
    6. Sachin Kahawala & Daswin De Silva & Seppo Sierla & Damminda Alahakoon & Rashmika Nawaratne & Evgeny Osipov & Andrew Jennings & Valeriy Vyatkin, 2021. "Robust Multi-Step Predictor for Electricity Markets with Real-Time Pricing," Energies, MDPI, vol. 14(14), pages 1-20, July.
    7. Harri Aaltonen & Seppo Sierla & Rakshith Subramanya & Valeriy Vyatkin, 2021. "A Simulation Environment for Training a Reinforcement Learning Agent Trading a Battery Storage," Energies, MDPI, vol. 14(17), pages 1-20, September.
    8. Piotr F. Borowski, 2020. "Zonal and Nodal Models of Energy Market in European Union," Energies, MDPI, vol. 13(16), pages 1-21, August.
    9. Omar, Noshin & Monem, Mohamed Abdel & Firouz, Yousef & Salminen, Justin & Smekens, Jelle & Hegazy, Omar & Gaulous, Hamid & Mulder, Grietus & Van den Bossche, Peter & Coosemans, Thierry & Van Mierlo, J, 2014. "Lithium iron phosphate based battery – Assessment of the aging parameters and development of cycle life model," Applied Energy, Elsevier, vol. 113(C), pages 1575-1585.
    10. Yang, Ting & Zhao, Liyuan & Li, Wei & Zomaya, Albert Y., 2021. "Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning," Energy, Elsevier, vol. 235(C).
    11. Grace Muriithi & Sunetra Chowdhury, 2021. "Optimal Energy Management of a Grid-Tied Solar PV-Battery Microgrid: A Reinforcement Learning Approach," Energies, MDPI, vol. 14(9), pages 1-24, May.
    12. Christian Giovanelli & Seppo Sierla & Ryutaro Ichise & Valeriy Vyatkin, 2018. "Exploiting Artificial Neural Networks for the Prediction of Ancillary Energy Market Prices," Energies, MDPI, vol. 11(7), pages 1-22, July.
    13. Zhou, Jianhao & Xue, Siwu & Xue, Yuan & Liao, Yuhui & Liu, Jun & Zhao, Wanzhong, 2021. "A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning," Energy, Elsevier, vol. 224(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Harri Aaltonen & Seppo Sierla & Rakshith Subramanya & Valeriy Vyatkin, 2021. "A Simulation Environment for Training a Reinforcement Learning Agent Trading a Battery Storage," Energies, MDPI, vol. 14(17), pages 1-20, September.
    2. Rakshith Subramanya & Matti Yli-Ojanperä & Seppo Sierla & Taneli Hölttä & Jori Valtakari & Valeriy Vyatkin, 2021. "A Virtual Power Plant Solution for Aggregating Photovoltaic Systems and Other Distributed Energy Resources for Northern European Primary Frequency Reserves," Energies, MDPI, vol. 14(5), pages 1-23, February.
    3. Fathy, Ahmed, 2023. "Bald eagle search optimizer-based energy management strategy for microgrid with renewable sources and electric vehicles," Applied Energy, Elsevier, vol. 334(C).
    4. Zhu, Ziqing & Hu, Ze & Chan, Ka Wing & Bu, Siqi & Zhou, Bin & Xia, Shiwei, 2023. "Reinforcement learning in deregulated energy market: A comprehensive review," Applied Energy, Elsevier, vol. 329(C).
    5. Gideon Ude Nnachi & Yskandar Hamam & Coneth Graham Richards, 2022. "Appraising the Optimal Power Flow and Generation Capacity in Existing Power Grid Topology with Increase in Energy Demand," Energies, MDPI, vol. 15(7), pages 1-23, March.
    6. Yin, Linfei & Qiu, Yao, 2022. "Neural network dynamic differential control for long-term price guidance mechanism of flexible energy service providers," Energy, Elsevier, vol. 255(C).
    7. Xue, Lin & Wang, Jianxue & Zhang, Yao & Yong, Weizhen & Qi, Jie & Li, Haotian, 2023. "Model-data-event based community integrated energy system low-carbon economic scheduling," Renewable and Sustainable Energy Reviews, Elsevier, vol. 182(C).
    8. Zhang, Bin & Wu, Xuewei & Ghias, Amer M.Y.M. & Chen, Zhe, 2023. "Coordinated carbon capture systems and power-to-gas dynamic economic energy dispatch strategy for electricity–gas coupled systems considering system uncertainty: An improved soft actor–critic approach," Energy, Elsevier, vol. 271(C).
    9. Zhou, Yanting & Ma, Zhongjing & Zhang, Jinhui & Zou, Suli, 2022. "Data-driven stochastic energy management of multi energy system using deep reinforcement learning," Energy, Elsevier, vol. 261(PA).
    10. Niko Karhula & Seppo Sierla & Valeriy Vyatkin, 2021. "Validating the Real-Time Performance of Distributed Energy Resources Participating on Primary Frequency Reserves," Energies, MDPI, vol. 14(21), pages 1-19, October.
    11. Ester Vasta & Tommaso Scimone & Giovanni Nobile & Otto Eberhardt & Daniele Dugo & Massimiliano Maurizio De Benedetti & Luigi Lanuzza & Giuseppe Scarcella & Luca Patanè & Paolo Arena & Mario Cacciato, 2023. "Models for Battery Health Assessment: A Comparative Evaluation," Energies, MDPI, vol. 16(2), pages 1-34, January.
    12. Zhu, Jiaoyiling & Hu, Weihao & Xu, Xiao & Liu, Haoming & Pan, Li & Fan, Haoyang & Zhang, Zhenyuan & Chen, Zhe, 2022. "Optimal scheduling of a wind energy dominated distribution network via a deep reinforcement learning approach," Renewable Energy, Elsevier, vol. 201(P1), pages 792-801.
    13. Zhang, Yijie & Ma, Tao & Elia Campana, Pietro & Yamaguchi, Yohei & Dai, Yanjun, 2020. "A techno-economic sizing method for grid-connected household photovoltaic battery systems," Applied Energy, Elsevier, vol. 269(C).
    14. Ma, Jian & Xu, Shu & Shang, Pengchao & ding, Yu & Qin, Weili & Cheng, Yujie & Lu, Chen & Su, Yuzhuan & Chong, Jin & Jin, Haizu & Lin, Yongshou, 2020. "Cycle life test optimization for different Li-ion power battery formulations using a hybrid remaining-useful-life prediction method," Applied Energy, Elsevier, vol. 262(C).
    15. Zeyue Sun & Mohsen Eskandari & Chaoran Zheng & Ming Li, 2022. "Handling Computation Hardness and Time Complexity Issue of Battery Energy Storage Scheduling in Microgrids by Deep Reinforcement Learning," Energies, MDPI, vol. 16(1), pages 1-20, December.
    16. Danijel Pavković & Mihael Cipek & Zdenko Kljaić & Tomislav Josip Mlinarić & Mario Hrgetić & Davor Zorc, 2018. "Damping Optimum-Based Design of Control Strategy Suitable for Battery/Ultracapacitor Electric Vehicles," Energies, MDPI, vol. 11(10), pages 1-26, October.
    17. Shovon Goutam & Jean-Marc Timmermans & Noshin Omar & Peter Van den Bossche & Joeri Van Mierlo, 2015. "Comparative Study of Surface Temperature Behavior of Commercial Li-Ion Pouch Cells of Different Chemistries and Capacities by Infrared Thermography," Energies, MDPI, vol. 8(8), pages 1-18, August.
    18. Emil Kraft & Dogan Keles & Wolf Fichtner, 2020. "Modeling of frequency containment reserve prices with econometrics and artificial intelligence," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 39(8), pages 1179-1197, December.
    19. Jiaming Zhou & Chunxiao Feng & Qingqing Su & Shangfeng Jiang & Zhixian Fan & Jiageng Ruan & Shikai Sun & Leli Hu, 2022. "The Multi-Objective Optimization of Powertrain Design and Energy Management Strategy for Fuel Cell–Battery Electric Vehicle," Sustainability, MDPI, vol. 14(10), pages 1-19, May.
    20. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:15:y:2022:i:14:p:4960-:d:857302. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.