IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v16y2023i15p5840-d1212091.html
   My bibliography  Save this article

Tracking Photovoltaic Power Output Schedule of the Energy Storage System Based on Reinforcement Learning

Author

Listed:
  • Meijun Guo

    (College of Electrical and Power Engineering, Taiyuan University of Technology, Taiyuan 030024, China)

  • Mifeng Ren

    (College of Electrical and Power Engineering, Taiyuan University of Technology, Taiyuan 030024, China)

  • Junghui Chen

    (Department of Chemical Engineering, Chung-Yuan Christian University, Taoyuan 320314, Taiwan)

  • Lan Cheng

    (College of Electrical and Power Engineering, Taiyuan University of Technology, Taiyuan 030024, China)

  • Zhile Yang

    (Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China)

Abstract

The inherent randomness, fluctuation, and intermittence of photovoltaic power generation make it difficult to track the scheduling plan. To improve the ability to track the photovoltaic plan to a greater extent, a real-time charge and discharge power control method based on deep reinforcement learning is proposed. Firstly, the photovoltaic and energy storage hybrid system and the mathematical model of the hybrid system are briefly introduced, and the tracking control problem is defined. Then, power generation plans on different days are clustered into four scenarios by the K-means clustering algorithm. The mean, standard deviation, and kurtosis of the power generation plant are used as the features. Based on the clustered results, the state, action, and reward required for reinforcement learning are set. In the constraint conditions of various variables, to increase the accuracy of the hybrid system for tracking the new generation schedule, the proximal policy optimization (PPO) algorithm is used to optimize the charging/discharging power of the energy storage system (ESS). Finally, the proposed control method is applied to a photovoltaic power station. The results of several valid experiments indicate that the average errors of tracking using the Proportion Integral Differential (PID), model predictive control (MPC) method, and the PPO algorithm in the same condition are 0.374 MW, 0.609 MW, and 0.104 MW, respectively, and the computing time is 1.134 s, 2.760 s, and 0.053 s, respectively. The consequence of these indicates that the proposed deep reinforcement learning-based control strategy is more competitive than the traditional methods in terms of generalization and computation time.

Suggested Citation

  • Meijun Guo & Mifeng Ren & Junghui Chen & Lan Cheng & Zhile Yang, 2023. "Tracking Photovoltaic Power Output Schedule of the Energy Storage System Based on Reinforcement Learning," Energies, MDPI, vol. 16(15), pages 1-15, August.
  • Handle: RePEc:gam:jeners:v:16:y:2023:i:15:p:5840-:d:1212091
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/16/15/5840/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/1996-1073/16/15/5840/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Ding, Huajie & Hu, Zechun & Song, Yonghua, 2012. "Stochastic optimization of the daily operation of wind farm and pumped-hydro-storage plant," Renewable Energy, Elsevier, vol. 48(C), pages 571-578.
    2. Yang, Ting & Zhao, Liyuan & Li, Wei & Zomaya, Albert Y., 2021. "Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning," Energy, Elsevier, vol. 235(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Su, Chengguo & Cheng, Chuntian & Wang, Peilin & Shen, Jianjian & Wu, Xinyu, 2019. "Optimization model for long-distance integrated transmission of wind farms and pumped-storage hydropower plants," Applied Energy, Elsevier, vol. 242(C), pages 285-293.
    2. Helge Bormann & Inge Andersen Martinez, 2014. "Towards an Indicator Based Framework Analysing the Suitability of Existing Dams for Energy Storage," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 28(6), pages 1613-1630, April.
    3. Zhu, Ziqing & Hu, Ze & Chan, Ka Wing & Bu, Siqi & Zhou, Bin & Xia, Shiwei, 2023. "Reinforcement learning in deregulated energy market: A comprehensive review," Applied Energy, Elsevier, vol. 329(C).
    4. Ming, Bo & Liu, Pan & Guo, Shenglian & Cheng, Lei & Zhou, Yanlai & Gao, Shida & Li, He, 2018. "Robust hydroelectric unit commitment considering integration of large-scale photovoltaic power: A case study in China," Applied Energy, Elsevier, vol. 228(C), pages 1341-1352.
    5. Bio Gassi, Karim & Baysal, Mustafa, 2023. "Improving real-time energy decision-making model with an actor-critic agent in modern microgrids with energy storage devices," Energy, Elsevier, vol. 263(PE).
    6. Zheng, Lingwei & Wu, Hao & Guo, Siqi & Sun, Xinyu, 2023. "Real-time dispatch of an integrated energy system based on multi-stage reinforcement learning with an improved action-choosing strategy," Energy, Elsevier, vol. 277(C).
    7. Ming, Bo & Liu, Pan & Guo, Shenglian & Zhang, Xiaoqi & Feng, Maoyuan & Wang, Xianxun, 2017. "Optimizing utility-scale photovoltaic power generation for integration into a hydropower reservoir by incorporating long- and short-term operational decisions," Applied Energy, Elsevier, vol. 204(C), pages 432-445.
    8. Guoqiang Sun & Weihang Qian & Wenjin Huang & Zheng Xu & Zhongxing Fu & Zhinong Wei & Sheng Chen, 2019. "Stochastic Adaptive Robust Dispatch for Virtual Power Plants Using the Binding Scenario Identification Approach," Energies, MDPI, vol. 12(10), pages 1-23, May.
    9. Dhillon, Javed & Kumar, Arun & Singal, S.K., 2014. "Optimization methods applied for Wind–PSP operation and scheduling under deregulated market: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 30(C), pages 682-700.
    10. Fan, Wei & Ju, Liwei & Tan, Zhongfu & Li, Xiangguang & Zhang, Amin & Li, Xudong & Wang, Yueping, 2023. "Two-stage distributionally robust optimization model of integrated energy system group considering energy sharing and carbon transfer," Applied Energy, Elsevier, vol. 331(C).
    11. Milad Ghaisi & Milad Rahmani & Pedram Gharghabi & Ali Zoghi & Seyed Hossein Hosseinian, 2017. "Scheduling a Wind Hydro-Pumped-Storage Unit Considering the Economical Optimization," Post-Print hal-01478231, HAL.
    12. Jin, Xiaoyu & Liu, Benxi & Liao, Shengli & Cheng, Chuntian & Yan, Zhiyu, 2022. "A Wasserstein metric-based distributionally robust optimization approach for reliable-economic equilibrium operation of hydro-wind-solar energy systems," Renewable Energy, Elsevier, vol. 196(C), pages 204-219.
    13. Osmani, Atif & Zhang, Jun, 2014. "Optimal grid design and logistic planning for wind and biomass based renewable electricity supply chains under uncertainties," Energy, Elsevier, vol. 70(C), pages 514-528.
    14. Harri Aaltonen & Seppo Sierla & Ville Kyrki & Mahdi Pourakbari-Kasmaei & Valeriy Vyatkin, 2022. "Bidding a Battery on Electricity Markets and Minimizing Battery Aging Costs: A Reinforcement Learning Approach," Energies, MDPI, vol. 15(14), pages 1-19, July.
    15. Gideon Ude Nnachi & Yskandar Hamam & Coneth Graham Richards, 2022. "Appraising the Optimal Power Flow and Generation Capacity in Existing Power Grid Topology with Increase in Energy Demand," Energies, MDPI, vol. 15(7), pages 1-23, March.
    16. Jin, Xiaoyu & Liu, Benxi & Liao, Shengli & Cheng, Chuntian & Li, Gang & Liu, Lingjun, 2022. "Impacts of different wind and solar power penetrations on cascade hydroplants operation," Renewable Energy, Elsevier, vol. 182(C), pages 227-244.
    17. Zhou, Siyu & Han, Yang & Zalhaf, Amr S. & Chen, Shuheng & Zhou, Te & Yang, Ping & Elboshy, Bahaa, 2023. "A novel multi-objective scheduling model for grid-connected hydro-wind-PV-battery complementary system under extreme weather: A case study of Sichuan, China," Renewable Energy, Elsevier, vol. 212(C), pages 818-833.
    18. Papaefthymiou, Stefanos V. & Papathanassiou, Stavros A., 2014. "Optimum sizing of wind-pumped-storage hybrid power stations in island systems," Renewable Energy, Elsevier, vol. 64(C), pages 187-196.
    19. Guo, Chenyu & Wang, Xin & Zheng, Yihui & Zhang, Feng, 2022. "Real-time optimal energy management of microgrid with uncertainties based on deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    20. Rehman, Shafiqur & Al-Hadhrami, Luai M. & Alam, Md. Mahbub, 2015. "Pumped hydro energy storage system: A technological review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 44(C), pages 586-598.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:16:y:2023:i:15:p:5840-:d:1212091. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.