IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v16y2023i15p5840-d1212091.html
   My bibliography  Save this article

Tracking Photovoltaic Power Output Schedule of the Energy Storage System Based on Reinforcement Learning

Author

Listed:
  • Meijun Guo

    (College of Electrical and Power Engineering, Taiyuan University of Technology, Taiyuan 030024, China)

  • Mifeng Ren

    (College of Electrical and Power Engineering, Taiyuan University of Technology, Taiyuan 030024, China)

  • Junghui Chen

    (Department of Chemical Engineering, Chung-Yuan Christian University, Taoyuan 320314, Taiwan)

  • Lan Cheng

    (College of Electrical and Power Engineering, Taiyuan University of Technology, Taiyuan 030024, China)

  • Zhile Yang

    (Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China)

Abstract

The inherent randomness, fluctuation, and intermittence of photovoltaic power generation make it difficult to track the scheduling plan. To improve the ability to track the photovoltaic plan to a greater extent, a real-time charge and discharge power control method based on deep reinforcement learning is proposed. Firstly, the photovoltaic and energy storage hybrid system and the mathematical model of the hybrid system are briefly introduced, and the tracking control problem is defined. Then, power generation plans on different days are clustered into four scenarios by the K-means clustering algorithm. The mean, standard deviation, and kurtosis of the power generation plant are used as the features. Based on the clustered results, the state, action, and reward required for reinforcement learning are set. In the constraint conditions of various variables, to increase the accuracy of the hybrid system for tracking the new generation schedule, the proximal policy optimization (PPO) algorithm is used to optimize the charging/discharging power of the energy storage system (ESS). Finally, the proposed control method is applied to a photovoltaic power station. The results of several valid experiments indicate that the average errors of tracking using the Proportion Integral Differential (PID), model predictive control (MPC) method, and the PPO algorithm in the same condition are 0.374 MW, 0.609 MW, and 0.104 MW, respectively, and the computing time is 1.134 s, 2.760 s, and 0.053 s, respectively. The consequence of these indicates that the proposed deep reinforcement learning-based control strategy is more competitive than the traditional methods in terms of generalization and computation time.

Suggested Citation

  • Meijun Guo & Mifeng Ren & Junghui Chen & Lan Cheng & Zhile Yang, 2023. "Tracking Photovoltaic Power Output Schedule of the Energy Storage System Based on Reinforcement Learning," Energies, MDPI, vol. 16(15), pages 1-15, August.
  • Handle: RePEc:gam:jeners:v:16:y:2023:i:15:p:5840-:d:1212091
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/16/15/5840/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/1996-1073/16/15/5840/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Ding, Huajie & Hu, Zechun & Song, Yonghua, 2012. "Stochastic optimization of the daily operation of wind farm and pumped-hydro-storage plant," Renewable Energy, Elsevier, vol. 48(C), pages 571-578.
    2. Yang, Ting & Zhao, Liyuan & Li, Wei & Zomaya, Albert Y., 2021. "Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning," Energy, Elsevier, vol. 235(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ardizzon, G. & Cavazzini, G. & Pavesi, G., 2014. "A new generation of small hydro and pumped-hydro power plants: Advances and future challenges," Renewable and Sustainable Energy Reviews, Elsevier, vol. 31(C), pages 746-761.
    2. Yıldıran, Uğur & Kayahan, İsmail, 2018. "Risk-averse stochastic model predictive control-based real-time operation method for a wind energy generation system supported by a pumped hydro storage unit," Applied Energy, Elsevier, vol. 226(C), pages 631-643.
    3. Yi, Zonggen & Luo, Yusheng & Westover, Tyler & Katikaneni, Sravya & Ponkiya, Binaka & Sah, Suba & Mahmud, Sadab & Raker, David & Javaid, Ahmad & Heben, Michael J. & Khanna, Raghav, 2022. "Deep reinforcement learning based optimization for a tightly coupled nuclear renewable integrated energy system," Applied Energy, Elsevier, vol. 328(C).
    4. Kenfack, Joseph & Nzotcha, Urbain & Voufo, Joseph & Ngohe-Ekam, Paul Salomon & Nsangou, Jean Calvin & Bignom, Blaise, 2021. "Cameroon's hydropower potential and development under the vision of Central Africa power pool (CAPP): A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 151(C).
    5. Shaohua Hu & Xinlong Zhou & Yi Luo & Guang Zhang, 2019. "Numerical Simulation Three-Dimensional Nonlinear Seepage in a Pumped-Storage Power Station: Case Study," Energies, MDPI, vol. 12(1), pages 1-15, January.
    6. Ming, Zeng & Junjie, Feng & Song, Xue & Zhijie, Wang & Xiaoli, Zhu & Yuejin, Wang, 2013. "Development of China's pumped storage plant and related policy analysis," Energy Policy, Elsevier, vol. 61(C), pages 104-113.
    7. Thomas, Dimitrios & Deblecker, Olivier & Ioakimidis, Christos S., 2016. "Optimal design and techno-economic analysis of an autonomous small isolated microgrid aiming at high RES penetration," Energy, Elsevier, vol. 116(P1), pages 364-379.
    8. Su, Chengguo & Cheng, Chuntian & Wang, Peilin & Shen, Jianjian & Wu, Xinyu, 2019. "Optimization model for long-distance integrated transmission of wind farms and pumped-storage hydropower plants," Applied Energy, Elsevier, vol. 242(C), pages 285-293.
    9. Xiaowei Fan & Yongtao Chen & Ruimiao Wang & Jiaxin Luo & Jingang Wang & Decheng Cao, 2023. "Integrated Energy Microgrid Economic Dispatch Optimization Model Based on Information-Gap Decision Theory," Energies, MDPI, vol. 16(8), pages 1-24, April.
    10. Helge Bormann & Inge Andersen Martinez, 2014. "Towards an Indicator Based Framework Analysing the Suitability of Existing Dams for Energy Storage," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 28(6), pages 1613-1630, April.
    11. Zhu, Ziqing & Hu, Ze & Chan, Ka Wing & Bu, Siqi & Zhou, Bin & Xia, Shiwei, 2023. "Reinforcement learning in deregulated energy market: A comprehensive review," Applied Energy, Elsevier, vol. 329(C).
    12. Cozzolino, R. & Tribioli, L. & Bella, G., 2016. "Power management of a hybrid renewable system for artificial islands: A case study," Energy, Elsevier, vol. 106(C), pages 774-789.
    13. Yuan, Wenlin & Wang, Xinqi & Su, Chengguo & Cheng, Chuntian & Liu, Zhe & Wu, Zening, 2021. "Stochastic optimization model for the short-term joint operation of photovoltaic power and hydropower plants based on chance-constrained programming," Energy, Elsevier, vol. 222(C).
    14. Katsaprakakis, Dimitris Al. & Dakanali, Irini & Condaxakis, Constantinos & Christakis, Dimitris G., 2019. "Comparing electricity storage technologies for small insular grids," Applied Energy, Elsevier, vol. 251(C), pages 1-1.
    15. Ming, Bo & Liu, Pan & Guo, Shenglian & Cheng, Lei & Zhou, Yanlai & Gao, Shida & Li, He, 2018. "Robust hydroelectric unit commitment considering integration of large-scale photovoltaic power: A case study in China," Applied Energy, Elsevier, vol. 228(C), pages 1341-1352.
    16. Bio Gassi, Karim & Baysal, Mustafa, 2023. "Improving real-time energy decision-making model with an actor-critic agent in modern microgrids with energy storage devices," Energy, Elsevier, vol. 263(PE).
    17. Jianzhong Zhou & Yanhe Xu & Yang Zheng & Yuncheng Zhang, 2017. "Optimization of Guide Vane Closing Schemes of Pumped Storage Hydro Unit Using an Enhanced Multi-Objective Gravitational Search Algorithm," Energies, MDPI, vol. 10(7), pages 1-23, July.
    18. Zheng, Lingwei & Wu, Hao & Guo, Siqi & Sun, Xinyu, 2023. "Real-time dispatch of an integrated energy system based on multi-stage reinforcement learning with an improved action-choosing strategy," Energy, Elsevier, vol. 277(C).
    19. Ming, Bo & Liu, Pan & Guo, Shenglian & Zhang, Xiaoqi & Feng, Maoyuan & Wang, Xianxun, 2017. "Optimizing utility-scale photovoltaic power generation for integration into a hydropower reservoir by incorporating long- and short-term operational decisions," Applied Energy, Elsevier, vol. 204(C), pages 432-445.
    20. Guoqiang Sun & Weihang Qian & Wenjin Huang & Zheng Xu & Zhongxing Fu & Zhinong Wei & Sheng Chen, 2019. "Stochastic Adaptive Robust Dispatch for Virtual Power Plants Using the Binding Scenario Identification Approach," Energies, MDPI, vol. 12(10), pages 1-23, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:16:y:2023:i:15:p:5840-:d:1212091. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.