IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v239y2022ipas0360544221021745.html
   My bibliography  Save this article

Bi-level stochastic real-time pricing model in multi-energy generation system: A reinforcement learning approach

Author

Listed:
  • Zhang, Li
  • Gao, Yan
  • Zhu, Hongbo
  • Tao, Li

Abstract

With the penetration of intermittent renewable energy sources, greater uncertainty has been brought to the power generation system, creating increased challenges to real-time pricing (RTP). Different from the existing studies, this paper aims to design an RTP strategy for the smart grid which integrates multi-energy generation on the supply side. Without loss of generality, small-scale distributed energy generation and power storage devices for users are also considered. Taking the interests of both supply and demand sides into consideration, a bilevel stochastic model for real-time demand response in the framework of Markov decision process (MDP) is formulated. The model well captures the interactive characters of both sides. Regarding the difficulty of collecting exact information from users in a centralized way in practice, a novel distributed online multi-agent reinforcement learning algorithm is proposed to solve the MDP model without acquisition of the transition probabilities. Through the information interaction between the upper and lower levels, the real-time electricity prices are decided adaptively, meanwhile, the optimal strategy of power supply and consumption is obtained. Simulation results demonstrate that the proposed pricing method and algorithm have a good performance in cutting peak and filling the valley and guarantee the benefits of both supply and demand.

Suggested Citation

  • Zhang, Li & Gao, Yan & Zhu, Hongbo & Tao, Li, 2022. "Bi-level stochastic real-time pricing model in multi-energy generation system: A reinforcement learning approach," Energy, Elsevier, vol. 239(PA).
    • Handle: RePEc:eee:energy:v:239:y:2022:i:pa:s0360544221021745
    DOI: 10.1016/j.energy.2021.121926
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544221021745
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2021.121926?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Zareen, N. & Mustafa, M.W. & Sultana, U. & Nadia, R. & Khattak, M.A., 2015. "Optimal real time cost-benefit based demand response with intermittent resources," Energy, Elsevier, vol. 90(P2), pages 1695-1706.
    2. Chang, Zihan & Zhang, Yang & Chen, Wenbo, 2019. "Electricity price prediction based on hybrid model of adam optimized LSTM neural network and wavelet transform," Energy, Elsevier, vol. 187(C).
    3. Wang, Fei & Ge, Xinxin & Yang, Peng & Li, Kangping & Mi, Zengqiang & Siano, Pierluigi & Duić, Neven, 2020. "Day-ahead optimal bidding and scheduling strategies for DER aggregator considering responsive uncertainty under real-time pricing," Energy, Elsevier, vol. 213(C).
    4. Hossain, Md Alamgir & Pota, Hemanshu Roy & Squartini, Stefano & Abdou, Ahmed Fathi, 2019. "Modified PSO algorithm for real-time energy management in grid-connected microgrids," Renewable Energy, Elsevier, vol. 136(C), pages 746-757.
    5. Zhao, Xueyuan & Gao, Weijun & Qian, Fanyue & Ge, Jian, 2021. "Electricity cost comparison of dynamic pricing model based on load forecasting in home energy management system," Energy, Elsevier, vol. 229(C).
    6. Favuzza, S. & Galioto, G. & Ippolito, M.G. & Massaro, F. & Milazzo, F. & Pecoraro, G. & Riva Sanseverino, E. & Telaretti, E., 2015. "Real-time pricing for aggregates energy resources in the Italian energy market," Energy, Elsevier, vol. 87(C), pages 251-258.
    7. Javadi, Mohammad Sadegh & Gough, Matthew & Lotfi, Mohamed & Esmaeel Nezhad, Ali & Santos, Sérgio F. & Catalão, João P.S., 2020. "Optimal self-scheduling of home energy management system in the presence of photovoltaic power generation and batteries," Energy, Elsevier, vol. 210(C).
    8. Li, Yuanyuan & Li, Junxiang & He, Jianjia & Zhang, Shuyuan, 2021. "The real-time pricing optimization model of smart grid based on the utility function of the logistic function," Energy, Elsevier, vol. 224(C).
    9. Dehnavi, Ehsan & Abdi, Hamdi, 2016. "Optimal pricing in time of use demand response by integrating with dynamic economic dispatch problem," Energy, Elsevier, vol. 109(C), pages 1086-1094.
    10. Kaygusuz, Asim, 2019. "Closed loop elastic demand control by dynamic energy pricing in smart grids," Energy, Elsevier, vol. 176(C), pages 596-603.
    11. Motalleb, Mahdi & Annaswamy, Anuradha & Ghorbani, Reza, 2018. "A real-time demand response market through a repeated incomplete-information game," Energy, Elsevier, vol. 143(C), pages 424-438.
    12. Scott Fay & Jinhong Xie, 2010. "The Economics of Buyer Uncertainty: Advance Selling vs. Probabilistic Selling," Marketing Science, INFORMS, vol. 29(6), pages 1040-1057, 11-12.
    13. Lu, Renzhi & Hong, Seung Ho & Zhang, Xiongfeng, 2018. "A Dynamic pricing demand response algorithm for smart grid: Reinforcement learning approach," Applied Energy, Elsevier, vol. 220(C), pages 220-230.
    14. Srinivasan, Dipti & Rajgarhia, Sanjana & Radhakrishnan, Bharat Menon & Sharma, Anurag & Khincha, H.P., 2017. "Game-Theory based dynamic pricing strategies for demand side management in smart grids," Energy, Elsevier, vol. 126(C), pages 132-143.
    15. Chen, Zheng & Hu, Hengjie & Wu, Yitao & Zhang, Yuanjian & Li, Guang & Liu, Yonggang, 2020. "Stochastic model predictive control for energy management of power-split plug-in hybrid electric vehicles based on reinforcement learning," Energy, Elsevier, vol. 211(C).
    16. Yang, Ningkang & Han, Lijin & Xiang, Changle & Liu, Hui & Li, Xunmin, 2021. "An indirect reinforcement learning based real-time energy management strategy via high-order Markov Chain model for a hybrid electric vehicle," Energy, Elsevier, vol. 236(C).
    17. Nyong-Bassey, Bassey Etim & Giaouris, Damian & Patsios, Charalampos & Papadopoulou, Simira & Papadopoulos, Athanasios I. & Walker, Sara & Voutetakis, Spyros & Seferlis, Panos & Gadoue, Shady, 2020. "Reinforcement learning based adaptive power pinch analysis for energy management of stand-alone hybrid energy storage systems considering uncertainty," Energy, Elsevier, vol. 193(C).
    18. Ma, Tengfei & Pei, Wei & Xiao, Hao & Kong, Li & Mu, Yunfei & Pu, Tianjiao, 2020. "The energy management strategies based on dynamic energy pricing for community integrated energy system considering the interactions between suppliers and users," Energy, Elsevier, vol. 211(C).
    19. Kuznetsova, Elizaveta & Li, Yan-Fu & Ruiz, Carlos & Zio, Enrico & Ault, Graham & Bell, Keith, 2013. "Reinforcement learning for microgrid energy management," Energy, Elsevier, vol. 59(C), pages 133-146.
    20. Kong, Xiangyu & Kong, Deqian & Yao, Jingtao & Bai, Linquan & Xiao, Jie, 2020. "Online pricing of demand response based on long short-term memory and reinforcement learning," Applied Energy, Elsevier, vol. 271(C).
    21. Wang, Yongli & Huang, Yujing & Wang, Yudong & Zeng, Ming & Yu, Haiyang & Li, Fang & Zhang, Fuli, 2018. "Optimal scheduling of the RIES considering time-based demand response programs with energy price," Energy, Elsevier, vol. 164(C), pages 773-793.
    22. Ghasemi, Ahmad & Jamshidi Monfared, Houman & Loni, Abdolah & Marzband, Mousa, 2021. "CVaR-based retail electricity pricing in day-ahead scheduling of microgrids," Energy, Elsevier, vol. 227(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sabarathinam Srinivasan & Suresh Kumarasamy & Zacharias E. Andreadakis & Pedro G. Lind, 2023. "Artificial Intelligence and Mathematical Models of Power Grids Driven by Renewable Energy Sources: A Survey," Energies, MDPI, vol. 16(14), pages 1-56, July.
    2. Qiuyi Hong & Fanlin Meng & Jian Liu, 2023. "Customised Multi-Energy Pricing: Model and Solutions," Energies, MDPI, vol. 16(4), pages 1-31, February.
    3. Feng, Wenxiu & Ruiz, Carlos, 2023. "Risk management of energy communities with hydrogen production and storage technologies," Applied Energy, Elsevier, vol. 348(C).
    4. Fan, Lurong & Wang, Binyu & Song, Xiaoling, 2023. "An authority-enterprise equilibrium differentiated subsidy mechanism for promoting coalbed methane extraction in multiple coal seams," Energy, Elsevier, vol. 263(PA).
    5. Zhou, Yanting & Ma, Zhongjing & Zhang, Jinhui & Zou, Suli, 2022. "Data-driven stochastic energy management of multi energy system using deep reinforcement learning," Energy, Elsevier, vol. 261(PA).
    6. Wang, Yudong & Hu, Junjie, 2023. "Two-stage energy management method of integrated energy system considering pre-transaction behavior of energy service provider and users," Energy, Elsevier, vol. 271(C).
    7. Huang, Qian & Xu, Jiuping, 2023. "Carbon tax revenue recycling for biomass/coal co-firing using Stackelberg game: A case study of Jiangsu province, China," Energy, Elsevier, vol. 272(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sabarathinam Srinivasan & Suresh Kumarasamy & Zacharias E. Andreadakis & Pedro G. Lind, 2023. "Artificial Intelligence and Mathematical Models of Power Grids Driven by Renewable Energy Sources: A Survey," Energies, MDPI, vol. 16(14), pages 1-56, July.
    2. Zhu, Ziqing & Hu, Ze & Chan, Ka Wing & Bu, Siqi & Zhou, Bin & Xia, Shiwei, 2023. "Reinforcement learning in deregulated energy market: A comprehensive review," Applied Energy, Elsevier, vol. 329(C).
    3. Cai, Qiran & Xu, Qingyang & Qing, Jing & Shi, Gang & Liang, Qiao-Mei, 2022. "Promoting wind and photovoltaics renewable energy integration through demand response: Dynamic pricing mechanism design and economic analysis for smart residential communities," Energy, Elsevier, vol. 261(PB).
    4. Zhang, Xiongfeng & Lu, Renzhi & Jiang, Junhui & Hong, Seung Ho & Song, Won Seok, 2021. "Testbed implementation of reinforcement learning-based demand response energy management system," Applied Energy, Elsevier, vol. 297(C).
    5. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    6. Correa-Jullian, Camila & López Droguett, Enrique & Cardemil, José Miguel, 2020. "Operation scheduling in a solar thermal system: A reinforcement learning-based framework," Applied Energy, Elsevier, vol. 268(C).
    7. Wen, Lulu & Zhou, Kaile & Li, Jun & Wang, Shanyong, 2020. "Modified deep learning and reinforcement learning for an incentive-based demand response model," Energy, Elsevier, vol. 205(C).
    8. Xu, Fangyuan & Zhu, Weidong & Wang, Yi Fei & Lai, Chun Sing & Yuan, Haoliang & Zhao, Yujia & Guo, Siming & Fu, Zhengxin, 2022. "A new deregulated demand response scheme for load over-shifting city in regulated power market," Applied Energy, Elsevier, vol. 311(C).
    9. Meng, Fanlin & Ma, Qian & Liu, Zixu & Zeng, Xiao-Jun, 2023. "Multiple dynamic pricing for demand response with adaptive clustering-based customer segmentation in smart grids," Applied Energy, Elsevier, vol. 333(C).
    10. Lu, Renzhi & Bai, Ruichang & Ding, Yuemin & Wei, Min & Jiang, Junhui & Sun, Mingyang & Xiao, Feng & Zhang, Hai-Tao, 2021. "A hybrid deep learning-based online energy management scheme for industrial microgrid," Applied Energy, Elsevier, vol. 304(C).
    11. Dai, Yeming & Sun, Xilian & Qi, Yao & Leng, Mingming, 2021. "A real-time, personalized consumption-based pricing scheme for the consumptions of traditional and renewable energies," Renewable Energy, Elsevier, vol. 180(C), pages 452-466.
    12. Pinciroli, Luca & Baraldi, Piero & Compare, Michele & Zio, Enrico, 2023. "Optimal operation and maintenance of energy storage systems in grid-connected microgrids by deep reinforcement learning," Applied Energy, Elsevier, vol. 352(C).
    13. Mansouri, S.A. & Ahmarinejad, A. & Nematbakhsh, E. & Javadi, M.S. & Esmaeel Nezhad, A. & Catalão, J.P.S., 2022. "A sustainable framework for multi-microgrids energy management in automated distribution network by considering smart homes and high penetration of renewable energy resources," Energy, Elsevier, vol. 245(C).
    14. Dadashi-Rad, Mohammad Hosein & Ghasemi-Marzbali, Ali & Ahangar, Roya Ahmadi, 2020. "Modeling and planning of smart buildings energy in power system considering demand response," Energy, Elsevier, vol. 213(C).
    15. Huang, Pei & Sun, Yongjun, 2019. "A collaborative demand control of nearly zero energy buildings in response to dynamic pricing for performance improvements at cluster level," Energy, Elsevier, vol. 174(C), pages 911-921.
    16. Zhao, Liyuan & Yang, Ting & Li, Wei & Zomaya, Albert Y., 2022. "Deep reinforcement learning-based joint load scheduling for household multi-energy system," Applied Energy, Elsevier, vol. 324(C).
    17. Zheng, Lingwei & Wu, Hao & Guo, Siqi & Sun, Xinyu, 2023. "Real-time dispatch of an integrated energy system based on multi-stage reinforcement learning with an improved action-choosing strategy," Energy, Elsevier, vol. 277(C).
    18. Xu, Bo & Wang, Jiexin & Guo, Mengyuan & Lu, Jiayu & Li, Gehui & Han, Liang, 2021. "A hybrid demand response mechanism based on real-time incentive and real-time pricing," Energy, Elsevier, vol. 231(C).
    19. Yang, Jie & Ma, Tieding & Ma, Kai & Yang, Bo & Guerrero, Josep M. & Liu, Zhixin, 2021. "Trading mechanism and pricing strategy of integrated energy systems based on credit rating and Bayesian game," Energy, Elsevier, vol. 232(C).
    20. Dimitrios Vamvakas & Panagiotis Michailidis & Christos Korkas & Elias Kosmatopoulos, 2023. "Review and Evaluation of Reinforcement Learning Frameworks on Smart Grid Applications," Energies, MDPI, vol. 16(14), pages 1-38, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:239:y:2022:i:pa:s0360544221021745. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.