IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v322y2025ics0360544225009582.html
   My bibliography  Save this article

Bi-level day-ahead and real-time hybrid pricing model and its reinforcement learning method

Author

Listed:
  • He, Youmeng
  • Gu, Chunhua
  • Gao, Yan
  • Wang, Jingqi

Abstract

Both multi-time slot (day-ahead) pricing and single-time slot (real-time) pricing are vital parts of real-time pricing. The widespread utilization of renewable energy sources has increased flexibility and uncertainty of the grid system. A single pricing strategy is unable to meet the demand of the grid. This paper design a hybrid pricing strategy for smart grids that combines real-time and day-ahead pricing. This strategy considers multi-source energy generation on the supply side, also, the distributed energy generation and load transfer on the demand side. Within the framework of the Markov Decision Process, a bi-level stochastic model for real-time demand response is formulated to maximize the benefits of both the supply and demand sides. Subsequently, a deep deterministic policy gradient algorithm relying on prioritized experience replay is used to formulate a real-time price plan and user’s power consumption. Through the information interaction between the upper and lower levels, the real-time prices are decided adaptively. Meanwhile, the optimal strategy of power supply and consumption are obtained. Our simulation results demonstrate that the proposed hybrid pricing strategy guarantee the benefits of both the supply and demand sides, while achieving the balance between supply and demand.

Suggested Citation

  • He, Youmeng & Gu, Chunhua & Gao, Yan & Wang, Jingqi, 2025. "Bi-level day-ahead and real-time hybrid pricing model and its reinforcement learning method," Energy, Elsevier, vol. 322(C).
    • Handle: RePEc:eee:energy:v:322:y:2025:i:c:s0360544225009582
    DOI: 10.1016/j.energy.2025.135316
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544225009582
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2025.135316?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Chang, Zihan & Zhang, Yang & Chen, Wenbo, 2019. "Electricity price prediction based on hybrid model of adam optimized LSTM neural network and wavelet transform," Energy, Elsevier, vol. 187(C).
    2. Doostizadeh, Meysam & Ghasemi, Hassan, 2012. "A day-ahead electricity pricing model based on smart metering and demand-side management," Energy, Elsevier, vol. 46(1), pages 221-230.
    3. Li, Yuanyuan & Li, Junxiang & He, Jianjia & Zhang, Shuyuan, 2021. "The real-time pricing optimization model of smart grid based on the utility function of the logistic function," Energy, Elsevier, vol. 224(C).
    4. Shang, Yuwei & Wu, Wenchuan & Guo, Jianbo & Ma, Zhao & Sheng, Wanxing & Lv, Zhe & Fu, Chenran, 2020. "Stochastic dispatch of energy storage in microgrids: An augmented reinforcement learning approach," Applied Energy, Elsevier, vol. 261(C).
    5. Luan, Wenpeng & Tian, Longfei & Zhao, Bochao, 2023. "Leveraging hybrid probabilistic multi-objective evolutionary algorithm for dynamic tariff design," Applied Energy, Elsevier, vol. 342(C).
    6. Zhang, Li & Gao, Yan & Zhu, Hongbo & Tao, Li, 2022. "Bi-level stochastic real-time pricing model in multi-energy generation system: A reinforcement learning approach," Energy, Elsevier, vol. 239(PA).
    7. Lu, Renzhi & Hong, Seung Ho & Zhang, Xiongfeng, 2018. "A Dynamic pricing demand response algorithm for smart grid: Reinforcement learning approach," Applied Energy, Elsevier, vol. 220(C), pages 220-230.
    8. Srinivasan, Dipti & Rajgarhia, Sanjana & Radhakrishnan, Bharat Menon & Sharma, Anurag & Khincha, H.P., 2017. "Game-Theory based dynamic pricing strategies for demand side management in smart grids," Energy, Elsevier, vol. 126(C), pages 132-143.
    9. Chen, Zheng & Hu, Hengjie & Wu, Yitao & Zhang, Yuanjian & Li, Guang & Liu, Yonggang, 2020. "Stochastic model predictive control for energy management of power-split plug-in hybrid electric vehicles based on reinforcement learning," Energy, Elsevier, vol. 211(C).
    10. Yuan, Guanxiu & Gao, Yan & Ye, Bei, 2021. "Optimal dispatching strategy and real-time pricing for multi-regional integrated energy systems based on demand response," Renewable Energy, Elsevier, vol. 179(C), pages 1424-1446.
    11. Kong, Xiangyu & Lu, Wenqi & Wu, Jianzhong & Wang, Chengshan & Zhao, Xv & Hu, Wei & Shen, Yu, 2023. "Real-time pricing method for VPP demand response based on PER-DDPG algorithm," Energy, Elsevier, vol. 271(C).
    12. Yang, Ningkang & Han, Lijin & Xiang, Changle & Liu, Hui & Li, Xunmin, 2021. "An indirect reinforcement learning based real-time energy management strategy via high-order Markov Chain model for a hybrid electric vehicle," Energy, Elsevier, vol. 236(C).
    13. Norouzi, F. & Hoppe, T. & Kamp, L.M. & Manktelow, C. & Bauer, P., 2023. "Diagnosis of the implementation of smart grid innovation in The Netherlands and corrective actions," Renewable and Sustainable Energy Reviews, Elsevier, vol. 175(C).
    14. Kim, Icksung & Kim, Woohyun, 2023. "Application of market-based control with thermal energy storage system for demand limiting and real-time pricing control," Energy, Elsevier, vol. 263(PA).
    15. Wang, Liying & Lin, Jialin & Dong, Houqi & Wang, Yuqing & Zeng, Ming, 2023. "Demand response comprehensive incentive mechanism-based multi-time scale optimization scheduling for park integrated energy system," Energy, Elsevier, vol. 270(C).
    16. Kong, Xiangyu & Kong, Deqian & Yao, Jingtao & Bai, Linquan & Xiao, Jie, 2020. "Online pricing of demand response based on long short-term memory and reinforcement learning," Applied Energy, Elsevier, vol. 271(C).
    17. Li, Qi & Xiao, Xukang & Pu, Yuchen & Luo, Shuyu & Liu, Hong & Chen, Weirong, 2023. "Hierarchical optimal scheduling method for regional integrated energy systems considering electricity-hydrogen shared energy," Applied Energy, Elsevier, vol. 349(C).
    18. Yang, Ting & Zhao, Liyuan & Li, Wei & Zomaya, Albert Y., 2021. "Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning," Energy, Elsevier, vol. 235(C).
    19. Cai, Mengmeng & Pipattanasomporn, Manisa & Rahman, Saifur, 2019. "Day-ahead building-level load forecasts using deep learning vs. traditional time-series techniques," Applied Energy, Elsevier, vol. 236(C), pages 1078-1088.
    20. Dai, Yeming & Sun, Xilian & Qi, Yao & Leng, Mingming, 2021. "A real-time, personalized consumption-based pricing scheme for the consumptions of traditional and renewable energies," Renewable Energy, Elsevier, vol. 180(C), pages 452-466.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhang, Li & Gao, Yan & Zhu, Hongbo & Tao, Li, 2022. "Bi-level stochastic real-time pricing model in multi-energy generation system: A reinforcement learning approach," Energy, Elsevier, vol. 239(PA).
    2. Zhu, Ziqing & Hu, Ze & Chan, Ka Wing & Bu, Siqi & Zhou, Bin & Xia, Shiwei, 2023. "Reinforcement learning in deregulated energy market: A comprehensive review," Applied Energy, Elsevier, vol. 329(C).
    3. Wang, Yudong & Hu, Junjie, 2023. "Two-stage energy management method of integrated energy system considering pre-transaction behavior of energy service provider and users," Energy, Elsevier, vol. 271(C).
    4. Li, Ningning & Gao, Yan, 2023. "Real-time pricing based on convex hull method for smart grid with multiple generating units," Energy, Elsevier, vol. 285(C).
    5. Lee, Hyun-Suk, 2024. "Automated tariff design for energy supply–demand matching based on Bayesian optimization: Technical framework and policy implications," Energy Policy, Elsevier, vol. 188(C).
    6. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    7. Sabarathinam Srinivasan & Suresh Kumarasamy & Zacharias E. Andreadakis & Pedro G. Lind, 2023. "Artificial Intelligence and Mathematical Models of Power Grids Driven by Renewable Energy Sources: A Survey," Energies, MDPI, vol. 16(14), pages 1-56, July.
    8. Hu, Rong & Zhou, Kaile & Yin, Hui, 2024. "Reinforcement learning model for incentive-based integrated demand response considering demand-side coupling," Energy, Elsevier, vol. 308(C).
    9. Zhou, Kaile & Peng, Ning & Yin, Hui & Hu, Rong, 2023. "Urban virtual power plant operation optimization with incentive-based demand response," Energy, Elsevier, vol. 282(C).
    10. Wang, Tonghe & Hua, Haochen & Shi, Tianying & Wang, Rui & Sun, Yizhong & Naidoo, Pathmanathan, 2024. "A bi-level dispatch optimization of multi-microgrid considering green electricity consumption willingness under renewable portfolio standard policy," Applied Energy, Elsevier, vol. 356(C).
    11. Ibrahim, Muhammad Sohail & Dong, Wei & Yang, Qiang, 2020. "Machine learning driven smart electric power systems: Current trends and new perspectives," Applied Energy, Elsevier, vol. 272(C).
    12. Cai, Qiran & Xu, Qingyang & Qing, Jing & Shi, Gang & Liang, Qiao-Mei, 2022. "Promoting wind and photovoltaics renewable energy integration through demand response: Dynamic pricing mechanism design and economic analysis for smart residential communities," Energy, Elsevier, vol. 261(PB).
    13. Huang, Pei & Sun, Yongjun, 2019. "A collaborative demand control of nearly zero energy buildings in response to dynamic pricing for performance improvements at cluster level," Energy, Elsevier, vol. 174(C), pages 911-921.
    14. Zhao, Liyuan & Yang, Ting & Li, Wei & Zomaya, Albert Y., 2022. "Deep reinforcement learning-based joint load scheduling for household multi-energy system," Applied Energy, Elsevier, vol. 324(C).
    15. Zheng, Lingwei & Wu, Hao & Guo, Siqi & Sun, Xinyu, 2023. "Real-time dispatch of an integrated energy system based on multi-stage reinforcement learning with an improved action-choosing strategy," Energy, Elsevier, vol. 277(C).
    16. Jiang, Meihui & Xu, Zhenjiang & Zhu, Hongyu & Hwang Goh, Hui & Agustiono Kurniawan, Tonni & Liu, Tianhao & Zhang, Dongdong, 2024. "Integrated demand response modeling and optimization technologies supporting energy internet," Renewable and Sustainable Energy Reviews, Elsevier, vol. 203(C).
    17. Ma, Siyu & Liu, Hui & Wang, Ni & Huang, Lidong & Goh, Hui Hwang, 2023. "Incentive-based demand response under incomplete information based on the deep deterministic policy gradient," Applied Energy, Elsevier, vol. 351(C).
    18. Rong Xia & Jun Dai & Xiangjie Cheng & Jiaqing Fan & Jing Ye & Qiangang Jia & Sijie Chen & Qiang Zhang, 2024. "Demand Response of Integrated Zero-Carbon Power Plant: Model and Method," Energies, MDPI, vol. 17(14), pages 1-15, July.
    19. Grace Muriithi & Sunetra Chowdhury, 2021. "Optimal Energy Management of a Grid-Tied Solar PV-Battery Microgrid: A Reinforcement Learning Approach," Energies, MDPI, vol. 14(9), pages 1-24, May.
    20. Marouane Adnane & Ahmed Khoumsi & João Pedro F. Trovão, 2023. "Efficient Management of Energy Consumption of Electric Vehicles Using Machine Learning—A Systematic and Comprehensive Survey," Energies, MDPI, vol. 16(13), pages 1-39, June.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:322:y:2025:i:c:s0360544225009582. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.