IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v288y2021ics0306261921001586.html
   My bibliography  Save this article

Deep reinforcement learning framework for dynamic pricing demand response of regenerative electric heating

Author

Listed:
  • Zhong, Shengyuan
  • Wang, Xiaoyuan
  • Zhao, Jun
  • Li, Wenjia
  • Li, Hao
  • Wang, Yongzhen
  • Deng, Shuai
  • Zhu, Jiebei

Abstract

Applications of electric heating, which can improve carbon emission reduction and renewable energy utilization, have brought new challenges to the safe operation of energy systems around the world. Regenerative electric heating with load aggregators and demand response is an effective means to mitigate the wind curtailment and grid operational risks caused by electric heating. However, there is still a lack of models related to demand response, which results in participants not being able to obtain maximum benefits through dynamic subsidy prices. This study uses the Weber–Fechner law and a clustering algorithm to construct quantitative response characteristics models. The deep Q network was used to build a dynamic subsidy price generation framework for load aggregators. Through simulation analysis based on the evolutionary game model of a project in a rural area in Tianjin, China, the following conclusions were drawn: compared with the benchmark model, regenerative electric heating users can save up to 8.7% of costs, power grid companies can save 56.6% of their investment, and wind power plants can increase wind power consumption by 17.6%. The framework proposed in this study considers user behavior quantification of demand response participants and the differences among users. Therefore, the framework can provide a more reasonable, applicable, and intelligent system for regenerative electric heating.

Suggested Citation

  • Zhong, Shengyuan & Wang, Xiaoyuan & Zhao, Jun & Li, Wenjia & Li, Hao & Wang, Yongzhen & Deng, Shuai & Zhu, Jiebei, 2021. "Deep reinforcement learning framework for dynamic pricing demand response of regenerative electric heating," Applied Energy, Elsevier, vol. 288(C).
    • Handle: RePEc:eee:appene:v:288:y:2021:i:c:s0306261921001586
    DOI: 10.1016/j.apenergy.2021.116623
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261921001586
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2021.116623?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Fang, Kai & Tang, Yiqi & Zhang, Qifeng & Song, Junnian & Wen, Qi & Sun, Huaping & Ji, Chenyang & Xu, Anqi, 2019. "Will China peak its energy-related carbon emissions by 2030? Lessons from 30 Chinese provinces," Applied Energy, Elsevier, vol. 255(C).
    2. Al-Wakeel, Ali & Wu, Jianzhong & Jenkins, Nick, 2017. "k-means based load estimation of domestic smart meter measurements," Applied Energy, Elsevier, vol. 194(C), pages 333-342.
    3. Dorotić, Hrvoje & Ban, Marko & Pukšec, Tomislav & Duić, Neven, 2020. "Impact of wind penetration in electricity markets on optimal power-to-heat capacities in a local district heating system," Renewable and Sustainable Energy Reviews, Elsevier, vol. 132(C).
    4. Liu, Zuming & Zhao, Yingru & Wang, Xiaonan, 2020. "Long-term economic planning of combined cooling heating and power systems considering energy storage and demand response," Applied Energy, Elsevier, vol. 279(C).
    5. Zhou, Yue & Wang, Chengshan & Wu, Jianzhong & Wang, Jidong & Cheng, Meng & Li, Gen, 2017. "Optimal scheduling of aggregated thermostatically controlled loads with renewable generation in the intraday electricity market," Applied Energy, Elsevier, vol. 188(C), pages 456-465.
    6. Romanchenko, Dmytro & Odenberger, Mikael & Göransson, Lisa & Johnsson, Filip, 2017. "Impact of electricity price fluctuations on the operation of district heating systems: A case study of district heating in Göteborg, Sweden," Applied Energy, Elsevier, vol. 204(C), pages 16-30.
    7. Wu, Xiaomin & Cao, Weihua & Wang, Dianhong & Ding, Min & Yu, Liangjun & Nakanishi, Yosuke, 2021. "Demand response model based on improved Pareto optimum considering seasonal electricity prices for Dongfushan Island," Renewable Energy, Elsevier, vol. 164(C), pages 926-936.
    8. Ayón, X. & Gruber, J.K. & Hayes, B.P. & Usaola, J. & Prodanović, M., 2017. "An optimal day-ahead load scheduling approach based on the flexibility of aggregate demands," Applied Energy, Elsevier, vol. 198(C), pages 1-11.
    9. Patteeuw, Dieter & Bruninx, Kenneth & Arteconi, Alessia & Delarue, Erik & D’haeseleer, William & Helsen, Lieve, 2015. "Integrated modeling of active demand response with electric heating systems coupled to thermal energy storage systems," Applied Energy, Elsevier, vol. 151(C), pages 306-319.
    10. Wu, Jingda & He, Hongwen & Peng, Jiankun & Li, Yuecheng & Li, Zhanjiang, 2018. "Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus," Applied Energy, Elsevier, vol. 222(C), pages 799-811.
    11. Aunedi, Marko & Pantaleo, Antonio Marco & Kuriyan, Kamal & Strbac, Goran & Shah, Nilay, 2020. "Modelling of national and local interactions between heat and electricity networks in low-carbon energy systems," Applied Energy, Elsevier, vol. 276(C).
    12. Pan, Guangsheng & Gu, Wei & Wu, Zhi & Lu, Yuping & Lu, Shuai, 2019. "Optimal design and operation of multi-energy system with load aggregator considering nodal energy prices," Applied Energy, Elsevier, vol. 239(C), pages 280-295.
    13. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    14. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    15. Yang, Mao & Shi, Chaoyu & Liu, Huiyu, 2021. "Day-ahead wind power forecasting based on the clustering of equivalent power curves," Energy, Elsevier, vol. 218(C).
    16. Dong, Changgui & Qi, Ye & Dong, Wenjuan & Lu, Xi & Liu, Tianle & Qian, Shuai, 2018. "Decomposing driving factors for wind curtailment under economic new normal in China," Applied Energy, Elsevier, vol. 217(C), pages 178-188.
    17. Li, Xue & Li, Wenming & Zhang, Rufeng & Jiang, Tao & Chen, Houhe & Li, Guoqing, 2020. "Collaborative scheduling and flexibility assessment of integrated electricity and district heating systems utilizing thermal inertia of district heating network and aggregated buildings," Applied Energy, Elsevier, vol. 258(C).
    18. Fang, Yujuan & Chen, Laijun & Mei, Shengwei & Wei, Wei & Huang, Shaowei & Liu, Feng, 2019. "Coal or electricity? An evolutionary game approach to investigate fuel choices of urban heat supply systems," Energy, Elsevier, vol. 181(C), pages 107-122.
    19. Eyre, Nick & Baruah, Pranab, 2015. "Uncertainties in future energy demand in UK residential heating," Energy Policy, Elsevier, vol. 87(C), pages 641-653.
    20. Kong, Xiangyu & Kong, Deqian & Yao, Jingtao & Bai, Linquan & Xiao, Jie, 2020. "Online pricing of demand response based on long short-term memory and reinforcement learning," Applied Energy, Elsevier, vol. 271(C).
    21. Tarroja, Brian & Chiang, Felicia & AghaKouchak, Amir & Samuelsen, Scott & Raghavan, Shuba V. & Wei, Max & Sun, Kaiyu & Hong, Tianzhen, 2018. "Translating climate change and heating system electrification impacts on building energy use to future greenhouse gas emissions and electric grid capacity requirements in California," Applied Energy, Elsevier, vol. 225(C), pages 522-534.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Shen, Rendong & Zhong, Shengyuan & Wen, Xin & An, Qingsong & Zheng, Ruifan & Li, Yang & Zhao, Jun, 2022. "Multi-agent deep reinforcement learning optimization framework for building energy system with renewable energy," Applied Energy, Elsevier, vol. 312(C).
    2. Sabarathinam Srinivasan & Suresh Kumarasamy & Zacharias E. Andreadakis & Pedro G. Lind, 2023. "Artificial Intelligence and Mathematical Models of Power Grids Driven by Renewable Energy Sources: A Survey," Energies, MDPI, vol. 16(14), pages 1-56, July.
    3. Xu, Fangyuan & Zhu, Weidong & Wang, Yi Fei & Lai, Chun Sing & Yuan, Haoliang & Zhao, Yujia & Guo, Siming & Fu, Zhengxin, 2022. "A new deregulated demand response scheme for load over-shifting city in regulated power market," Applied Energy, Elsevier, vol. 311(C).
    4. Meng, Fanlin & Ma, Qian & Liu, Zixu & Zeng, Xiao-Jun, 2023. "Multiple dynamic pricing for demand response with adaptive clustering-based customer segmentation in smart grids," Applied Energy, Elsevier, vol. 333(C).
    5. Gao, Yuan & Matsunami, Yuki & Miyata, Shohei & Akashi, Yasunori, 2022. "Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system," Applied Energy, Elsevier, vol. 326(C).
    6. Xie, Jiahan & Ajagekar, Akshay & You, Fengqi, 2023. "Multi-Agent attention-based deep reinforcement learning for demand response in grid-responsive buildings," Applied Energy, Elsevier, vol. 342(C).
    7. Guixing Yang & Haoran Liu & Weiqing Wang & Junru Chen & Shunbo Lei, 2023. "Distributed Optimal Coordination of a Virtual Power Plant with Residential Regenerative Electric Heating Systems," Energies, MDPI, vol. 16(11), pages 1-15, May.
    8. Halid Kaplan & Kambiz Tehrani & Mo Jamshidi, 2021. "A Fault Diagnosis Design Based on Deep Learning Approach for Electric Vehicle Applications," Energies, MDPI, vol. 14(20), pages 1-14, October.
    9. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    10. Yilin Liang & Yuping Hu & Dongjun Luo & Qi Zhu & Qingxuan Chen & Chunmei Wang, 2023. "Distributed Dynamic Pricing Strategy Based on Deep Reinforcement Learning Approach in a Presale Mechanism," Sustainability, MDPI, vol. 15(13), pages 1-20, July.
    11. Eduardo J. Salazar & Mauro Jurado & Mauricio E. Samper, 2023. "Reinforcement Learning-Based Pricing and Incentive Strategy for Demand Response in Smart Grids," Energies, MDPI, vol. 16(3), pages 1-33, February.
    12. Mahdi Khodayar & Jacob Regan, 2023. "Deep Neural Networks in Power Systems: A Review," Energies, MDPI, vol. 16(12), pages 1-38, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    2. Fan, Man & Wang, Jia & Kong, Xiangfei & Suo, Hanxiao & Zheng, Wandong & Li, Han, 2023. "Experimental evaluation of the cascaded energy storage radiator for constructing indoor thermal environment in winter," Applied Energy, Elsevier, vol. 332(C).
    3. Pied, Marie & Anjos, Miguel F. & Malhamé, Roland P., 2020. "A flexibility product for electric water heater aggregators on electricity markets," Applied Energy, Elsevier, vol. 280(C).
    4. Verástegui, Felipe & Lorca, Álvaro & Negrete-Pincetic, Matias & Olivares, Daniel, 2020. "Firewood heat electrification impacts in the Chilean power system," Energy Policy, Elsevier, vol. 144(C).
    5. Ahmad, Tanveer & Madonski, Rafal & Zhang, Dongdong & Huang, Chao & Mujeeb, Asad, 2022. "Data-driven probabilistic machine learning in sustainable smart energy/smart energy systems: Key developments, challenges, and future research opportunities in the context of smart grid paradigm," Renewable and Sustainable Energy Reviews, Elsevier, vol. 160(C).
    6. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Data-driven battery operation for energy arbitrage using rainbow deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    7. Iria, José & Soares, Filipe & Matos, Manuel, 2019. "Optimal bidding strategy for an aggregator of prosumers in energy and secondary reserve markets," Applied Energy, Elsevier, vol. 238(C), pages 1361-1372.
    8. Zhu, Ziqing & Hu, Ze & Chan, Ka Wing & Bu, Siqi & Zhou, Bin & Xia, Shiwei, 2023. "Reinforcement learning in deregulated energy market: A comprehensive review," Applied Energy, Elsevier, vol. 329(C).
    9. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Renewable energy integration and microgrid energy trading using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 318(C).
    10. Ding, Jianyong & Gao, Ciwei & Song, Meng & Yan, Xingyu & Chen, Tao, 2022. "Bi-level optimal scheduling of virtual energy station based on equal exergy replacement mechanism," Applied Energy, Elsevier, vol. 327(C).
    11. Homod, Raad Z. & Togun, Hussein & Kadhim Hussein, Ahmed & Noraldeen Al-Mousawi, Fadhel & Yaseen, Zaher Mundher & Al-Kouz, Wael & Abd, Haider J. & Alawi, Omer A. & Goodarzi, Marjan & Hussein, Omar A., 2022. "Dynamics analysis of a novel hybrid deep clustering for unsupervised learning by reinforcement of multi-agent to energy saving in intelligent buildings," Applied Energy, Elsevier, vol. 313(C).
    12. Iria, José & Soares, Filipe & Matos, Manuel, 2018. "Optimal supply and demand bidding strategy for an aggregator of small prosumers," Applied Energy, Elsevier, vol. 213(C), pages 658-669.
    13. Kong, Xiangyu & Kong, Deqian & Yao, Jingtao & Bai, Linquan & Xiao, Jie, 2020. "Online pricing of demand response based on long short-term memory and reinforcement learning," Applied Energy, Elsevier, vol. 271(C).
    14. Xiao, Boyi & Yang, Weiwei & Wu, Jiamin & Walker, Paul D. & Zhang, Nong, 2022. "Energy management strategy via maximum entropy reinforcement learning for an extended range logistics vehicle," Energy, Elsevier, vol. 253(C).
    15. Shen, Rendong & Zhong, Shengyuan & Wen, Xin & An, Qingsong & Zheng, Ruifan & Li, Yang & Zhao, Jun, 2022. "Multi-agent deep reinforcement learning optimization framework for building energy system with renewable energy," Applied Energy, Elsevier, vol. 312(C).
    16. Zhou, Jianhao & Xue, Siwu & Xue, Yuan & Liao, Yuhui & Liu, Jun & Zhao, Wanzhong, 2021. "A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning," Energy, Elsevier, vol. 224(C).
    17. Biemann, Marco & Scheller, Fabian & Liu, Xiufeng & Huang, Lizhen, 2021. "Experimental evaluation of model-free reinforcement learning algorithms for continuous HVAC control," Applied Energy, Elsevier, vol. 298(C).
    18. Xie, Jiahan & Ajagekar, Akshay & You, Fengqi, 2023. "Multi-Agent attention-based deep reinforcement learning for demand response in grid-responsive buildings," Applied Energy, Elsevier, vol. 342(C).
    19. Zeng, Huibin & Shao, Bilin & Dai, Hongbin & Tian, Ning & Zhao, Wei, 2023. "Incentive-based demand response strategies for natural gas considering carbon emissions and load volatility," Applied Energy, Elsevier, vol. 348(C).
    20. Eduardo J. Salazar & Mauro Jurado & Mauricio E. Samper, 2023. "Reinforcement Learning-Based Pricing and Incentive Strategy for Demand Response in Smart Grids," Energies, MDPI, vol. 16(3), pages 1-33, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:288:y:2021:i:c:s0306261921001586. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.