IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v15y2022i9p3290-d806551.html
   My bibliography  Save this article

Unlocking the Flexibility of District Heating Pipeline Energy Storage with Reinforcement Learning

Author

Listed:
  • Ksenija Stepanovic

    (Faculty of Electrical Engineering, Mathematics and Computer Sciences, Delft University of Technology, Van Mourik Broekmanweg 6, 2628 XE Delft, The Netherlands)

  • Jichen Wu

    (Faculty of Electrical Engineering, Mathematics and Computer Sciences, Delft University of Technology, Van Mourik Broekmanweg 6, 2628 XE Delft, The Netherlands
    Flex Technologies, Atoomweg 7, 3542 AA Utrecht, The Netherlands)

  • Rob Everhardt

    (Flex Technologies, Atoomweg 7, 3542 AA Utrecht, The Netherlands)

  • Mathijs de Weerdt

    (Faculty of Electrical Engineering, Mathematics and Computer Sciences, Delft University of Technology, Van Mourik Broekmanweg 6, 2628 XE Delft, The Netherlands)

Abstract

The integration of pipeline energy storage in the control of a district heating system can lead to profit gain, for example by adjusting the electricity production of a combined heat and power (CHP) unit to the fluctuating electricity price. The uncertainty from the environment, the computational complexity of an accurate model, and the scarcity of placed sensors in a district heating system make the operational use of pipeline energy storage challenging. A vast majority of previous works determined a control strategy by a decomposition of a mixed-integer nonlinear model and significant simplifications. To mitigate consequential stability, feasibility, and computational complexity challenges, we model CHP economic dispatch as a Markov decision process. We use a reinforcement learning (RL) algorithm to estimate the system’s dynamics through interactions with the simulation environment. The RL approach is compared with a detailed nonlinear mathematical optimizer on day-ahead and real-time electricity markets and two district heating grid models. The proposed method achieves moderate profit impacted by environment stochasticity. The advantages of the RL approach are reflected in three aspects: stability, feasibility, and time scale flexibility. From this, it can be concluded that RL is a promising alternative for real-time control of complex, nonlinear industrial systems.

Suggested Citation

  • Ksenija Stepanovic & Jichen Wu & Rob Everhardt & Mathijs de Weerdt, 2022. "Unlocking the Flexibility of District Heating Pipeline Energy Storage with Reinforcement Learning," Energies, MDPI, vol. 15(9), pages 1-25, April.
  • Handle: RePEc:gam:jeners:v:15:y:2022:i:9:p:3290-:d:806551
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/15/9/3290/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/1996-1073/15/9/3290/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Lund, Rasmus & Mathiesen, Brian Vad, 2015. "Large combined heat and power plants in sustainable energy systems," Applied Energy, Elsevier, vol. 142(C), pages 389-395.
    2. Zou, Dexuan & Li, Steven & Kong, Xiangyong & Ouyang, Haibin & Li, Zongyan, 2019. "Solving the combined heat and power economic dispatch problems by an improved genetic algorithm and a new constraint handling strategy," Applied Energy, Elsevier, vol. 237(C), pages 646-670.
    3. Guelpa, Elisa & Verda, Vittorio, 2019. "Thermal energy storage in district heating and cooling systems: A review," Applied Energy, Elsevier, vol. 252(C), pages 1-1.
    4. Ping Li & Haixia Wang & Quan Lv & Weidong Li, 2017. "Combined Heat and Power Dispatch Considering Heat Storage of Both Buildings and Pipelines in District Heating System for Wind Power Integration," Energies, MDPI, vol. 10(7), pages 1-19, June.
    5. Lund, Henrik & Werner, Sven & Wiltshire, Robin & Svendsen, Svend & Thorsen, Jan Eric & Hvelplund, Frede & Mathiesen, Brian Vad, 2014. "4th Generation District Heating (4GDH)," Energy, Elsevier, vol. 68(C), pages 1-11.
    6. Bonan Huang & Chaoming Zheng & Qiuye Sun & Ruixue Hu, 2019. "Optimal Economic Dispatch for Integrated Power and Heating Systems Considering Transmission Losses," Energies, MDPI, vol. 12(13), pages 1-19, June.
    7. Averfalk, Helge & Werner, Sven, 2020. "Economic benefits of fourth generation district heating," Energy, Elsevier, vol. 193(C).
    8. Lennart Merkert & Ashvar Abdoul Haime & Sören Hohmann, 2019. "Optimal Scheduling of Combined Heat and Power Generation Units Using the Thermal Inertia of the Connected District Heating Grid as Energy Storage," Energies, MDPI, vol. 12(2), pages 1-9, January.
    9. Sorknæs, Peter & Lund, Henrik & Andersen, Anders N., 2015. "Future power market and sustainable energy solutions – The treatment of uncertainties in the daily operation of combined heat and power plants," Applied Energy, Elsevier, vol. 144(C), pages 129-138.
    10. Vandermeulen, Annelies & van der Heijde, Bram & Helsen, Lieve, 2018. "Controlling district heating and cooling networks to unlock flexibility: A review," Energy, Elsevier, vol. 151(C), pages 103-115.
    11. Bloess, Andreas, 2020. "Modeling of combined heat and power generation in the context of increasing renewable energy penetration," Applied Energy, Elsevier, vol. 267(C).
    12. Cai, Hanmin & Ziras, Charalampos & You, Shi & Li, Rongling & Honoré, Kristian & Bindner, Henrik W., 2018. "Demand side management in urban district heating networks," Applied Energy, Elsevier, vol. 230(C), pages 506-518.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Danica Djurić Ilić, 2020. "Classification of Measures for Dealing with District Heating Load Variations—A Systematic Review," Energies, MDPI, vol. 14(1), pages 1-27, December.
    2. Egging-Bratseth, Ruud & Kauko, Hanne & Knudsen, Brage Rugstad & Bakke, Sara Angell & Ettayebi, Amina & Haufe, Ina Renate, 2021. "Seasonal storage and demand side management in district heating systems with demand uncertainty," Applied Energy, Elsevier, vol. 285(C).
    3. Møller Sneum, Daniel, 2021. "Barriers to flexibility in the district energy-electricity system interface – A taxonomy," Renewable and Sustainable Energy Reviews, Elsevier, vol. 145(C).
    4. Guelpa, Elisa & Verda, Vittorio, 2021. "Demand response and other demand side management techniques for district heating: A review," Energy, Elsevier, vol. 219(C).
    5. Levihn, Fabian, 2017. "CHP and heat pumps to balance renewable power production: Lessons from the district heating network in Stockholm," Energy, Elsevier, vol. 137(C), pages 670-678.
    6. He, Ke-Lun & Zhao, Tian & Ma, Huan & Chen, Qun, 2023. "Optimal operation of integrated power and thermal systems for flexibility improvement based on evaluation and utilization of heat storage in district heating systems," Energy, Elsevier, vol. 274(C).
    7. Lund, Rasmus & Persson, Urban, 2016. "Mapping of potential heat sources for heat pumps for district heating in Denmark," Energy, Elsevier, vol. 110(C), pages 129-138.
    8. Meibodi, Saleh S. & Loveridge, Fleur, 2022. "The future role of energy geostructures in fifth generation district heating and cooling networks," Energy, Elsevier, vol. 240(C).
    9. Merkert, Lennart & Listmann, Kim & Hohmann, Sören, 2019. "Optimization of thermo-hydraulic systems using multiparametric delay modeling," Energy, Elsevier, vol. 189(C).
    10. Cai, Hanmin & You, Shi & Wu, Jianzhong, 2020. "Agent-based distributed demand response in district heating systems," Applied Energy, Elsevier, vol. 262(C).
    11. Dincer, Ibrahim & Acar, Canan, 2017. "Smart energy systems for a sustainable future," Applied Energy, Elsevier, vol. 194(C), pages 225-235.
    12. Østergaard, Poul Alberg & Andersen, Anders N., 2016. "Booster heat pumps and central heat pumps in district heating," Applied Energy, Elsevier, vol. 184(C), pages 1374-1388.
    13. Saletti, Costanza & Zimmerman, Nathan & Morini, Mirko & Kyprianidis, Konstantinos & Gambarotta, Agostino, 2021. "Enabling smart control by optimally managing the State of Charge of district heating networks," Applied Energy, Elsevier, vol. 283(C).
    14. Nageler, P. & Heimrath, R. & Mach, T. & Hochenauer, C., 2019. "Prototype of a simulation framework for georeferenced large-scale dynamic simulations of district energy systems," Applied Energy, Elsevier, vol. 252(C), pages 1-1.
    15. Lund, Henrik & Østergaard, Poul Alberg & Chang, Miguel & Werner, Sven & Svendsen, Svend & Sorknæs, Peter & Thorsen, Jan Eric & Hvelplund, Frede & Mortensen, Bent Ole Gram & Mathiesen, Brian Vad & Boje, 2018. "The status of 4th generation district heating: Research and results," Energy, Elsevier, vol. 164(C), pages 147-159.
    16. Jimenez-Navarro, Juan-Pablo & Kavvadias, Konstantinos & Filippidou, Faidra & Pavičević, Matija & Quoilin, Sylvain, 2020. "Coupling the heating and power sectors: The role of centralised combined heat and power plants and district heat in a European decarbonised power system," Applied Energy, Elsevier, vol. 270(C).
    17. Sorknæs, Peter & Østergaard, Poul Alberg & Thellufsen, Jakob Zinck & Lund, Henrik & Nielsen, Steffen & Djørup, Søren & Sperling, Karl, 2020. "The benefits of 4th generation district heating in a 100% renewable energy system," Energy, Elsevier, vol. 213(C).
    18. Vandermeulen, Annelies & Van Oevelen, Tijs & van der Heijde, Bram & Helsen, Lieve, 2020. "A simulation-based evaluation of substation models for network flexibility characterisation in district heating networks," Energy, Elsevier, vol. 201(C).
    19. Selva Calixto & Marco Cozzini & Giampaolo Manzolini, 2021. "Modelling of an Existing Neutral Temperature District Heating Network: Detailed and Approximate Approaches," Energies, MDPI, vol. 14(2), pages 1-16, January.
    20. De Lorenzi, Andrea & Gambarotta, Agostino & Morini, Mirko & Rossi, Michele & Saletti, Costanza, 2020. "Setup and testing of smart controllers for small-scale district heating networks: An integrated framework," Energy, Elsevier, vol. 205(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:15:y:2022:i:9:p:3290-:d:806551. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.