IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v278y2020ics0306261920311399.html
   My bibliography  Save this article

Control of superheat of organic Rankine cycle under transient heat source based on deep reinforcement learning

Author

Listed:
  • Wang, Xuan
  • Wang, Rui
  • Jin, Ming
  • Shu, Gequn
  • Tian, Hua
  • Pan, Jiaying

Abstract

The organic Rankine cycle (ORC) is a promising technology for engine waste heat recovery. During real-world operation, the engine working condition varies frequently to satisfy the power demand; thus, the transient nature of engine waste heat presents significant control challenges for the ORC. To control the superheat of the ORC precisely under a transient heat source, several optimal control methods have been used such as model predictive control and dynamic programing. However, most of them depend strongly on the accurate prediction of future disturbances. Deep reinforcement learning (DRL) is an artificial-intelligence algorithm that can overcome the aforementioned disadvantage, but the potential of DRL in control of thermodynamic systems has not yet been investigated. Thus, this paper proposes two DRL-based control methods for controlling the superheat of ORC under a transient heat source. One directly uses the DRL agent to learn the control strategy (DRL control), and the other uses the DRL agent to optimize the parameters of the proportional–integral–derivative (PID) controller (DRL-based PID control). Additionally, a switching mechanism between different DRL controllers is proposed for improving the training efficiency and enlarging the operation range of the controller. The results of this study indicate that the DRL agent can satisfactorily perform the control task and optimize the traditional controller under the trained and untrained transient heat source. Specifically, the DRL control can track the reference superheat with an average error of only 0.19 K, whereas that of the traditional PID control is 2.16 K. Furthermore, the proposed switching DRL control exhibits excellent tracking performance with an average error of only 0.21 K and robustness over a wide range of operation conditions. The successful application of DRL demonstrates its considerable potential for the control of thermodynamic systems, providing a useful reference and motivation for the application to other thermodynamic systems.

Suggested Citation

  • Wang, Xuan & Wang, Rui & Jin, Ming & Shu, Gequn & Tian, Hua & Pan, Jiaying, 2020. "Control of superheat of organic Rankine cycle under transient heat source based on deep reinforcement learning," Applied Energy, Elsevier, vol. 278(C).
    • Handle: RePEc:eee:appene:v:278:y:2020:i:c:s0306261920311399
    DOI: 10.1016/j.apenergy.2020.115637
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261920311399
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2020.115637?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Wang, Xuan & Shu, Gequn & Tian, Hua & Wang, Rui & Cai, Jinwen, 2020. "Dynamic performance comparison of different cascade waste heat recovery systems for internal combustion engine in combined cooling, heating and power," Applied Energy, Elsevier, vol. 260(C).
    2. Wieczorek, Maciej & Lewandowski, Mirosław, 2017. "A mathematical representation of an energy management strategy for hybrid energy storage system in electric vehicle and real time optimization using a genetic algorithm," Applied Energy, Elsevier, vol. 192(C), pages 222-233.
    3. Xuan Wang & Hua Tian & Gequn Shu, 2016. "Part-Load Performance Prediction and Operation Strategy Design of Organic Rankine Cycles with a Medium Cycle Used for Recovering Waste Heat from Gaseous Fuel Engines," Energies, MDPI, vol. 9(7), pages 1-21, July.
    4. Quoilin, Sylvain & Aumann, Richard & Grill, Andreas & Schuster, Andreas & Lemort, Vincent & Spliethoff, Hartmut, 2011. "Dynamic modeling and optimal control strategy of waste heat recovery Organic Rankine Cycles," Applied Energy, Elsevier, vol. 88(6), pages 2183-2190, June.
    5. Abedin, M.J. & Masjuki, H.H. & Kalam, M.A. & Sanjid, A. & Rahman, S.M. Ashrafur & Masum, B.M., 2013. "Energy balance of internal combustion engines using alternative fuels," Renewable and Sustainable Energy Reviews, Elsevier, vol. 26(C), pages 20-33.
    6. Xiong, Rui & Cao, Jiayi & Yu, Quanqing, 2018. "Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle," Applied Energy, Elsevier, vol. 211(C), pages 538-548.
    7. Han, Xuefeng & He, Hongwen & Wu, Jingda & Peng, Jiankun & Li, Yuecheng, 2019. "Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle," Applied Energy, Elsevier, vol. 254(C).
    8. Ma, Xiaonan & Shu, Gequn & Tian, Hua & Xu, Wen & Chen, Tianyu, 2019. "Performance assessment of engine exhaust-based segmented thermoelectric generators by length ratio optimization," Applied Energy, Elsevier, vol. 248(C), pages 614-625.
    9. Xu, Bin & Rathod, Dhruvang & Yebi, Adamu & Filipi, Zoran & Onori, Simona & Hoffman, Mark, 2019. "A comprehensive review of organic rankine cycle waste heat recovery systems in heavy-duty diesel engine applications," Renewable and Sustainable Energy Reviews, Elsevier, vol. 107(C), pages 145-170.
    10. Rech, Sergio & Zandarin, Simone & Lazzaretto, Andrea & Frangopoulos, Christos A., 2017. "Design and off-design models of single and two-stage ORC systems on board a LNG carrier for the search of the optimal performance and control strategy," Applied Energy, Elsevier, vol. 204(C), pages 221-241.
    11. David Silver & Julian Schrittwieser & Karen Simonyan & Ioannis Antonoglou & Aja Huang & Arthur Guez & Thomas Hubert & Lucas Baker & Matthew Lai & Adrian Bolton & Yutian Chen & Timothy Lillicrap & Fan , 2017. "Mastering the game of Go without human knowledge," Nature, Nature, vol. 550(7676), pages 354-359, October.
    12. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    13. Wu, Xialai & Chen, Junghui & Xie, Lei, 2019. "Fast economic nonlinear model predictive control strategy of Organic Rankine Cycle for waste heat recovery: Simulation-based studies," Energy, Elsevier, vol. 180(C), pages 520-534.
    14. Zhang, H.G. & Wang, E.H. & Fan, B.Y., 2013. "A performance analysis of a novel system of a dual loop bottoming organic Rankine cycle (ORC) with a light-duty diesel engine," Applied Energy, Elsevier, vol. 102(C), pages 1504-1513.
    15. Shu, Gequn & Li, Xiaoning & Tian, Hua & Liang, Xingyu & Wei, Haiqiao & Wang, Xu, 2014. "Alkanes as working fluids for high-temperature exhaust heat recovery of diesel engine using organic Rankine cycle," Applied Energy, Elsevier, vol. 119(C), pages 204-217.
    16. Rathod, Dhruvang & Xu, Bin & Filipi, Zoran & Hoffman, Mark, 2019. "An experimentally validated, energy focused, optimal control strategy for an Organic Rankine Cycle waste heat recovery system," Applied Energy, Elsevier, vol. 256(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lin, Runze & Luo, Yangyang & Wu, Xialai & Chen, Junghui & Huang, Biao & Su, Hongye & Xie, Lei, 2024. "Surrogate empowered Sim2Real transfer of deep reinforcement learning for ORC superheat control," Applied Energy, Elsevier, vol. 356(C).
    2. Zhang, Xuanang & Wang, Xuan & Cai, Jinwen & He, Zhaoxian & Tian, Hua & Shu, Gequn & Shi, Lingfeng, 2022. "Experimental study on operating parameters matching characteristic of the organic Rankine cycle for engine waste heat recovery," Energy, Elsevier, vol. 244(PA).
    3. Zhang, Xuanang & Wang, Xuan & Cai, Jinwen & Wang, Rui & Bian, Xingyan & He, Zhaoxian & Tian, Hua & Shu, Gequn, 2023. "Operation strategy of a multi-mode Organic Rankine cycle system for waste heat recovery from engine cooling water," Energy, Elsevier, vol. 263(PE).
    4. Li, Xiaoya & Xu, Bin & Tian, Hua & Shu, Gequn, 2021. "Towards a novel holistic design of organic Rankine cycle (ORC) systems operating under heat source fluctuations and intermittency," Renewable and Sustainable Energy Reviews, Elsevier, vol. 147(C).
    5. Shi, Yao & Zhang, Zhiming & Xie, Lei & Wu, Xialai & Liu, Xueqin Amy & Lu, Shan & Su, Hongye, 2022. "Modified hierarchical strategy for transient performance improvement of the ORC based waste heat recovery system," Energy, Elsevier, vol. 261(PA).
    6. Ping, Xu & Yang, Fubin & Zhang, Hongguang & Xing, Chengda & Yao, Baofeng & Wang, Yan, 2022. "An outlier removal and feature dimensionality reduction framework with unsupervised learning and information theory intervention for organic Rankine cycle (ORC)," Energy, Elsevier, vol. 254(PB).
    7. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    8. Seppo Sierla & Heikki Ihasalo & Valeriy Vyatkin, 2022. "A Review of Reinforcement Learning Applications to Control of Heating, Ventilation and Air Conditioning Systems," Energies, MDPI, vol. 15(10), pages 1-25, May.
    9. Dong, Wenhui & Cao, Zezhou & Zhao, Pengchong & Yang, Zhenbiao & Yuan, Yichen & Zhao, Ziwen & Chen, Diyi & Wu, Yajun & Xu, Beibei & Venkateshkumar, M., 2023. "A segmented optimal PID method to consider both regulation performance and damping characteristic of hydroelectric power system," Renewable Energy, Elsevier, vol. 207(C), pages 1-12.
    10. Attila R. Imre & Sindu Daniarta & Przemysław Błasiak & Piotr Kolasiński, 2023. "Design, Integration, and Control of Organic Rankine Cycles with Thermal Energy Storage and Two-Phase Expansion System Utilizing Intermittent and Fluctuating Heat Sources—A Review," Energies, MDPI, vol. 16(16), pages 1-25, August.
    11. Daniarta, Sindu & Nemś, Magdalena & Kolasiński, Piotr, 2023. "A review on thermal energy storage applicable for low- and medium-temperature organic Rankine cycle," Energy, Elsevier, vol. 278(PA).
    12. Fan, Chengcheng & Wu, Zhe & Wang, Jiadian & Chen, Yongping & Zhang, Chengbin, 2023. "Thermodynamic process control of ocean thermal energy conversion," Renewable Energy, Elsevier, vol. 210(C), pages 810-821.
    13. Yin, Linfei & Lu, Yuejiang, 2021. "Expandable deep width learning for voltage control of three-state energy model based smart grids containing flexible energy sources," Energy, Elsevier, vol. 226(C).
    14. Johannes Petrus Bester & Martin Van Eldik & Philip van Zyl Venter, 2023. "Energy Recovery Maximisation Modelling Subject to Constrained Cooling," Energies, MDPI, vol. 17(1), pages 1-23, December.
    15. Hou, Guolian & Huang, Ting & Zheng, Fumeng & Huang, Congzhi, 2024. "A hierarchical reinforcement learning GPC for flexible operation of ultra-supercritical unit considering economy," Energy, Elsevier, vol. 289(C).
    16. Chen, Ruihua & Deng, Shuai & Zhao, Li & Zhao, Ruikai & Xu, Weicong, 2022. "Energy recovery from wastewater in deep-sea mining: Feasibility study on an energy supply solution with cold wastewater," Applied Energy, Elsevier, vol. 305(C).
    17. Miao, Zheng & Yan, Peiwei & Xiao, Meng & Zhang, Manzheng & Xu, Jinliang, 2023. "Comparative study on operating strategies of the organic Rankine cycle under transient heat source," Energy, Elsevier, vol. 285(C).
    18. Chen, Minghao & Xie, Zhiyuan & Sun, Yi & Zheng, Shunlin, 2023. "The predictive management in campus heating system based on deep reinforcement learning and probabilistic heat demands forecasting," Applied Energy, Elsevier, vol. 350(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jiménez-Arreola, Manuel & Wieland, Christoph & Romagnoli, Alessandro, 2019. "Direct vs indirect evaporation in Organic Rankine Cycle (ORC) systems: A comparison of the dynamic behavior for waste heat recovery of engine exhaust," Applied Energy, Elsevier, vol. 242(C), pages 439-452.
    2. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    3. Imran, Muhammad & Pili, Roberto & Usman, Muhammad & Haglind, Fredrik, 2020. "Dynamic modeling and control strategies of organic Rankine cycle systems: Methods and challenges," Applied Energy, Elsevier, vol. 276(C).
    4. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    5. Xu, Bin & Rathod, Dhruvang & Zhang, Darui & Yebi, Adamu & Zhang, Xueyu & Li, Xiaoya & Filipi, Zoran, 2020. "Parametric study on reinforcement learning optimized energy management strategy for a hybrid electric vehicle," Applied Energy, Elsevier, vol. 259(C).
    6. Hoang, Anh Tuan, 2018. "Waste heat recovery from diesel engines based on Organic Rankine Cycle," Applied Energy, Elsevier, vol. 231(C), pages 138-166.
    7. Ouyang, Tiancheng & Su, Zixiang & Yang, Rui & Wang, Zhiping & Mo, Xiaoyu & Huang, Haozhong, 2021. "Advanced waste heat harvesting strategy for marine dual-fuel engine considering gas-liquid two-phase flow of turbine," Energy, Elsevier, vol. 224(C).
    8. Shi, Yao & Zhang, Zhiming & Xie, Lei & Wu, Xialai & Liu, Xueqin Amy & Lu, Shan & Su, Hongye, 2022. "Modified hierarchical strategy for transient performance improvement of the ORC based waste heat recovery system," Energy, Elsevier, vol. 261(PA).
    9. Li, Xiaoya & Xu, Bin & Tian, Hua & Shu, Gequn, 2021. "Towards a novel holistic design of organic Rankine cycle (ORC) systems operating under heat source fluctuations and intermittency," Renewable and Sustainable Energy Reviews, Elsevier, vol. 147(C).
    10. Surendran, Anandu & Seshadri, Satyanarayanan, 2020. "Design and performance analysis of a novel Transcritical Regenerative Series Two stage Organic Rankine Cycle for dual source waste heat recovery," Energy, Elsevier, vol. 203(C).
    11. Shi, Yao & Lin, Runze & Wu, Xialai & Zhang, Zhiming & Sun, Pei & Xie, Lei & Su, Hongye, 2022. "Dual-mode fast DMC algorithm for the control of ORC based waste heat recovery system," Energy, Elsevier, vol. 244(PA).
    12. Vaupel, Yannic & Huster, Wolfgang R. & Mhamdi, Adel & Mitsos, Alexander, 2021. "Optimal operating policies for organic Rankine cycles for waste heat recovery under transient conditions," Energy, Elsevier, vol. 224(C).
    13. Xu, Bin & Li, Xiaoya, 2021. "A Q-learning based transient power optimization method for organic Rankine cycle waste heat recovery system in heavy duty diesel engine applications," Applied Energy, Elsevier, vol. 286(C).
    14. Nyong-Bassey, Bassey Etim & Giaouris, Damian & Patsios, Charalampos & Papadopoulou, Simira & Papadopoulos, Athanasios I. & Walker, Sara & Voutetakis, Spyros & Seferlis, Panos & Gadoue, Shady, 2020. "Reinforcement learning based adaptive power pinch analysis for energy management of stand-alone hybrid energy storage systems considering uncertainty," Energy, Elsevier, vol. 193(C).
    15. Pili, Roberto & Romagnoli, Alessandro & Jiménez-Arreola, Manuel & Spliethoff, Hartmut & Wieland, Christoph, 2019. "Simulation of Organic Rankine Cycle – Quasi-steady state vs dynamic approach for optimal economic performance," Energy, Elsevier, vol. 167(C), pages 619-640.
    16. Yang, Ting & Zhao, Liyuan & Li, Wei & Zomaya, Albert Y., 2021. "Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning," Energy, Elsevier, vol. 235(C).
    17. Zhu, Tao & Wills, Richard G.A. & Lot, Roberto & Ruan, Haijun & Jiang, Zhihao, 2021. "Adaptive energy management of a battery-supercapacitor energy storage system for electric vehicles based on flexible perception and neural network fitting," Applied Energy, Elsevier, vol. 292(C).
    18. Zhang, Wei & Wang, Jixin & Liu, Yong & Gao, Guangzong & Liang, Siwen & Ma, Hongfeng, 2020. "Reinforcement learning-based intelligent energy management architecture for hybrid construction machinery," Applied Energy, Elsevier, vol. 275(C).
    19. Shi, Yao & Zhang, Zhiming & Chen, Xiaoqiang & Xie, Lei & Liu, Xueqin & Su, Hongye, 2023. "Data-Driven model identification and efficient MPC via quasi-linear parameter varying representation for ORC waste heat recovery system," Energy, Elsevier, vol. 271(C).
    20. Gu, Zhengzhao & Feng, Kewen & Ge, Lei & Quan, Long, 2023. "Dynamic modeling and optimization of organic Rankine cycle in the waste heat recovery of the hydraulic system," Energy, Elsevier, vol. 263(PB).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:278:y:2020:i:c:s0306261920311399. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.