IDEAS home Printed from https://ideas.repec.org/a/spr/waterr/v34y2020i11d10.1007_s11269-020-02566-9.html
   My bibliography  Save this article

Multidimensional Parallel Dynamic Programming Algorithm Based on Spark for Large-Scale Hydropower Systems

Author

Listed:
  • Yufei Ma

    (Hohai University)

  • Ping-an Zhong

    (Hohai University)

  • Bin Xu

    (Hohai University
    Nanjing Hydraulic Research Institute)

  • Feilin Zhu

    (Hohai University)

  • Yao Xiao

    (Hohai University)

  • Qingwen Lu

    (Hohai University)

Abstract

The “curse of dimensionality” is a major problem in dynamic programming (DP) algorithms for large-scale hydropower systems. This study proposes a parallel DP algorithm based on Spark (PDPoS) to alleviate the “curse of dimensionality”. Parallel computing experiments are formulated by varying the number of reservoirs, the number of discrete water levels and the number of CPU cores to analyze the quality and efficiency of PDPoS. The methodologies were applied to a cascade reservoir system made up of eight reservoirs in the Yuanshui River Basin in China. The results are as follows. (1) The number of discrete water levels is the dominant factor in the solution quality, while the number of reservoirs is the dominant factor in the solving efficiency. (2) The runtime of PDPoS is markedly affected by the calculational scale (determined by the number of reservoirs and discrete water levels), and the relationship between the number of CPU cores and the runtime is triphasic with increasing calculational scale. (3) The larger the calculational scale is, the better the parallel performance (i.e., the parallel speedup and parallel efficiency). The proposed PDPoS method has strong generality, high parallel performance, and high practical value.

Suggested Citation

  • Yufei Ma & Ping-an Zhong & Bin Xu & Feilin Zhu & Yao Xiao & Qingwen Lu, 2020. "Multidimensional Parallel Dynamic Programming Algorithm Based on Spark for Large-Scale Hydropower Systems," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 34(11), pages 3427-3444, September.
    • Handle: RePEc:spr:waterr:v:34:y:2020:i:11:d:10.1007_s11269-020-02566-9
    DOI: 10.1007/s11269-020-02566-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11269-020-02566-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11269-020-02566-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Mohammad Ehteram & Hojat Karami & Saeed Farzin, 2018. "Reservoir Optimization for Energy Production Using a New Evolutionary Algorithm Based on Multi-Criteria Decision-Making Models," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 32(7), pages 2539-2560, May.
    2. Feng, Zhong-kai & Niu, Wen-jing & Cheng, Chun-tian & Liao, Sheng-li, 2017. "Hydropower system operation optimization by discrete differential dynamic programming based on orthogonal experiment design," Energy, Elsevier, vol. 126(C), pages 720-732.
    3. Sheng-li Liao & Ben-xi Liu & Chun-tian Cheng & Zhi-fu Li & Xin-yu Wu, 2017. "Long-Term Generation Scheduling of Hydropower System Using Multi-Core Parallelization of Particle Swarm Optimization," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 31(9), pages 2791-2807, July.
    4. Ping Sun & Zhi-qiang Jiang & Ting-ting Wang & Yan-ke Zhang, 2016. "Research and Application of Parallel Normal Cloud Mutation Shuffled Frog Leaping Algorithm in Cascade Reservoirs Optimal Operation," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 30(3), pages 1019-1035, February.
    5. Xu, Bin & Zhu, Feilin & Zhong, Ping-an & Chen, Juan & Liu, Weifeng & Ma, Yufei & Guo, Le & Deng, Xiaoliang, 2019. "Identifying long-term effects of using hydropower to complement wind power uncertainty through stochastic programming," Applied Energy, Elsevier, vol. 253(C), pages 1-1.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Fang, Zhou & Liao, Shengli & Cheng, Chuntian & Zhao, Hongye & Liu, Benxi & Su, Huaying, 2023. "Parallel improved DPSA algorithm for medium-term optimal scheduling of large-scale cascade hydropower plants," Renewable Energy, Elsevier, vol. 210(C), pages 134-147.
    2. Shengli Liao & Yan Zhang & Jie Liu & Benxi Liu & Zhanwei Liu, 2021. "Short-Term Peak-Shaving Operation of Single-Reservoir and Multicascade Hydropower Plants Serving Multiple Power Grids," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 35(2), pages 689-705, January.
    3. Schäffer, Linn Emelie & Helseth, Arild & Korpås, Magnus, 2022. "A stochastic dynamic programming model for hydropower scheduling with state-dependent maximum discharge constraints," Renewable Energy, Elsevier, vol. 194(C), pages 571-581.
    4. Wen-jing Niu & Zhong-kai Feng & Yu-rong Li & Shuai Liu, 2021. "Cooperation Search Algorithm for Power Generation Production Operation Optimization of Cascade Hydropower Reservoirs," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 35(8), pages 2465-2485, June.
    5. Yufei Ma & Ping-an Zhong & Bin Xu & Feilin Zhu & Jieyu Li & Han Wang & Qingwen Lu, 2021. "Cloud-Based Multidimensional Parallel Dynamic Programming Algorithm for a Cascade Hydropower System," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 35(9), pages 2705-2721, July.
    6. Arya Yaghoubzadeh-Bavandpour & Omid Bozorg-Haddad & Mohammadreza Rajabi & Babak Zolghadr-Asli & Xuefeng Chu, 2022. "Application of Swarm Intelligence and Evolutionary Computation Algorithms for Optimal Reservoir Operation," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 36(7), pages 2275-2292, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Feng, Zhong-kai & Niu, Wen-jing & Cheng, Chun-tian, 2018. "Optimization of hydropower reservoirs operation balancing generation benefit and ecological requirement with parallel multi-objective genetic algorithm," Energy, Elsevier, vol. 153(C), pages 706-718.
    2. Jin, Xiaoyu & Liu, Benxi & Liao, Shengli & Cheng, Chuntian & Yan, Zhiyu, 2022. "A Wasserstein metric-based distributionally robust optimization approach for reliable-economic equilibrium operation of hydro-wind-solar energy systems," Renewable Energy, Elsevier, vol. 196(C), pages 204-219.
    3. Jin, Xiaoyu & Liu, Benxi & Liao, Shengli & Cheng, Chuntian & Li, Gang & Liu, Lingjun, 2022. "Impacts of different wind and solar power penetrations on cascade hydroplants operation," Renewable Energy, Elsevier, vol. 182(C), pages 227-244.
    4. Yang, Zhe & Wang, Yufeng & Yang, Kan, 2022. "The stochastic short-term hydropower generation scheduling considering uncertainty in load output forecasts," Energy, Elsevier, vol. 241(C).
    5. Feng, Zhong-kai & Niu, Wen-jing & Wang, Wen-chuan & Zhou, Jian-zhong & Cheng, Chun-tian, 2019. "A mixed integer linear programming model for unit commitment of thermal plants with peak shaving operation aspect in regional power grid lack of flexible hydropower energy," Energy, Elsevier, vol. 175(C), pages 618-629.
    6. Yuhong Wang & Lei Chen & Hong Zhou & Xu Zhou & Zongsheng Zheng & Qi Zeng & Li Jiang & Liang Lu, 2021. "Flexible Transmission Network Expansion Planning Based on DQN Algorithm," Energies, MDPI, vol. 14(7), pages 1-21, April.
    7. Zhiqiang Jiang & Zhengyang Tang & Yi Liu & Yuyun Chen & Zhongkai Feng & Yang Xu & Hairong Zhang, 2019. "Area Moment and Error Based Forecasting Difficulty and its Application in Inflow Forecasting Level Evaluation," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 33(13), pages 4553-4568, October.
    8. Hu Hu & Kan Yang & Lyuwen Su & Zhe Yang, 2019. "A Novel Adaptive Multi-Objective Particle Swarm Optimization Based on Decomposition and Dominance for Long-term Generation Scheduling of Cascade Hydropower System," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 33(11), pages 4007-4026, September.
    9. Jin, Xiaoyu & Liu, Benxi & Liao, Shengli & Cheng, Chuntian & Zhang, Yi & Zhao, Zhipeng & Lu, Jia, 2022. "Wasserstein metric-based two-stage distributionally robust optimization model for optimal daily peak shaving dispatch of cascade hydroplants under renewable energy uncertainties," Energy, Elsevier, vol. 260(C).
    10. Jiang, Zhiqiang & Li, Anqiang & Ji, Changming & Qin, Hui & Yu, Shan & Li, Yuanzheng, 2016. "Research and application of key technologies in drawing energy storage operation chart by discriminant coefficient method," Energy, Elsevier, vol. 114(C), pages 774-786.
    11. Majid Mohammadi & Saeed Farzin & Sayed-Farhad Mousavi & Hojat Karami, 2019. "Investigation of a New Hybrid Optimization Algorithm Performance in the Optimal Operation of Multi-Reservoir Benchmark Systems," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 33(14), pages 4767-4782, November.
    12. Yang, Yuqi & Zhou, Jianzhong & Liu, Guangbiao & Mo, Li & Wang, Yongqiang & Jia, Benjun & He, Feifei, 2020. "Multi-plan formulation of hydropower generation considering uncertainty of wind power," Applied Energy, Elsevier, vol. 260(C).
    13. Riepin, Iegor & Möbius, Thomas & Müsgens, Felix, 2021. "Modelling uncertainty in coupled electricity and gas systems—Is it worth the effort?," Applied Energy, Elsevier, vol. 285(C).
    14. Qiao-feng Tan & Guo-hua Fang & Xin Wen & Xiao-hui Lei & Xu Wang & Chao Wang & Yi Ji, 2020. "Bayesian Stochastic Dynamic Programming for Hydropower Generation Operation Based on Copula Functions," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 34(5), pages 1589-1607, March.
    15. Jun Dong & Anyuan Fu & Yao Liu & Shilin Nie & Peiwen Yang & Linpeng Nie, 2019. "Two-Stage Optimization Model for Two-Side Daily Reserve Capacity of a Power System Considering Demand Response and Wind Power Consumption," Sustainability, MDPI, vol. 11(24), pages 1-22, December.
    16. Zhe Yang & Kan Yang & Hu Hu & Lyuwen Su, 2019. "The Cascade Reservoirs Multi-Objective Ecological Operation Optimization Considering Different Ecological Flow Demand," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 33(1), pages 207-228, January.
    17. Gong, Yu & Liu, Pan & Ming, Bo & Xu, Weifeng & Huang, Kangdi & Li, Xiao, 2021. "Deriving pack rules for hydro–photovoltaic hybrid power systems considering diminishing marginal benefit of energy," Applied Energy, Elsevier, vol. 304(C).
    18. Changjun Wang & Shutong Chen, 2019. "Planning of Cascade Hydropower Stations with the Consideration of Long-Term Operations under Uncertainties," Complexity, Hindawi, vol. 2019, pages 1-23, November.
    19. Zhang, Yi & Cheng, Chuntian & Yang, Tiantian & Jin, Xiaoyu & Jia, Zebin & Shen, Jianjian & Wu, Xinyu, 2022. "Assessment of climate change impacts on the hydro-wind-solar energy supply system," Renewable and Sustainable Energy Reviews, Elsevier, vol. 162(C).
    20. Niu, Wen-jing & Feng, Zhong-kai & Cheng, Chun-tian, 2018. "Optimization of variable-head hydropower system operation considering power shortage aspect with quadratic programming and successive approximation," Energy, Elsevier, vol. 143(C), pages 1020-1028.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:waterr:v:34:y:2020:i:11:d:10.1007_s11269-020-02566-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.