IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v353y2024ipas0306261923013934.html
   My bibliography  Save this article

A novel multi-objective optimization based multi-agent deep reinforcement learning approach for microgrid resources planning

Author

Listed:
  • Abid, Md. Shadman
  • Apon, Hasan Jamil
  • Hossain, Salman
  • Ahmed, Ashik
  • Ahshan, Razzaqul
  • Lipu, M.S. Hossain

Abstract

Multi-agent deep reinforcement learning (MADRL) approaches are at the forefront of contemporary research in optimum electric vehicle (EV) charging scheduling challenges. These techniques involve multiple agents that respond to a dynamic simulation environment to strategically integrate EV charging stations (EVCSs) on microgrids by incorporating the constraints posed by stochastic trip durations. In addition, recent research works have demonstrated that planning frameworks based on multi-objective optimization (MOO) techniques are suitable for the efficient functioning of microgrids comprising renewable energy sources (RESs) and battery energy storage systems (BESSs). Even though MADRL techniques have been used to solve the optimum EV charging scheduling challenges and MOO frameworks have been developed to determine the optimal RES-BESS allocation, the potential of merging MADRL and MOO is yet to be explored. Therefore, this research provides an opportunity to determine the effectiveness of combined MOO-MADRL dynamics and their computational efficacy. In this context, this work presents a novel Multi-objective Artificial Vultures Optimization Algorithm based on Multi-agent Deep Deterministic Policy Gradient (MOAVOA-MADDPG) planning framework for allocating RESs, BESSs, and EVCSs on microgrids. The objective function is formulated to optimize the network power losses, total installation and operational costs, greenhouse gas emissions, and system voltage stability. Moreover, the proposed framework incorporates the sporadic nature of RES systems and intends to improve the state of charge (SOC) of the EVs present in the network. The presented approach is validated using practical weather data and EV commuting behavior on the modified IEEE 33 bus network, two practical distribution feeders in Bangladesh, and the Turkish 141 bus network. According to the findings, the MOAVOA-MADDPG framework effectively accommodated the financial, technical, and environmental considerations with improved average SOC of the vehicles. Furthermore, statistical analysis, spacing, convergence, and hyper-volume metrics are employed to compare the suggested MOAVOA-MADDPG framework with five contemporary techniques. The findings indicate that, in every metric considered, the MOAVOA-MADDPG Pareto fronts provide superior solutions.

Suggested Citation

  • Abid, Md. Shadman & Apon, Hasan Jamil & Hossain, Salman & Ahmed, Ashik & Ahshan, Razzaqul & Lipu, M.S. Hossain, 2024. "A novel multi-objective optimization based multi-agent deep reinforcement learning approach for microgrid resources planning," Applied Energy, Elsevier, vol. 353(PA).
    • Handle: RePEc:eee:appene:v:353:y:2024:i:pa:s0306261923013934
    DOI: 10.1016/j.apenergy.2023.122029
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261923013934
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2023.122029?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Mohamed, Ahmed A.S. & Shaier, Ahmed A. & Metwally, Hamid & Selem, Sameh I., 2020. "A comprehensive overview of inductive pad in electric vehicles stationary charging," Applied Energy, Elsevier, vol. 262(C).
    2. Ahmadi, Bahman & Ceylan, Oguzhan & Ozdemir, Aydogan & Fotuhi-Firuzabad, Mahmoud, 2022. "A multi-objective framework for distributed energy resources planning and storage management," Applied Energy, Elsevier, vol. 314(C).
    3. Annan, Aaron M. & Lackner, Matthew A. & Manwell, James F., 2023. "Multi-objective optimization for an autonomous unmoored offshore wind energy system substructure," Applied Energy, Elsevier, vol. 344(C).
    4. Chitchai Srithapon & Prasanta Ghosh & Apirat Siritaratiwat & Rongrit Chatthaworn, 2020. "Optimization of Electric Vehicle Charging Scheduling in Urban Village Networks Considering Energy Arbitrage and Distribution Cost," Energies, MDPI, vol. 13(2), pages 1-20, January.
    5. Zhou, Guangyou & Zhu, Zhiwei & Luo, Sumei, 2022. "Location optimization of electric vehicle charging stations: Based on cost model and genetic algorithm," Energy, Elsevier, vol. 247(C).
    6. O’Dwyer, Edward & Pan, Indranil & Acha, Salvador & Shah, Nilay, 2019. "Smart energy systems for sustainable smart cities: Current developments, trends and future directions," Applied Energy, Elsevier, vol. 237(C), pages 581-597.
    7. Duan, Ditao & Poursoleiman, Roza, 2021. "Modified teaching-learning-based optimization by orthogonal learning for optimal design of an electric vehicle charging station," Utilities Policy, Elsevier, vol. 72(C).
    8. Dorokhova, Marina & Martinson, Yann & Ballif, Christophe & Wyrsch, Nicolas, 2021. "Deep reinforcement learning control of electric vehicle charging in the presence of photovoltaic generation," Applied Energy, Elsevier, vol. 301(C).
    9. Li, Jiawen, 2022. "A multi-objective energy coordinative and management policy for solid oxide fuel cell using triune brain large-scale multi-agent deep deterministic policy gradient," Applied Energy, Elsevier, vol. 324(C).
    10. Luo, Lizi & Gu, Wei & Zhou, Suyang & Huang, He & Gao, Song & Han, Jun & Wu, Zhi & Dou, Xiaobo, 2018. "Optimal planning of electric vehicle charging stations comprising multi-types of charging facilities," Applied Energy, Elsevier, vol. 226(C), pages 1087-1099.
    11. Costa, Alberto & Ng, Tsan Sheng & Su, Bin, 2023. "Long-term solar PV planning: An economic-driven robust optimization approach," Applied Energy, Elsevier, vol. 335(C).
    12. Das, Choton K. & Bass, Octavian & Kothapalli, Ganesh & Mahmoud, Thair S. & Habibi, Daryoush, 2018. "Optimal placement of distributed energy storage systems in distribution networks using artificial bee colony algorithm," Applied Energy, Elsevier, vol. 232(C), pages 212-228.
    13. Mejia, Cristian & Kajikawa, Yuya, 2020. "Emerging topics in energy storage based on a large-scale analysis of academic articles and patents," Applied Energy, Elsevier, vol. 263(C).
    14. Fathy, Ahmed, 2022. "A novel artificial hummingbird algorithm for integrating renewable based biomass distributed generators in radial distribution systems," Applied Energy, Elsevier, vol. 323(C).
    15. Woo, Hyeon & Son, Yongju & Cho, Jintae & Kim, Sung-Yul & Choi, Sungyun, 2023. "Optimal expansion planning of electric vehicle fast charging stations," Applied Energy, Elsevier, vol. 342(C).
    16. Qiu, Dawei & Wang, Yi & Hua, Weiqi & Strbac, Goran, 2023. "Reinforcement learning for electric vehicle applications in power systems:A critical review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 173(C).
    17. Yao, Zhaosheng & Wang, Zhiyuan & Ran, Lun, 2023. "Smart charging and discharging of electric vehicles based on multi-objective robust optimization in smart cities," Applied Energy, Elsevier, vol. 343(C).
    18. Park, Keonwoo & Moon, Ilkyeong, 2022. "Multi-agent deep reinforcement learning approach for EV charging scheduling in a smart grid," Applied Energy, Elsevier, vol. 328(C).
    19. Arias-Rosales, Andrés & Mejía-Gutiérrez, Ricardo, 2018. "Optimization of V-Trough photovoltaic concentrators through genetic algorithms with heuristics based on Weibull distributions," Applied Energy, Elsevier, vol. 212(C), pages 122-140.
    20. Le, Tay Son & Nguyen, Tuan Ngoc & Bui, Dac-Khuong & Ngo, Tuan Duc, 2023. "Optimal sizing of renewable energy storage: A techno-economic analysis of hydrogen, battery and hybrid systems considering degradation and seasonal storage," Applied Energy, Elsevier, vol. 336(C).
    21. Mu, Yunfei & Wu, Jianzhong & Jenkins, Nick & Jia, Hongjie & Wang, Chengshan, 2014. "A Spatial–Temporal model for grid impact analysis of plug-in electric vehicles," Applied Energy, Elsevier, vol. 114(C), pages 456-465.
    22. Jia Liu & Jin Huang & Jinzhi Hu, 2022. "Multi-objective optimisation method of electric vehicle charging station based on non-dominated sorting genetic algorithm," International Journal of Global Energy Issues, Inderscience Enterprises Ltd, vol. 44(5/6), pages 413-426.
    23. Charbonnier, Flora & Morstyn, Thomas & McCulloch, Malcolm D., 2022. "Coordination of resources at the edge of the electricity grid: Systematic review and taxonomy," Applied Energy, Elsevier, vol. 318(C).
    24. Xu, Bin & Li, Xiaoya, 2021. "A Q-learning based transient power optimization method for organic Rankine cycle waste heat recovery system in heavy duty diesel engine applications," Applied Energy, Elsevier, vol. 286(C).
    25. Tuchnitz, Felix & Ebell, Niklas & Schlund, Jonas & Pruckner, Marco, 2021. "Development and Evaluation of a Smart Charging Strategy for an Electric Vehicle Fleet Based on Reinforcement Learning," Applied Energy, Elsevier, vol. 285(C).
    26. Mukhopadhyay, Bineeta & Das, Debapriya, 2020. "Multi-objective dynamic and static reconfiguration with optimized allocation of PV-DG and battery energy storage system," Renewable and Sustainable Energy Reviews, Elsevier, vol. 124(C).
    27. Abdullah-Al-Nahid, Syed & Jamal, Taskin & Aziz, Tareq & Bhuiyan, Ashraf Hossain & Khan, Tafsir Ahmed, 2023. "Additive linear modelling and genetic algorithm based electric vehicle outlook and policy formulation for decarbonizing the future transport sector of Bangladesh," Transport Policy, Elsevier, vol. 136(C), pages 21-46.
    28. Carbajo, Ruth & Cabeza, Luisa F., 2018. "Renewable energy research and technologies through responsible research and innovation looking glass: Reflexions, theoretical approaches and contemporary discourses," Applied Energy, Elsevier, vol. 211(C), pages 792-808.
    29. Adetunji, Kayode E. & Hofsajer, Ivan W. & Abu-Mahfouz, Adnan M. & Cheng, Ling, 2022. "An optimization planning framework for allocating multiple distributed energy resources and electric vehicle charging stations in distribution networks," Applied Energy, Elsevier, vol. 322(C).
    30. Brinkel, N.B.G. & Schram, W.L. & AlSkaif, T.A. & Lampropoulos, I. & van Sark, W.G.J.H.M., 2020. "Should we reinforce the grid? Cost and emission optimization of electric vehicle charging under different transformer limits," Applied Energy, Elsevier, vol. 276(C).
    31. Kourkoumpas, Dimitrios-Sotirios & Benekos, Georgios & Nikolopoulos, Nikolaos & Karellas, Sotirios & Grammelis, Panagiotis & Kakaras, Emmanouel, 2018. "A review of key environmental and energy performance indicators for the case of renewable energy systems when integrated with storage solutions," Applied Energy, Elsevier, vol. 231(C), pages 380-398.
    32. Jaguemont, J. & Boulon, L. & Dubé, Y., 2016. "A comprehensive review of lithium-ion batteries used in hybrid and electric vehicles at cold temperatures," Applied Energy, Elsevier, vol. 164(C), pages 99-114.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Abid, Md. Shadman & Ahshan, Razzaqul & Al Abri, Rashid & Al-Badi, Abdullah & Albadi, Mohammed, 2024. "Techno-economic and environmental assessment of renewable energy sources, virtual synchronous generators, and electric vehicle charging stations in microgrids," Applied Energy, Elsevier, vol. 353(PA).
    2. Sami M. Alshareef & Ahmed Fathy, 2023. "Efficient Red Kite Optimization Algorithm for Integrating the Renewable Sources and Electric Vehicle Fast Charging Stations in Radial Distribution Networks," Mathematics, MDPI, vol. 11(15), pages 1-30, July.
    3. Julia Vopava & Christian Koczwara & Anna Traupmann & Thomas Kienberger, 2019. "Investigating the Impact of E-Mobility on the Electrical Power Grid Using a Simplified Grid Modelling Approach," Energies, MDPI, vol. 13(1), pages 1-23, December.
    4. Hamza El Hafdaoui & Hamza El Alaoui & Salma Mahidat & Zakaria El Harmouzi & Ahmed Khallaayoun, 2023. "Impact of Hot Arid Climate on Optimal Placement of Electric Vehicle Charging Stations," Energies, MDPI, vol. 16(2), pages 1-19, January.
    5. Li, Yanbin & Wang, Jiani & Wang, Weiye & Liu, Chang & Li, Yun, 2023. "Dynamic pricing based electric vehicle charging station location strategy using reinforcement learning," Energy, Elsevier, vol. 281(C).
    6. Luo, Lizi & Gu, Wei & Wu, Zhi & Zhou, Suyang, 2019. "Joint planning of distributed generation and electric vehicle charging stations considering real-time charging navigation," Applied Energy, Elsevier, vol. 242(C), pages 1274-1284.
    7. Loni, Abdolah & Asadi, Somayeh, 2023. "Data-driven equitable placement for electric vehicle charging stations: Case study San Francisco," Energy, Elsevier, vol. 282(C).
    8. Powell, Siobhan & Vianna Cezar, Gustavo & Apostolaki-Iosifidou, Elpiniki & Rajagopal, Ram, 2022. "Large-scale scenarios of electric vehicle charging with a data-driven model of control," Energy, Elsevier, vol. 248(C).
    9. Yingyue Li & Hongjun Li & Rui Miao & He Qi & Yi Zhang, 2023. "Energy–Environment–Economy (3E) Analysis of the Performance of Introducing Photovoltaic and Energy Storage Systems into Residential Buildings: A Case Study in Shenzhen, China," Sustainability, MDPI, vol. 15(11), pages 1-25, June.
    10. Kumar, Abhishek & Deng, Yan & He, Xiangning & Singh, Arvind R. & Kumar, Praveen & Bansal, R.C. & Bettayeb, M. & Ghenai, C. & Naidoo, R.M., 2023. "Impact of demand side management approaches for the enhancement of voltage stability loadability and customer satisfaction index," Applied Energy, Elsevier, vol. 339(C).
    11. Zhao, Zhonghao & Lee, Carman K.M. & Ren, Jingzheng, 2024. "A two-level charging scheduling method for public electric vehicle charging stations considering heterogeneous demand and nonlinear charging profile," Applied Energy, Elsevier, vol. 355(C).
    12. Niphon Kaewdornhan & Chitchai Srithapon & Rittichai Liemthong & Rongrit Chatthaworn, 2023. "Real-Time Multi-Home Energy Management with EV Charging Scheduling Using Multi-Agent Deep Reinforcement Learning Optimization," Energies, MDPI, vol. 16(5), pages 1-25, March.
    13. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    14. Zhang, Tianren & Huang, Yuping & Liao, Hui & Liang, Yu, 2023. "A hybrid electric vehicle load classification and forecasting approach based on GBDT algorithm and temporal convolutional network," Applied Energy, Elsevier, vol. 351(C).
    15. Panyawoot Boonluk & Sirote Khunkitti & Pradit Fuangfoo & Apirat Siritaratiwat, 2021. "Optimal Siting and Sizing of Battery Energy Storage: Case Study Seventh Feeder at Nakhon Phanom Substation in Thailand," Energies, MDPI, vol. 14(5), pages 1-20, March.
    16. Li, Xiaohui & Wang, Zhenpo & Zhang, Lei & Sun, Fengchun & Cui, Dingsong & Hecht, Christopher & Figgener, Jan & Sauer, Dirk Uwe, 2023. "Electric vehicle behavior modeling and applications in vehicle-grid integration: An overview," Energy, Elsevier, vol. 268(C).
    17. Dimitrios Vamvakas & Panagiotis Michailidis & Christos Korkas & Elias Kosmatopoulos, 2023. "Review and Evaluation of Reinforcement Learning Frameworks on Smart Grid Applications," Energies, MDPI, vol. 16(14), pages 1-38, July.
    18. Amjad, Muhammad & Farooq-i-Azam, Muhammad & Ni, Qiang & Dong, Mianxiong & Ansari, Ejaz Ahmad, 2022. "Wireless charging systems for electric vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 167(C).
    19. Ferreira, Paula & Rocha, Ana & Araujo, Madalena & Afonso, Joao L. & Antunes, Carlos Henggeler & Lopes, Marta A.R. & Osório, Gerardo J. & Catalão, João P.S. & Lopes, João Peças, 2023. "Assessing the societal impact of smart grids: Outcomes of a collaborative research project," Technology in Society, Elsevier, vol. 72(C).
    20. Deveci, Muhammet & Erdogan, Nuh & Pamucar, Dragan & Kucuksari, Sadik & Cali, Umit, 2023. "A rough Dombi Bonferroni based approach for public charging station type selection," Applied Energy, Elsevier, vol. 345(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:353:y:2024:i:pa:s0306261923013934. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.