IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v335y2023ics0306261923000715.html
   My bibliography  Save this article

Design and planning of flexible mobile Micro-Grids using Deep Reinforcement Learning

Author

Listed:
  • Caputo, Cesare
  • Cardin, Michel-Alexandre
  • Ge, Pudong
  • Teng, Fei
  • Korre, Anna
  • Antonio del Rio Chanona, Ehecatl

Abstract

Ongoing risks from climate change have significantly impacted the livelihood of global nomadic communities and are likely to lead to increased migratory movements in coming years. As a result, mobility considerations are becoming increasingly important in energy systems planning, particularly to achieve energy access in developing countries. Advanced “Plug and Play” control strategies have been recently developed with such a decentralized framework in mind, allowing easier interconnection of nomadic communities, both to each other and to the main grid. Considering the above, the design and planning strategy of a mobile multi-energy supply system for a nomadic community is investigated in this work. Motivated by the scale and dimensionality of the associated uncertainties, impacting all major design and decision variables over the 30-year planning horizon, Deep Reinforcement Learning (DRL) Flexibility Analysis is implemented for the design and planning problem. DRL based solutions are benchmarked against several rigid baseline design options to compare expected performance under uncertainty. The results on a case study for ger communities in Mongolia suggest that mobile nomadic energy systems can be both technically and economically feasible, particularly when considering flexibility, although the degree of spatial dispersion among households is an important limiting factor. Additionally, the DRL based policies lead to the development of dynamic evolution and adaptability strategies, which can be used by the targeted communities under a very wide range of potential scenarios. Key economic, sustainability and resilience indicators such as Cost, Equivalent Emissions and Total Unmet Load are measured, suggesting potential improvements compared to available baselines of up to 25%, 67% and 76%, respectively. Finally, the decomposition of values of flexibility and plug and play operation is presented using a variation of real options theory, with important implications for both nomadic communities and policymakers focused on enabling their energy access.

Suggested Citation

  • Caputo, Cesare & Cardin, Michel-Alexandre & Ge, Pudong & Teng, Fei & Korre, Anna & Antonio del Rio Chanona, Ehecatl, 2023. "Design and planning of flexible mobile Micro-Grids using Deep Reinforcement Learning," Applied Energy, Elsevier, vol. 335(C).
    • Handle: RePEc:eee:appene:v:335:y:2023:i:c:s0306261923000715
    DOI: 10.1016/j.apenergy.2023.120707
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261923000715
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2023.120707?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Martínez-Ceseña, E.A. & Mutale, J., 2011. "Application of an advanced real options approach for renewable energy generation projects planning," Renewable and Sustainable Energy Reviews, Elsevier, vol. 15(4), pages 2087-2094, May.
    2. Fernandes, Bartolomeu & Cunha, Jorge & Ferreira, Paula, 2011. "The use of real options approach in energy sector investments," Renewable and Sustainable Energy Reviews, Elsevier, vol. 15(9), pages 4491-4497.
    3. Otsuki, Takashi, 2017. "Costs and benefits of large-scale deployment of wind turbines and solar PV in Mongolia for international power exports," Renewable Energy, Elsevier, vol. 108(C), pages 321-335.
    4. Lammers, Katrin & Bertheau, Paul & Blechinger, Philipp, 2020. "Exploring requirements for sustainable energy supply planning with regard to climate resilience of Southeast Asian islands," Energy Policy, Elsevier, vol. 146(C).
    5. Jonas Degrave & Federico Felici & Jonas Buchli & Michael Neunert & Brendan Tracey & Francesco Carpanese & Timo Ewalds & Roland Hafner & Abbas Abdolmaleki & Diego de las Casas & Craig Donner & Leslie F, 2022. "Magnetic control of tokamak plasmas through deep reinforcement learning," Nature, Nature, vol. 602(7897), pages 414-419, February.
    6. Andriotis, C.P. & Papakonstantinou, K.G., 2019. "Managing engineering systems with large state and action spaces through deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 191(C).
    7. Randy L. Maddalena & Melissa M. Lunden & Daniel L. Wilson & Cristina Ceballos & Thomas W. Kirchstetter & Jonathan L. Slack & Larry L. Dale, 2014. "Quantifying Space Heating Stove Emissions Related to Different Use Patterns in Mongolia," Energy and Environment Research, Canadian Center of Science and Education, vol. 4(3), pages 147-147, December.
    8. Borchers, Allison M. & Duke, Joshua M. & Parsons, George R., 2007. "Does willingness to pay for green energy differ by source?," Energy Policy, Elsevier, vol. 35(6), pages 3327-3334, June.
    9. Bat-Erdene Bayandelger & Yuzuru Ueda & Amarbayar Adiyabat, 2020. "Experimental Investigation and Energy Performance Simulation of Mongolian Ger with ETS Heater and Solar PV in Ulaanbaatar City," Energies, MDPI, vol. 13(21), pages 1-13, November.
    10. Abdin, Adam F. & Caunhye, Aakil & Zio, Enrico & Cardin, Michel-Alexandre, 2022. "Optimizing generation expansion planning with operational uncertainty: A multistage adaptive robust approach," Applied Energy, Elsevier, vol. 306(PA).
    11. Pecenak, Zachary K. & Stadler, Michael & Fahy, Kelsey, 2019. "Efficient multi-year economic energy planning in microgrids," Applied Energy, Elsevier, vol. 255(C).
    12. Staffell, Iain & Pfenninger, Stefan, 2016. "Using bias-corrected reanalysis to simulate current and future wind power output," Energy, Elsevier, vol. 114(C), pages 1224-1239.
    13. Aakil M. Caunhye & Michel-Alexandre Cardin & Muhammad Rahmat, 2022. "Flexibility and real options analysis in power system generation expansion planning under uncertainty," IISE Transactions, Taylor & Francis Journals, vol. 54(9), pages 832-844, June.
    14. Ehsan, Ali & Yang, Qiang, 2018. "Optimal integration and planning of renewable distributed generation in the power distribution networks: A review of analytical techniques," Applied Energy, Elsevier, vol. 210(C), pages 44-59.
    15. Kaoru Kakinuma & Aki Yanagawa & Takehiro Sasaki & Mukund Palat Rao & Shinjiro Kanae, 2019. "Socio-ecological Interactions in a Changing Climate: A Review of the Mongolian Pastoral System," Sustainability, MDPI, vol. 11(21), pages 1-17, October.
    16. Xiujuan Yang & Jiying Sun & Julin Gao & Shuaishuai Qiao & Baolin Zhang & Haizhu Bao & Xinwei Feng & Songyu Wang, 2021. "Effects of Climate Change on Cultivation Patterns and Climate Suitability of Spring Maize in Inner Mongolia," Sustainability, MDPI, vol. 13(14), pages 1-21, July.
    17. Roman Hoffmann & Anna Dimitrova & Raya Muttarak & Jesus Crespo Cuaresma & Jonas Peisker, 2020. "A meta-analysis of country-level studies on environmental change and migration," Nature Climate Change, Nature, vol. 10(10), pages 904-912, October.
    18. Mandelli, Stefano & Barbieri, Jacopo & Mereu, Riccardo & Colombo, Emanuela, 2016. "Off-grid systems for rural electrification in developing countries: Definitions, classification and a comprehensive literature review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 58(C), pages 1621-1646.
    19. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    20. Postali, Fernando A.S. & Picchetti, Paulo, 2006. "Geometric Brownian Motion and structural breaks in oil prices: A quantitative analysis," Energy Economics, Elsevier, vol. 28(4), pages 506-522, July.
    21. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    22. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    23. Pfenninger, Stefan & Staffell, Iain, 2016. "Long-term patterns of European PV output using 30 years of validated hourly reanalysis and satellite data," Energy, Elsevier, vol. 114(C), pages 1251-1265.
    24. Michel-Alexandre Cardin & Qihui Xie & Tsan Sheng Ng & Shuming Wang & Junfei Hu, 2017. "An approach for analyzing and managing flexibility in engineering systems design based on decision rules and multistage stochastic programming," IISE Transactions, Taylor & Francis Journals, vol. 49(1), pages 1-12, January.
    25. Subbiah, Adritha & Mansoor, Sahar & Misra, Rachita & Jaffer, Huda & Tiwary, Raunak, 2016. "Addressing developmental needs through energy access in informal settlements," LSE Research Online Documents on Economics 83626, London School of Economics and Political Science, LSE Library.
    26. Martha N. Acosta & Choidorj Adiyabazar & Francisco Gonzalez-Longatt & Manuel A. Andrade & José Rueda Torres & Ernesto Vazquez & Jesús Manuel Riquelme Santos, 2020. "Optimal Under-Frequency Load Shedding Setting at Altai-Uliastai Regional Power System, Mongolia," Energies, MDPI, vol. 13(20), pages 1-18, October.
    27. Gamarra, Carlos & Guerrero, Josep M., 2015. "Computational optimization techniques applied to microgrids planning: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 48(C), pages 413-424.
    28. Peter Boait & Rupert Gammon & Varun Advani & Neal Wade & David Greenwood & Peter Davison, 2017. "ESCoBox: A Set of Tools for Mini-Grid Sustainability in the Developing World," Sustainability, MDPI, vol. 9(5), pages 1-15, May.
    29. Wang, Yi & Rousis, Anastasios Oulis & Strbac, Goran, 2020. "On microgrids and resilience: A comprehensive review on modeling and operational strategies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 134(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bigestans, Davis & Cardin, Michel-Alexandre & Kazantzis, Nikolaos, 2023. "Economic performance evaluation of flexible centralised and decentralised blue hydrogen production systems design under uncertainty," Applied Energy, Elsevier, vol. 352(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Petkov, Ivalin & Gabrielli, Paolo, 2020. "Power-to-hydrogen as seasonal energy storage: an uncertainty analysis for optimal design of low-carbon multi-energy systems," Applied Energy, Elsevier, vol. 274(C).
    2. Qiu, Dawei & Wang, Yi & Hua, Weiqi & Strbac, Goran, 2023. "Reinforcement learning for electric vehicle applications in power systems:A critical review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 173(C).
    3. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    4. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Data-driven battery operation for energy arbitrage using rainbow deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    5. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Renewable energy integration and microgrid energy trading using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 318(C).
    6. Petrelli, Marina & Fioriti, Davide & Berizzi, Alberto & Bovo, Cristian & Poli, Davide, 2021. "A novel multi-objective method with online Pareto pruning for multi-year optimization of rural microgrids," Applied Energy, Elsevier, vol. 299(C).
    7. Biemann, Marco & Scheller, Fabian & Liu, Xiufeng & Huang, Lizhen, 2021. "Experimental evaluation of model-free reinforcement learning algorithms for continuous HVAC control," Applied Energy, Elsevier, vol. 298(C).
    8. Abada, Ibrahim & Othmani, Mehdi & Tatry, Léa, 2021. "An innovative approach for the optimal sizing of mini-grids in rural areas integrating the demand, the supply, and the grid," Renewable and Sustainable Energy Reviews, Elsevier, vol. 146(C).
    9. Petkov, Ivalin & Mavromatidis, Georgios & Knoeri, Christof & Allan, James & Hoffmann, Volker H., 2022. "MANGOret: An optimization framework for the long-term investment planning of building multi-energy system and envelope retrofits," Applied Energy, Elsevier, vol. 314(C).
    10. Maeder, Mattia & Weiss, Olga & Boulouchos, Konstantinos, 2021. "Assessing the need for flexibility technologies in decarbonized power systems: A new model applied to Central Europe," Applied Energy, Elsevier, vol. 282(PA).
    11. de Guibert, Paul & Shirizadeh, Behrang & Quirion, Philippe, 2020. "Variable time-step: A method for improving computational tractability for energy system models with long-term storage," Energy, Elsevier, vol. 213(C).
    12. Marko Hočevar & Lovrenc Novak & Primož Drešar & Gašper Rak, 2022. "The Status Quo and Future of Hydropower in Slovenia," Energies, MDPI, vol. 15(19), pages 1-13, September.
    13. Lukas Kriechbaum & Philipp Gradl & Romeo Reichenhauser & Thomas Kienberger, 2020. "Modelling Grid Constraints in a Multi-Energy Municipal Energy System Using Cumulative Exergy Consumption Minimisation," Energies, MDPI, vol. 13(15), pages 1-23, July.
    14. Behrang Shirizadeh, Quentin Perrier, and Philippe Quirion, 2022. "How Sensitive are Optimal Fully Renewable Power Systems to Technology Cost Uncertainty?," The Energy Journal, International Association for Energy Economics, vol. 0(Number 1).
    15. Liu, Hailiang & Andresen, Gorm Bruun & Greiner, Martin, 2018. "Cost-optimal design of a simplified highly renewable Chinese electricity network," Energy, Elsevier, vol. 147(C), pages 534-546.
    16. Antoine Boche & Clément Foucher & Luiz Fernando Lavado Villa, 2022. "Understanding Microgrid Sustainability: A Systemic and Comprehensive Review," Energies, MDPI, vol. 15(8), pages 1-29, April.
    17. Huang, Ruchen & He, Hongwen & Gao, Miaojue, 2023. "Training-efficient and cost-optimal energy management for fuel cell hybrid electric bus based on a novel distributed deep reinforcement learning framework," Applied Energy, Elsevier, vol. 346(C).
    18. Géremi Gilson Dranka & Paula Ferreira, 2020. "Electric Vehicles and Biofuels Synergies in the Brazilian Energy System," Energies, MDPI, vol. 13(17), pages 1-22, August.
    19. Shirizadeh, Behrang & Quirion, Philippe, 2022. "The importance of renewable gas in achieving carbon-neutrality: Insights from an energy system optimization model," Energy, Elsevier, vol. 255(C).
    20. Liu, Hailiang & Brown, Tom & Andresen, Gorm Bruun & Schlachtberger, David P. & Greiner, Martin, 2019. "The role of hydro power, storage and transmission in the decarbonization of the Chinese power system," Applied Energy, Elsevier, vol. 239(C), pages 1308-1321.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:335:y:2023:i:c:s0306261923000715. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.