IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v9y2016i9p725-d77836.html
   My bibliography  Save this article

Application of a Gradient Descent Continuous Actor-Critic Algorithm for Double-Side Day-Ahead Electricity Market Modeling

Author

Listed:
  • Huiru Zhao

    (School of Economics and Management, North China Electric Power University, Beijing 102206, China)

  • Yuwei Wang

    (School of Economics and Management, North China Electric Power University, Beijing 102206, China)

  • Sen Guo

    (School of Economics and Management, North China Electric Power University, Beijing 102206, China)

  • Mingrui Zhao

    (School of Economics and Management, North China Electric Power University, Beijing 102206, China)

  • Chao Zhang

    (School of Economics and Management, North China Electric Power University, Beijing 102206, China)

Abstract

An important goal of China’s electric power system reform is to create a double-side day-ahead wholesale electricity market in the future, where the suppliers (represented by GenCOs) and demanders (represented by DisCOs) compete simultaneously with each other in one market. Therefore, modeling and simulating the dynamic bidding process and the equilibrium in the double-side day-ahead electricity market scientifically is not only important to some developed countries, but also to China to provide a bidding decision-making tool to help GenCOs and DisCOs obtain more profits in market competition. Meanwhile, it can also provide an economic analysis tool to help government officials design the proper market mechanisms and policies. The traditional dynamic game model and table-based reinforcement learning algorithm have already been employed in the day-ahead electricity market modeling. However, those models are based on some assumptions, such as taking the probability distribution function of market clearing price ( MCP ) and each rival’s bidding strategy as common knowledge (in dynamic game market models), and assuming the discrete state and action sets of every agent (in table-based reinforcement learning market models), which are no longer applicable in a realistic situation. In this paper, a modified reinforcement learning method, called gradient descent continuous Actor-Critic (GDCAC) algorithm was employed in the double-side day-ahead electricity market modeling and simulation. This algorithm can not only get rid of the abovementioned unrealistic assumptions, but also cope with the Markov decision-making process with continuous state and action sets just like the real electricity market. Meanwhile, the time complexity of our proposed model is only O( n ). The simulation result of employing the proposed model in the double-side day-ahead electricity market shows the superiority of our approach in terms of participant’s profit or social welfare compared with traditional reinforcement learning methods.

Suggested Citation

  • Huiru Zhao & Yuwei Wang & Sen Guo & Mingrui Zhao & Chao Zhang, 2016. "Application of a Gradient Descent Continuous Actor-Critic Algorithm for Double-Side Day-Ahead Electricity Market Modeling," Energies, MDPI, vol. 9(9), pages 1-20, September.
    • Handle: RePEc:gam:jeners:v:9:y:2016:i:9:p:725-:d:77836
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/9/9/725/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/9/9/725/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Li, Hongyan & Tesfatsion, Leigh, 2012. "Co-learning patterns as emergent market phenomena: An electricity market illustration," Journal of Economic Behavior & Organization, Elsevier, vol. 82(2), pages 395-419.
    2. Sun, Qi & Xu, Lin & Yin, Hua, 2016. "Energy pricing reform and energy efficiency in China: Evidence from the automobile market," Resource and Energy Economics, Elsevier, vol. 44(C), pages 39-51.
    3. Mahvi, M. & Ardehali, M.M., 2011. "Optimal bidding strategy in a competitive electricity market based on agent-based approach and numerical sensitivity analysis," Energy, Elsevier, vol. 36(11), pages 6367-6374.
    4. Azadeh, A. & Skandari, M.R. & Maleki-Shoja, B., 2010. "An integrated ant colony optimization approach to compare strategies of clearing market in electricity markets: Agent-based simulation," Energy Policy, Elsevier, vol. 38(10), pages 6307-6319, October.
    5. Salehizadeh, Mohammad Reza & Soltaniyan, Salman, 2016. "Application of fuzzy Q-learning for electricity market modeling by considering renewable power penetration," Renewable and Sustainable Energy Reviews, Elsevier, vol. 56(C), pages 1172-1181.
    6. Mou, Dunguo, 2014. "Understanding China’s electricity market reform from the perspective of the coal-fired power disparity," Energy Policy, Elsevier, vol. 74(C), pages 224-234.
    7. Ringler, Philipp & Keles, Dogan & Fichtner, Wolf, 2016. "Agent-based modelling and simulation of smart electricity grids and markets – A literature review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 57(C), pages 205-215.
    8. Wang, Jianhui & Zhou, Zhi & Botterud, Audun, 2011. "An evolutionary game approach to analyzing bidding strategies in electricity markets with elastic demand," Energy, Elsevier, vol. 36(5), pages 3459-3467.
    9. Al-Agtash, Salem Y., 2010. "Supply curve bidding of electricity in constrained power networks," Energy, Elsevier, vol. 35(7), pages 2886-2892.
    10. Ma, Chunbo & Zhao, Xiaoli, 2015. "China's electricity market restructuring and technology mandates: Plant-level evidence for changing operational efficiency," Energy Economics, Elsevier, vol. 47(C), pages 227-237.
    11. Shivaie, Mojtaba & Ameli, Mohammad T., 2015. "An environmental/techno-economic approach for bidding strategy in security-constrained electricity markets by a bi-level harmony search algorithm," Renewable Energy, Elsevier, vol. 83(C), pages 881-896.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kaveh Dehghanpour & Christopher Colson & Hashem Nehrir, 2017. "A Survey on Smart Agent-Based Microgrids for Resilient/Self-Healing Grids," Energies, MDPI, vol. 10(5), pages 1-25, May.
    2. Alexander Kell, 2021. "Modelling the transition to a low-carbon energy supply," Papers 2111.00987, arXiv.org.
    3. Guanglei Huang & Tian Mao & Bin Zhang & Renli Cheng & Mingyu Ou, 2023. "An Intelligent Algorithm for Solving Unit Commitments Based on Deep Reinforcement Learning," Sustainability, MDPI, vol. 15(14), pages 1-19, July.
    4. Nur Mohammad & Yateendra Mishra, 2018. "The Role of Demand Response Aggregators and the Effect of GenCos Strategic Bidding on the Flexibility of Demand," Energies, MDPI, vol. 11(12), pages 1-22, November.
    5. Xiaoya Shang & Zhigang Li & Tianyao Ji & P. Z. Wu & Qinghua Wu, 2017. "Online Area Load Modeling in Power Systems Using Enhanced Reinforcement Learning," Energies, MDPI, vol. 10(11), pages 1-17, November.
    6. Chuanjia Han & Bo Yang & Tao Bao & Tao Yu & Xiaoshun Zhang, 2017. "Bacteria Foraging Reinforcement Learning for Risk-Based Economic Dispatch via Knowledge Transfer," Energies, MDPI, vol. 10(5), pages 1-24, May.
    7. Huiru Zhao & Yuwei Wang & Mingrui Zhao & Qingkun Tan & Sen Guo, 2017. "Day-Ahead Market Modeling for Strategic Wind Power Producers under Robust Market Clearing," Energies, MDPI, vol. 10(7), pages 1-27, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Huiru Zhao & Yuwei Wang & Mingrui Zhao & Qingkun Tan & Sen Guo, 2017. "Day-Ahead Market Modeling for Strategic Wind Power Producers under Robust Market Clearing," Energies, MDPI, vol. 10(7), pages 1-27, July.
    2. HuiHui Liu & ZhongXiang Zhang & ZhanMing Chen & DeSheng Dou, 2018. "The Impact of China’s Electricity Deregulation on Coal and Power Industries: Two-stage Game Modeling Approach," Working Papers 2018.17, Fondazione Eni Enrico Mattei.
    3. Afshar, Karim & Ghiasvand, Farshad Shamsini & Bigdeli, Nooshin, 2018. "Optimal bidding strategy of wind power producers in pay-as-bid power markets," Renewable Energy, Elsevier, vol. 127(C), pages 575-586.
    4. She, Zhen-Yu & Meng, Gang & Xie, Bai-Chen & O'Neill, Eoghan, 2020. "The effectiveness of the unbundling reform in China’s power system from a dynamic efficiency perspective," Applied Energy, Elsevier, vol. 264(C).
    5. Shivaie, Mojtaba & Ameli, Mohammad T., 2015. "An environmental/techno-economic approach for bidding strategy in security-constrained electricity markets by a bi-level harmony search algorithm," Renewable Energy, Elsevier, vol. 83(C), pages 881-896.
    6. Li, Gong & Shi, Jing & Qu, Xiuli, 2011. "Modeling methods for GenCo bidding strategy optimization in the liberalized electricity spot market–A state-of-the-art review," Energy, Elsevier, vol. 36(8), pages 4686-4700.
    7. Zeng, Ming & Yang, Yongqi & Wang, Lihua & Sun, Jinghui, 2016. "The power industry reform in China 2015: Policies, evaluations and solutions," Renewable and Sustainable Energy Reviews, Elsevier, vol. 57(C), pages 94-110.
    8. Ke Wang & Jieming Zhang & Yi-Ming Wei, 2017. "Operational and environmental performance in China¡¯s thermal power industry: Taking an effectiveness measure as complement to an efficiency measure," CEEP-BIT Working Papers 100, Center for Energy and Environmental Policy Research (CEEP), Beijing Institute of Technology.
    9. Liu, HuiHui & Zhang, ZhongXiang & Chen, Zhan-Ming & Dou, DeSheng, 2019. "The impact of China's electricity price deregulation on coal and power industries: Two-stage game modeling," Energy Policy, Elsevier, vol. 134(C).
    10. Min, C.G. & Kim, M.K. & Park, J.K. & Yoon, Y.T., 2013. "Game-theory-based generation maintenance scheduling in electricity markets," Energy, Elsevier, vol. 55(C), pages 310-318.
    11. Li, Gao & Ruonan, Li & Yingdan, Mei & Xiaoli, Zhao, 2022. "Improve technical efficiency of China's coal-fired power enterprises: Taking a coal-fired-withdrawl context," Energy, Elsevier, vol. 252(C).
    12. Yu, Liying & Wang, Peng & Chen, Zhe & Li, Dewen & Li, Ning & Cherkaoui, Rachid, 2023. "Finding Nash equilibrium based on reinforcement learning for bidding strategy and distributed algorithm for ISO in imperfect electricity market," Applied Energy, Elsevier, vol. 350(C).
    13. Alqahtani, Mohammed & Hu, Mengqi, 2022. "Dynamic energy scheduling and routing of multiple electric vehicles using deep reinforcement learning," Energy, Elsevier, vol. 244(PA).
    14. Pinto, T. & Morais, H. & Oliveira, P. & Vale, Z. & Praça, I. & Ramos, C., 2011. "A new approach for multi-agent coalition formation and management in the scope of electricity markets," Energy, Elsevier, vol. 36(8), pages 5004-5015.
    15. Chen, Hao & Cui, Jian & Song, Feng & Jiang, Zhigao, 2022. "Evaluating the impacts of reforming and integrating China's electricity sector," Energy Economics, Elsevier, vol. 108(C).
    16. Dunguo Mou, 2018. "Wind Power Development and Energy Storage under China’s Electricity Market Reform—A Case Study of Fujian Province," Sustainability, MDPI, vol. 10(2), pages 1-20, January.
    17. Ning Wang & Weisheng Xu & Weihui Shao & Zhiyu Xu, 2019. "A Q-Cube Framework of Reinforcement Learning Algorithm for Continuous Double Auction among Microgrids," Energies, MDPI, vol. 12(15), pages 1-26, July.
    18. Weron, Tomasz & Kowalska-Pyzalska, Anna & Weron, Rafał, 2018. "The role of educational trainings in the diffusion of smart metering platforms: An agent-based modeling approach," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 505(C), pages 591-600.
    19. Sueyoshi, Toshiyuki & Goto, Mika, 2015. "Environmental assessment on coal-fired power plants in U.S. north-east region by DEA non-radial measurement," Energy Economics, Elsevier, vol. 50(C), pages 125-139.
    20. Bubak, Baran, 2020. "تخمین نرخ بهینه عوارض تراکم برای آزادراه‏های ایران با استفاده از مدل قیمت‏گذاری ارزش [Estimation of Optimal Rate for Compact Tariffs in Highways across Iran applying the Value Pricing Methodology]," MPRA Paper 105490, University Library of Munich, Germany.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:9:y:2016:i:9:p:725-:d:77836. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.