IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v325y2022ics030626192201087x.html
   My bibliography  Save this article

The emerging threat of artificial intelligence on competition in liberalized electricity markets: A deep Q-network approach

Author

Listed:
  • Esmaeili Aliabadi, Danial
  • Chan, Katrina

Abstract

According to Sustainable Development Goals (SDGs), societies should have access to affordable, reliable, and sustainable energy. Liberalized electricity markets have been established to provide affordable electricity for end-users through advertising competition. Although these new markets are designed to serve competition, there are recorded incidents where participants abused their market power and disrupted the competition through collusion. Unfortunately, modern autonomous pricing algorithms may further assist myopic players to discover collusive strategies with a minimum amount of sensitive information. Therefore, in this study, we investigate the impact of emerging learning algorithms on the bidding strategies of Power Generating Companies (GenCos) and compare their performance against game-theoretic expectations. A novel deep Q-network (DQN) model is developed, by which GenCos determine the bidding strategies to maximize average long-term payoffs in a day-ahead market. The presented DQN model assumes that GenCos have no information regarding the rivals’ true generation costs and profits. To the best of the authors’ knowledge, this is the first study that thoroughly investigates players’ behavior utilizing a modern DQN model and compares its results with equilibria of the non-cooperative single-stage and infinitely-repeated games in the context of electricity markets. The outcomes articulate that GenCos equipped with advanced learning models may be able to collude unintentionally while trying to ameliorate long-term profits. Moreover, GenCos that employ the presented DQN model could discover and sustain more profitable (e.g., collusive) strategies vis-à-vis a conventional Q-learning method. Collusive strategies can lead to exorbitant electric bills for end-users, which is one of the influential factors in energy poverty. Thus, policymakers and market designers should be vigilant regarding the combined effect of information disclosure and autonomous pricing, as new models exploit information more effectively.

Suggested Citation

  • Esmaeili Aliabadi, Danial & Chan, Katrina, 2022. "The emerging threat of artificial intelligence on competition in liberalized electricity markets: A deep Q-network approach," Applied Energy, Elsevier, vol. 325(C).
    • Handle: RePEc:eee:appene:v:325:y:2022:i:c:s030626192201087x
    DOI: 10.1016/j.apenergy.2022.119813
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S030626192201087X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2022.119813?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Pär Holmberg & Frank Wolak, 2015. "Electricity markets: Designing auctions where suppliers have uncertain costs," Working Papers EPRG 1523, Energy Policy Research Group, Cambridge Judge Business School, University of Cambridge.
    2. Heim, Sven & Götz, Georg, 2021. "Do Pay-As-Bid Auctions Favor Collusion? Evidence from Germany's market for reserve power," Energy Policy, Elsevier, vol. 155(C).
    3. Frank A. Wolak, 2014. "Regulating Competition in Wholesale Electricity Supply," NBER Chapters, in: Economic Regulation and Its Reform: What Have We Learned?, pages 195-289, National Bureau of Economic Research, Inc.
    4. Kursad Derinkuyu & Fehmi Tanrisever & Nermin Kurt & Gokhan Ceyhan, 2020. "Optimizing Day-Ahead Electricity Market Prices: Increasing the Total Surplus for Energy Exchange Istanbul," Manufacturing & Service Operations Management, INFORMS, vol. 22(4), pages 700-716, July.
    5. Douglas Cooke, 2011. "Empowering Customer Choice in Electricity Markets," IEA Energy Papers 2011/13, OECD Publishing.
    6. Nils-Henrik M. von der Fehr, 2013. "Transparency in Electricity Markets," Economics of Energy & Environmental Policy, International Association for Energy Economics, vol. 0(Number 2).
    7. Mahdi Hemmati & Masoud Nili & Nasser Sadati, 2010. "Reinforcement Learning of Heterogeneous Private Agents in a Macroeconomic Policy Game," Lecture Notes in Economics and Mathematical Systems, in: Marco Li Calzi & Lucia Milone & Paolo Pellizzari (ed.), Progress in Artificial Economics, pages 215-226, Springer.
    8. Emilio Calvano & Giacomo Calzolari & Vincenzo Denicolò & Sergio Pastorello, 2020. "Artificial Intelligence, Algorithmic Pricing, and Collusion," American Economic Review, American Economic Association, vol. 110(10), pages 3267-3297, October.
    9. Haufe, Marie-Christin & Ehrhart, Karl-Martin, 2018. "Auctions for renewable energy support – Suitability, design, and first lessons learned," Energy Policy, Elsevier, vol. 121(C), pages 217-224.
    10. Joseph E Harrington, 2018. "Developing Competition Law For Collusion By Autonomous Artificial Agents," Journal of Competition Law and Economics, Oxford University Press, vol. 14(3), pages 331-363.
    11. Aliabadi, Danial Esmaeili & Kaya, Murat & Şahin, Güvenç, 2017. "An agent-based simulation of power generation company behavior in electricity markets under different market-clearing mechanisms," Energy Policy, Elsevier, vol. 100(C), pages 191-205.
    12. Fabra, Natalia & Toro, Juan, 2005. "Price wars and collusion in the Spanish electricity market," International Journal of Industrial Organization, Elsevier, vol. 23(3-4), pages 155-181, April.
    13. Richard Bellman, 1954. "Some Applications of the Theory of Dynamic Programming---A Review," Operations Research, INFORMS, vol. 2(3), pages 275-288, August.
    14. Papada, Lefkothea & Kaliampakos, Dimitris, 2016. "Measuring energy poverty in Greece," Energy Policy, Elsevier, vol. 94(C), pages 157-165.
    15. Lea Bernhardt & Ralf Dewenter, 2020. "Collusion by code or algorithmic collusion? When pricing algorithms take over," European Competition Journal, Taylor & Francis Journals, vol. 16(2-3), pages 312-342, September.
    16. David P. Brown & Andrew Eckert & James Lin, 2018. "Information and transparency in wholesale electricity markets: evidence from Alberta," Journal of Regulatory Economics, Springer, vol. 54(3), pages 292-330, December.
    17. Huisman, Ronald & Huurman, Christian & Mahieu, Ronald, 2007. "Hourly electricity prices in day-ahead markets," Energy Economics, Elsevier, vol. 29(2), pages 240-248, March.
    18. Richard Benjamin, 2016. "Tacit Collusion in Electricity Markets with Uncertain Demand," Review of Industrial Organization, Springer;The Industrial Organization Society, vol. 48(1), pages 69-93, February.
    19. Andrew Sweeting, 2007. "Market Power In The England And Wales Wholesale Electricity Market 1995-2000," Economic Journal, Royal Economic Society, vol. 117(520), pages 654-685, April.
    20. Timo Klein, 2021. "Autonomous algorithmic collusion: Q‐learning under sequential pricing," RAND Journal of Economics, RAND Corporation, vol. 52(3), pages 538-558, September.
    21. Richard Benjamin, 2016. "Tacit Collusion in Electricity Markets with Uncertain Demand," Review of Industrial Organization, Springer;The Industrial Organization Society, vol. 48(1), pages 69-93, February.
    22. Poplavskaya, Ksenia & Lago, Jesus & Strömer, Stefan & de Vries, Laurens, 2021. "Making the most of short-term flexibility in the balancing market: Opportunities and challenges of voluntary bids in the new balancing market design," Energy Policy, Elsevier, vol. 158(C).
    23. Özbuğday, Fatih Cemil & Öğünlü, Bilal & Alma, Hasan, 2016. "The sustainability of Turkish electricity distributors and last-resort electricity suppliers: What did transition from vertically integrated public monopoly to regulated competition with privatized an," Utilities Policy, Elsevier, vol. 39(C), pages 50-67.
    24. Yang Yang & Minglei Bao & Yi Ding & Yonghua Song & Zhenzhi Lin & Changzheng Shao, 2018. "Review of Information Disclosure in Different Electricity Markets," Energies, MDPI, vol. 11(12), pages 1-20, December.
    25. Nur Mohammad & Yateendra Mishra, 2018. "The Role of Demand Response Aggregators and the Effect of GenCos Strategic Bidding on the Flexibility of Demand," Energies, MDPI, vol. 11(12), pages 1-22, November.
    26. Erev, Ido & Roth, Alvin E, 1998. "Predicting How People Play Games: Reinforcement Learning in Experimental Games with Unique, Mixed Strategy Equilibria," American Economic Review, American Economic Association, vol. 88(4), pages 848-881, September.
    27. Richard Bellman, 1954. "On some applications of the theory of dynamic programming to logistics," Naval Research Logistics Quarterly, John Wiley & Sons, vol. 1(2), pages 141-153, June.
    28. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    29. Poplavskaya, Ksenia & Lago, Jesus & de Vries, Laurens, 2020. "Effect of market design on strategic bidding behavior: Model-based analysis of European electricity balancing markets," Applied Energy, Elsevier, vol. 270(C).
    30. Markard, Jochen & Holt, Edward, 2003. "Disclosure of electricity products--lessons from consumer research as guidance for energy policy," Energy Policy, Elsevier, vol. 31(14), pages 1459-1474, November.
    31. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    32. Yihsu Chen & Benjamin Hobbs & Sven Leyffer & Todd Munson, 2006. "Leader-Follower Equilibria for Electric Power and NO x Allowances Markets," Computational Management Science, Springer, vol. 3(4), pages 307-330, September.
    33. Esmaeili Aliabadi, Danial & Kaya, Murat & Sahin, Guvenc, 2017. "Competition, risk and learning in electricity markets: An agent-based simulation study," Applied Energy, Elsevier, vol. 195(C), pages 1000-1011.
    34. Klemperer, Paul D & Meyer, Margaret A, 1989. "Supply Function Equilibria in Oligopoly under Uncertainty," Econometrica, Econometric Society, vol. 57(6), pages 1243-1277, November.
    35. David Silver & Aja Huang & Chris J. Maddison & Arthur Guez & Laurent Sifre & George van den Driessche & Julian Schrittwieser & Ioannis Antonoglou & Veda Panneershelvam & Marc Lanctot & Sander Dieleman, 2016. "Mastering the game of Go with deep neural networks and tree search," Nature, Nature, vol. 529(7587), pages 484-489, January.
    36. Joskow, Paul L, 1996. "Introducing Competition into Regulated Network Industries: From Hierarchies to Markets in Electricity," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 5(2), pages 341-382.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yuna Hao & Behrang Vand & Benjamin Manrique Delgado & Simone Baldi, 2023. "Market Manipulation in Stock and Power Markets: A Study of Indicator-Based Monitoring and Regulatory Challenges," Energies, MDPI, vol. 16(4), pages 1-28, February.
    2. Cheng, Xiong & Lv, Xin & Li, Xianshan & Zhong, Hao & Feng, Jia, 2023. "Market power evaluation in the electricity market based on the weighted maintenance object," Energy, Elsevier, vol. 284(C).
    3. Danial Esmaeili Aliabadi & David Manske & Lena Seeger & Reinhold Lehneis & Daniela Thrän, 2023. "Integrating Knowledge Acquisition, Visualization, and Dissemination in Energy System Models: BENOPTex Study," Energies, MDPI, vol. 16(13), pages 1-14, July.
    4. Ogbuabor, Jonathan E. & Ukwueze, Ezebuilo R. & Mba, Ifeoma C. & Ojonta, Obed I. & Orji, Anthony, 2023. "The asymmetric impact of economic policy uncertainty on global retail energy markets: Are the markets responding to the fear of the unknown?," Applied Energy, Elsevier, vol. 334(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Leonardo Madio & Aldo Pignataro, 2022. "Collusion sustainability with a capacity constrained firm," "Marco Fanno" Working Papers 0295, Dipartimento di Scienze Economiche "Marco Fanno".
    2. Leonardo Madio & Aldo Pignataro, 2022. "Collusion Sustainability with a Capacity Constrained Firm," CESifo Working Paper Series 10170, CESifo.
    3. David P. Brown & Andrew Eckert, 2022. "Pricing Patterns in Wholesale Electricity Markets: Unilateral Market Power or Coordinated Behavior?," Journal of Industrial Economics, Wiley Blackwell, vol. 70(1), pages 168-216, March.
    4. David P. Brown & Andrew Eckert & James Lin, 2018. "Information and transparency in wholesale electricity markets: evidence from Alberta," Journal of Regulatory Economics, Springer, vol. 54(3), pages 292-330, December.
    5. Viktorija Bobinaite & Marialaura Di Somma & Giorgio Graditi & Irina Oleinikova, 2019. "The Regulatory Framework for Market Transparency in Future Power Systems under the Web-of-Cells Concept," Energies, MDPI, vol. 12(5), pages 1-26, March.
    6. Brown, David P. & Eckert, Andrew & Silveira, Douglas, 2023. "Screening for Collusion in Wholesale Electricity Markets: A Review of the Literature," Working Papers 2023-7, University of Alberta, Department of Economics.
    7. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Data-driven battery operation for energy arbitrage using rainbow deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    8. Johnson, Justin Pappas & Rhodes, Andrew & Wildenbeest, Matthijs, 2020. "Platform Design when Sellers Use Pricing Algorithms," TSE Working Papers 20-1146, Toulouse School of Economics (TSE).
    9. Christoph Graf & Viktor Zobernig & Johannes Schmidt & Claude Klöckl, 2024. "Computational Performance of Deep Reinforcement Learning to Find Nash Equilibria," Computational Economics, Springer;Society for Computational Economics, vol. 63(2), pages 529-576, February.
    10. Xiaoyue Li & John M. Mulvey, 2023. "Optimal Portfolio Execution in a Regime-switching Market with Non-linear Impact Costs: Combining Dynamic Program and Neural Network," Papers 2306.08809, arXiv.org.
    11. Mahmoud Mahfouz & Angelos Filos & Cyrine Chtourou & Joshua Lockhart & Samuel Assefa & Manuela Veloso & Danilo Mandic & Tucker Balch, 2019. "On the Importance of Opponent Modeling in Auction Markets," Papers 1911.12816, arXiv.org.
    12. David P. Brown & Andrew Eckert, 2018. "Analyzing the Impact of Electricity Market Structure Changes and Mergers: The Importance of Forward Commitments," Review of Industrial Organization, Springer;The Industrial Organization Society, vol. 52(1), pages 101-137, February.
    13. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    14. Boute, Robert N. & Gijsbrechts, Joren & van Jaarsveld, Willem & Vanvuchelen, Nathalie, 2022. "Deep reinforcement learning for inventory control: A roadmap," European Journal of Operational Research, Elsevier, vol. 298(2), pages 401-412.
    15. Oleh Lukianykhin & Tetiana Bogodorova, 2021. "Voltage Control-Based Ancillary Service Using Deep Reinforcement Learning," Energies, MDPI, vol. 14(8), pages 1-22, April.
    16. Emilio Calvano & Giacomo Calzolari & Vincenzo Denicolò & Sergio Pastorello, 2019. "Algorithmic Pricing What Implications for Competition Policy?," Review of Industrial Organization, Springer;The Industrial Organization Society, vol. 55(1), pages 155-171, August.
    17. Bayona, Anna & Brandts, Jordi & Vives, Xavier, 2020. "Information frictions and market power: A laboratory study," Games and Economic Behavior, Elsevier, vol. 122(C), pages 354-369.
    18. Gonzalo Ballestero, 2021. "Collusion and Artificial Intelligence: A computational experiment with sequential pricing algorithms under stochastic costs," Young Researchers Working Papers 1, Universidad de San Andres, Departamento de Economia, revised Oct 2022.
    19. Priyanka Shinde & Ioannis Boukas & David Radu & Miguel Manuel de Villena & Mikael Amelin, 2021. "Analyzing Trade in Continuous Intra-Day Electricity Market: An Agent-Based Modeling Approach," Energies, MDPI, vol. 14(13), pages 1-31, June.
    20. Jason D. Hartline & Sheng Long & Chenhao Zhang, 2024. "Regulation of Algorithmic Collusion," Papers 2401.15794, arXiv.org.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:325:y:2022:i:c:s030626192201087x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.