Strategic bidding with price-quantity pairs based on deep reinforcement learning considering competitors' behaviors

In a smart electricity market, self-interested market participants may leverage a large amount of market data to bid strategically to maximize their profits. However, the existing studies in strategic bidding often ignore competitors' bidding behaviors and only consider strategic actions on prices without quantities. To bridge the gap, this paper develops a novel deep reinforcement learning-based framework to model and solve the strategic bidding problem of a producer. To capture competitors' historical bidding behaviors in the market environment, their demand-bid mappings are established based on a data-driven method combining K-medoids clustering and a deep neural network. To make full use of the bidding action space and increase the profit of the strategic producer, a bilevel optimization model considering bids in price-quantity pairs is formulated. To efficiently solve the problem with competitors' bidding behaviors, a twin delayed deep deterministic policy gradient-based algorithm is developed. Case studies on the IEEE 57-bus system show that the proposed framework obtains a 27.37 % higher expected value and a 47.60 % lower standard deviation of the profit compared to the existing approach, demonstrating its profitability and robustness under market dynamics. Another case on the IEEE 118-bus test system achieves a 33.34 % increase in the expected profit, further validating the advantages in profitability. These cases together demonstrate the effectiveness and scalability of our approach in systems of different sizes, as well as its potential application to strategic bidding in smart electricity markets.