A co-state self-learning energy management strategy for plug-in hybrid electric buses integrating Q-learning and Pontryagin's minimum principle

This paper presents a novel co-state self-learning energy management strategy for plug-in hybrid electric buses (PHEBs) by integrating Q-learning and Pontryagin's minimum principle (PMP). Two distinct approaches are developed for fixed and non-fixed bus routes to overcome the limitations of conventional strategies, including reliance on fixed parameters and poor adaptability to dynamic driving conditions. For fixed routes, a two-dimensional “chessboard” architecture is proposed using battery state of charge (SOC) and normalized driving distance (NDD) as state variables, which effectively reduces the state space. For non-fixed routes, a three-dimensional “Rubik's cube” architecture is further proposed by introducing current driving distance (CDD) as an additional state variable to improve generalization ability. In both architectures, the PMP co-state is directly used as the learning action, and a reward function is designed based on the optimal SOC reference trajectory obtained from offline PMP optimization. Offline training is conducted under real-world driving cycles using a ε-greedy algorithm to realize stable algorithm convergence. Hardware-in-the-loop (HIL) verification shows that the proposed methods can effectively regulate terminal SOC and achieve fuel economy highly comparable to the offline PMP optimum. The proposed approach eliminates the heavy offline computation of the shooting method and achieves reliable online adaptive adjustment of the co-state with favorable real-time performance.