An indirect reinforcement learning based real-time energy management strategy via high-order Markov Chain model for a hybrid electric vehicle

This paper proposes a real-time indirect reinforcement learning based strategy to reduce the fuel consumption. In order to improve the real-time performance and achieve learning online, the simulated experience from environment model is adopted for the learning process, which is called indirect reinforcement learning. To establish an accurate environment model, a high-order Markov Chain is introduced and detailed, which is more precise than a widely used first-order Markov Chain. Corresponding with the model, how the reinforcement learning algorithm learns from the simulated experience is illustrated. Furthermore, an online recursive form of the transition probability matrix is derived, through which the statistical characteristics from the practical driving conditions can be collected. The induced matrix norm is chosen as a criterion to quantify the differences between the transition probability matrices and to determine the time for updating the environment model and triggering the recalculation of the reinforcement learning algorithm. Simulation results demonstrate that, compared with the direct RL, the proposed strategy can effectively reduce the learning time while maintains satisfied fuel economy. Furthermore, a hardware-in-the-loop experiment verifies its real-time capability and actual applicability.