Dynamic ad hoc teaming and mutual distillation for cooperative learning of powertrain control policies for vehicle fleets

Data-driven deep reinforcement learning (DRL)-based approaches have shown significant potential for improving the performance of vehicle control systems, in terms of energy consumption and other metrics, by allowing adaptation to the environments in which the vehicles are deployed. However, training DRL policies that work well in highly dynamic real-world environments is challenged by data efficiency and learning stability issues accompanied by high variances in performance. In this paper, we propose a novel cooperative learning approach to improve learning performance and reduce variances by continuously sharing experiences among powertrain control agents for a fleet of vehicles. The key contribution is the concept of a dynamic ad hoc teaming mechanism for decentralized and scalable mutual knowledge distillation between vehicles serving a distribution of routes. Our approach enables an asynchronous implementation that can operate whenever connectivity is available, thus removing a constraint for practical adoption. We compare two variants of the proposed framework with two other state-of-the-art alternatives in three scenarios that represent various deployments for a fleet. We find that the proposed framework significantly accelerates learning by reducing variances and improves long-term fleet mean total cycle rewards by up to 14 % compared to a baseline of individually learning agents. This improvement is on the same order as that achieved with centralized shared learning approaches, but without suffering their limitations of computational complexity and poor scalability. We also find that the proposed shared learning approach improves the adaptability of vehicle control agents to unfamiliar routes.