Spatiotemporal Features-Aware relocating for idle vehicles using spatial mean field deep Q network reinforcement learning

The cruising behavior of idle ride-hailing vehicles in search of passengers is a key influencing factor that restricts the spatiotemporal balance between online ride-hailing supply and passenger demands. This paper aims to simulate the strategy of transferring idle vehicles in multiple hexagonal partitions to adjacent grid partitions by proposing a spatiotemporal features-aware relocating approach (STFAR) that integrates spatiotemporal features of ride hailing into deep reinforcement learning. Specifically, spatial clustering algorithm and time series clustering algorithm are used to identify the spatiotemporal pattern of ride-hailing demand in each hexagonal partition. In addition, the direction of central hot spot is determined by accurately predicting the future short-term travel demand of each hexagonal partition. Finally, a spatial mean field deep Q network (SMFDQN) reinforcement learning method which regards the hexagonal partition as limited and fixed numbers spatial multi-agents is proposed to optimize the efficiency of idle vehicle transfer. STFAR improves the SMFDQN method by integrating the above spatiotemporal features into state space and action space designs and effectively improves the supply and demand balance in the entire region. Experiments based on Didi Chuxing order data during a certain time period in Chengdu showed that STFAR increases the cumulative order revenue by 3.64%, increases the completion rate of demand by 4.03%, and increases the dispatched rate of idle vehicles by 2.98% compared with the state-of-the-art algorithms.