Distributional multi-agent reinforcement learning for air traffic flow and capacity management in a multiple-airport system

Most flight delays are caused by imbalances between traffic demand and capacity in the airport or in airspace. In the pre-tactical phase, air traffic flow and capacity management (ATFCM) is employed to align traffic demand with air traffic control (ATC) capacity, thereby enabling airlines to conduct more efficient flight operations. This paper proposes an approach that integrates flight schedule optimization (demand management) and airspace fix (i.e., waypoint) capacity setting (capacity management) in the pre-tactical phase, aiming to reduce flight delays and enhance flow stability at the target airport in a multiple-airport system (MAS). The approach is developed within a distributional reinforcement learning framework, where particle swarm optimization (PSO) replaces the traditional epsilon greedy strategy to improve training efficiency. The framework estimates reward quantiles over discrete actions to facilitate effective policy learning (optimizing flight schedules and waypoint capacity settings). Our distributional reinforcement learning framework incorporates two centralized agents, namely a flight agent and a waypoint agent, which collaborate via a shared reward function. A case study based on the MAS in the Greater Bay Area in Guangdong-Hong Kong-Macao demonstrates that limited adjustments to flight schedules, together with optimal waypoint capacity settings, can significantly reduce flight delays and ensure safe and steady traffic flow at target airport. The results also demonstrate that the proposed method achieves better performance in terms of reward acquisition, characterized by reduced flight delays and more stable target airport traffic flow, compared to traditional reinforcement learning and heuristic approaches.