Yard crane real-time scheduling among multi-block at terminal: A reinforcement learning based proximal policy optimization approach

In container terminal yards, operational efficiency is significantly hindered by the mismatch between stochastic workload variations and static configurations of Yard Cranes (YCs) across multiple blocks. The real-time YC scheduling problem (YCSP-r) aims to develop adaptive, instantaneous scheduling policies that dynamically respond to workload fluctuations. In this paper, we propose a multi-agent reinforcement learning (RL) method to address the YCSP-r. Specifically, the YCSP-r is formulated as a Markov Decision Process (MDP) within an asynchronous timestep framework. Considering the non-negligible redeployment cost of YCs in real-time operations, the MDP is designed to balance redeployment costs with overall operational efficiency. A general simulator for the YC scheduling system is developed to execute action decisions and provides performance feedback. Proximal Policy Optimization (PPO) is employed to train the scheduling policy. A multi-agent shared-policy framework and a global–local mixed state structure is tailored to mitigate the challenges posed by high dimensional state and action spaces, thereby enhancing both convergence and training stability. To evaluate the solution quality, a mixed integer programming model for YCSP-r is developed and solved by a commercial solver as a benchmark for comparison. The proposed approach is further compared with other advanced RL and heuristic methods. Experimental results demonstrate that the proposed PPO-based approach is able to provide high-quality solutions in real time—typically within seconds—meeting the practical demands of container terminal operations. Notably, compared to a static YC deployment strategy, our scheduling strategy achieves a substantial 12.29% reduction in operational costs. We believe that our study provides valuable insights for port managers in developing practical and reliable YC scheduling solutions.