A multi-objective hierarchical reinforcement learning framework for energy consumption optimization of wing sail-assisted ships

Improving the energy efficiency of oceangoing vessels and reducing fuel consumption represent a significant engineering challenge. For hybrid-powered ships, the energy efficiency optimization involves complex decisions and multiple competing objectives, where conventional reinforcement learning algorithms often suffer from performance imbalance. To address this, this study proposes a Multi-objective Hierarchical Reinforcement Learning (MHRL) framework for sail-diesel hybrid vessels, aiming to achieve coordinated optimization of ship speed and sail control under dynamic sea conditions. A data-driven approach is first adopted to develop a deep neural network model for predicting fuel consumption and speed, establishing a baseline for optimization. The MHRL framework employs a hierarchical architecture to decouple objectives and the action space into two sub-tasks: a high-level agent optimizes main engine speed to balance voyage schedule and energy consumption, while a low-level agent adjusts sail angle and deployment to maximize wind energy utilization. Both agents utilize the Proximal Policy Optimization (PPO) algorithm, and the modular low-level agent design allows for extensibility. Validation was conducted on a sail-equipped Very Large Crude Carrier (VLCC) across three representative routes covering various time scales. Results demonstrate that the MHRL framework enables wind sail-assisted ships to achieve a fuel saving rate of 3.66%-5.02%, with the equivalent energy saving from sails increasing with voyage length. Compared to the baseline PPO algorithm, MHRL improves the optimization success rate by 55.55% and demonstrates significant enhancements in energy-saving ratio and convergence speed. Moreover, the framework exhibits strong robustness, effectively solving the multi-objective optimization problem for sail-assisted ships.