Realistic Curriculum Reinforcement Learning for Autonomous and Sustainable Marine Vessel Navigation

TL;DR

This work tackles sustainable autonomous maritime navigation by integrating Curriculum Reinforcement Learning with a data-driven AIS-based simulator, a diffusion-model traffic generator, and a machine-learning fuel predictor. The framework uses a multi-objective reward to jointly optimize safety, emissions, timeliness, and goal completion, and employs PPO with a CNN-based image representation of the environment. A diffusion model enriches traffic scenarios, while XGBoost provides realistic fuel-consumption feedback, enabling emission-aware planning. Experiments in a congested Indian Ocean region demonstrate that CRL achieves superior trade-offs between cumulative reward, safety, and fuel usage compared with state-of-the-art DRL baselines, with curriculum learning significantly improving stability and convergence. The approach offers a principled, scalable path toward safer and more sustainable autonomous vessel operations.

Abstract

Sustainability is becoming increasingly critical in the maritime transport, encompassing both environmental and social impacts, such as Greenhouse Gas (GHG) emissions and navigational safety. Traditional vessel navigation heavily relies on human experience, often lacking autonomy and emission awareness, and is prone to human errors that may compromise safety. In this paper, we propose a Curriculum Reinforcement Learning (CRL) framework integrated with a realistic, data-driven marine simulation environment and a machine learning-based fuel consumption prediction module. The simulation environment is constructed using real-world vessel movement data and enhanced with a Diffusion Model to simulate dynamic maritime conditions. Vessel fuel consumption is estimated using historical operational data and learning-based regression. The surrounding environment is represented as image-based inputs to capture spatial complexity. We design a lightweight, policy-based CRL agent with a comprehensive reward mechanism that considers safety, emissions, timeliness, and goal completion. This framework effectively handles complex tasks progressively while ensuring stable and efficient learning in continuous action spaces. We validate the proposed approach in a sea area of the Indian Ocean, demonstrating its efficacy in enabling sustainable and safe vessel navigation.

Figures (8)

• Figure 1: IMO’s GHG reduction pathway.
• Figure 2: Overview of the methodological workflow in this study.
• Figure 3: Distribution of vessel features
• Figure 4: Architecture of the actor network in CRL.
• Figure 5: Vessel navigation trajectory for Instance Case 1.