Ocean Diviner: A Diffusion-Augmented Reinforcement Learning Framework for AUV Robust Control in Underwater Tasks

TL;DR

The paper tackles robust AUV control under nonlinear dynamics, disturbances, and localization uncertainty. It introduces a diffusion-augmented RL framework that combines diffusion-based action generation with a high-dimensional state encoding via a diffusion U-Net and a hybrid RL-diffusion policy (TD3), supplemented by an S-Surface low-level controller. The approach yields physically feasible, high-quality actions and sample-efficient learning with improved long-horizon planning, outperforming conventional controllers under dynamic marine conditions in simulations. This framework enhances adaptability and reliability for underwater tasks, with plans to release code to accelerate further research and real-world validation.

Abstract

Autonomous Underwater Vehicles (AUVs) are essential for marine exploration, yet their control remains highly challenging due to nonlinear dynamics and uncertain environmental disturbances. This paper presents a diffusion-augmented Reinforcement Learning (RL) framework for robust AUV control, aiming to improve AUV's adaptability in dynamic underwater environments. The proposed framework integrates two core innovations: (1) A diffusion-based action generation framework that produces physically feasible and high-quality actions, enhanced by a high-dimensional state encoding mechanism combining current observations with historical states and actions through a novel diffusion U-Net architecture, significantly improving long-horizon planning capacity for robust control. (2) A sample-efficient hybrid learning architecture that synergizes diffusion-guided exploration with RL policy optimization, where the diffusion model generates diverse candidate actions and the RL critic selects the optimal action, achieving higher exploration efficiency and policy stability in dynamic underwater environments. Extensive simulation experiments validate the framework's superior robustness and flexibility, outperforming conventional control methods in challenging marine conditions, offering enhanced adaptability and reliability for AUV operations in underwater tasks. Finally, we will release the code publicly soon to support future research in this area.

Figures (7)

• Figure 1: Illustration of AUV control using the proposed framework. This framework utilizes a diffusion model for action generation, while RL's critic network selects the action that gains the maximum Q value from candidates as the optimal action to execute.
• Figure 2: Architecture of proposed framework for AUV robust control. This framework consists of two components: (A) Diffusion-Based Actions Generation and (B) Hybrid RL-Diffusion Policy.
• Figure 3: Visualization of five candidate actions across three denoising stages under the Diffusion+RL+S-Surface framework. Trajectories evolve from scattered exploration to smooth, task-aligned paths, demonstrating diffusion’s ability to generate diverse yet optimized plans.
• Figure 4: Performance comparison between diffusion+RL and standard RL across underwater data collection task metrics under ideal, ES and VES conditions, respectively.
• Figure 5: Tracking performance of Diffusion+RL with S-Surface, PID, and SMC controllers. S-Surface achieves the most accurate and stable tracking, with the lowest yaw and depth errors. (a) Yaw and depth trajectories. (b) Yaw and depth tracking errors (MSE).