Make Your AUV Adaptive: An Environment-Aware Reinforcement Learning Framework For Underwater Tasks

TL;DR

The paper addresses the challenge of robust AUV operation in variable ocean environments by proposing an environment-aware reinforcement learning framework. It introduces a PINN-based environment module to embed flow-field information into the RL state, and an LLM-driven mechanism to iteratively optimize the AUV's structure, enabling co-optimization of control policies and morphology. Through three experiments (normal and complex conditions, plus target tracking) and comparisons with traditional RL, the approach shows improved data collection, tracking accuracy, and significant gains in data rate and energy efficiency, validating the framework's adaptability and robustness. The work highlights the practical impact of coupling environment sensing with structural optimization to enhance underwater autonomy.

Abstract

This study presents a novel environment-aware reinforcement learning (RL) framework designed to augment the operational capabilities of autonomous underwater vehicles (AUVs) in underwater environments. Departing from traditional RL architectures, the proposed framework integrates an environment-aware network module that dynamically captures flow field data, effectively embedding this critical environmental information into the state space. This integration facilitates real-time environmental adaptation, significantly enhancing the AUV's situational awareness and decision-making capabilities. Furthermore, the framework incorporates AUV structure characteristics into the optimization process, employing a large language model (LLM)-based iterative refinement mechanism that leverages both environmental conditions and training outcomes to optimize task performance. Comprehensive experimental evaluations demonstrate the framework's superior performance, robustness and adaptability.

Figures (7)

• Figure 1: Illustration of AUV conducting underwater tasks using our proposed environment-aware RL framework. We jointly optimize the structure and control policy of the AUV through RL training integrated with its onboard environment-aware module, thereby enhancing its adaptivity in the complex ocean environment.
• Figure 2: The overall architecture of our proposed environment-aware RL framework, which comprises of three modules: (A) Environment-aware module. (B) RL-based training module. (C) LLM-based AUV structure optimization module.
• Figure 3: The workflow of the LLM-based AUV structure optimization module, which demonstrates the interaction between the user and the task-analysis agent, task-implementation agent and code-generation agent, ultimately leading to the iterative optimization of the AUV structure design.
• Figure 4: Performance Metrics Comparison: Three generations of AUVs under our proposed RL framework vs. traditional RL framework: (a) Cumulative Reward (CR). (b) Sum Data Rate (SDR). (c) Energy Consumption (EC).
• Figure 5: Schematic diagram illustrating the iterative evolution of three generations of AUV structural designs and their corresponding surrounding flow field environments: (a) 1st Generation: Capsule-structured design; (b) 2nd Generation: Ice cream cone-structured configuration; (c) 3rd Generation: Teardrop-structured morphology.