Scaling Multi Agent Reinforcement Learning for Underwater Acoustic Tracking via Autonomous Vehicles

TL;DR

This work tackles scaling MARL for underwater acoustic tracking by introducing a GPU-accelerated distillation pipeline (PyLrauv and JaxLrauv) that preserves high-level dynamics from a high-fidelity Gazebo backend. It proposes TransfMAPPO, a Transformer-enhanced MAPPO that yields team-size invariant policies through a latent coordination graph and curriculum learning, enabling effective coordination with up to 5 agents tracking 5 targets. Extensive GPU-based training followed by Gazebo evaluation shows robust tracking with average errors around a few meters, while exposing safety challenges (e.g., collisions) that must be mitigated before real-world deployment. Overall, the paper presents a scalable, end-to-end pipeline that closes the sim-to-real gap for large-scale MARL in autonomous underwater missions, with practical implications for fleet control in challenging marine environments.

Abstract

Autonomous vehicles (AV) offer a cost-effective solution for scientific missions such as underwater tracking. Recently, reinforcement learning (RL) has emerged as a powerful method for controlling AVs in complex marine environments. However, scaling these techniques to a fleet--essential for multi-target tracking or targets with rapid, unpredictable motion--presents significant computational challenges. Multi-Agent Reinforcement Learning (MARL) is notoriously sample-inefficient, and while high-fidelity simulators like Gazebo's LRAUV provide 100x faster-than-real-time single-robot simulations, they offer no significant speedup for multi-vehicle scenarios, making MARL training impractical. To address these limitations, we propose an iterative distillation method that transfers high-fidelity simulations into a simplified, GPU-accelerated environment while preserving high-level dynamics. This approach achieves up to a 30,000x speedup over Gazebo through parallelization, enabling efficient training via end-to-end GPU acceleration. Additionally, we introduce a novel Transformer-based architecture (TransfMAPPO) that learns multi-agent policies invariant to the number of agents and targets, significantly improving sample efficiency. Following large-scale curriculum learning conducted entirely on GPU, we perform extensive evaluations in Gazebo, demonstrating that our method maintains tracking errors below 5 meters over extended durations, even in the presence of multiple fast-moving targets. This work bridges the gap between large-scale MARL training and high-fidelity deployment, providing a scalable framework for autonomous fleet control in real-world sea missions.

Figures (12)

• Figure 1: Overview of our training and evaluation pipeline. PyLrauv$(a)$ is a new Python package to control multiple robots in the C++ high-fidelity LRAUV simulator. JaxLrauv$(b)$ is a GPU-accelerated, simplified environment that supports massive parallelization while preserving the dynamics of the LRAUV simulator. TransfMAPPO$(c)$ employs transformers to train progressively larger fleets of vehicles to coordinate via curriculum learning. The final policies trained in JaxLrauv with TransfMAPPO are then evaluated in the realistic LRAUV simulator prior to real-world deployment.
• Figure 2: Five agents trained in the GPU simplified environment follow in Gazebo simulator five fast targets over several kms. https://mttga.github.io/posts/pylrauv/images/5v5.gif
• Figure 3: Overview of TransfMAPPO architecture.
• Figure 4: Overview of our Curriculum Learning Procedure.
• Figure 5: Training multiple agents to track a very fast target.