Optimal navigation in two-dimensional flows: control theory and reinforcement learning

TL;DR

This work compares optimal control and reinforcement learning for time-minimizing navigation of an active agent in 2D flows ranging from steady to fully turbulent. While optimal control yields exact trajectories in simple flows, their instability in chaotic regimes motivates RL, with both Q-learning and one-step actor-critic delivering robust navigation across tested flows. A key finding is that RL agents trained on coarse-grained, large-scale flow information generalize effectively to the full turbulent field, enabling practical deployment under incomplete information. The results highlight the complementary roles of theory and data-driven learning for autonomous navigation in realistic oceanic and atmospheric environments, and point to future extensions using deep RL and multi-objective criteria.

Abstract

Zermelo's navigation problem seeks the trajectory of minimal travel time between two points in a fluid flow. We address this problem for an agent -- such as a micro-robot or active particle -- that is advected by a two-dimensional flow, self-propels at a fixed speed smaller than or comparable to the characteristic flow velocity, and can steer its direction. The flows considered span increasing levels of complexity, from steady solid-body rotation to the Taylor-Green flow and fully developed turbulence in the inverse cascade regime. Although optimal control theory provides time-minimizing trajectories, these solutions become unstable in chaotic regimes realized for complex background flows. To design robust navigation strategies under such conditions, we apply reinforcement learning. Both action-value (Q-learning) and policy-gradient (one-step actor-critic) methods achieve successful navigation with comparable performance. Crucially, we show that agents trained on coarse-grained turbulent flows -- retaining only large-scale features -- generalize effectively to the full velocity field. This robustness to incomplete flow information is essential for practical navigation in real-world oceanic and atmospheric environments.

Figures (18)

• Figure 1: Energy spectrum for DNS of 2d turbulence.
• Figure 2: Optimal control solution (white line) for the steady vortex flow, see Sections \ref{['sec:Ex_A']}. Near-optimal trajectories are shown in cyan and blue for the initial control angle $\theta_0 \pm \pi/360$, and in magenta and red for the initial position error $\Delta = 0.025$ in $x_A$ and $y_A$.
• Figure 3: Optimal control solution (white line) for the steady sink and time-varying vortex flow, see Sections \ref{['sec:Ex_B']}. Near-optimal trajectories are shown in cyan and blue for the initial control angle $\theta_0 \pm \pi/360$, and in magenta and red for the initial position error $\Delta = 0.025$ in $x_A$ and $y_A$.
• Figure 4: Optimal control solution (white line) for the Taylor-Green flow, see Sections \ref{['sec:Ex_C']}. Perturbed trajectories are shown in cyan and blue for the initial control angle $\theta_0 \pm \pi/9000$, and in magenta and red for the initial position error $\Delta = 10^{-3}$ in $x_A$ and $y_A$.
• Figure 5: Normalized FTLEs $\lambda_i T$ for the optimal trajectory found in the Taylor-Green flow.