A critical assessment of reinforcement learning methods for microswimmer navigation in complex flows

TL;DR

The study formulates a well-posed POMDP benchmark for microswimmer navigation in complex flows, aiming to maximize the vertical displacement $Z$ using only on-board observations. It benchmarks three RL methods—Q-Learning, A2C, and PPO—across three flows (Taylor-Green vortices, ABC, Turbulent 2D flow), showing vanilla methods underperform while PPO achieves near-optimal, robust performance with careful implementation. The learned PPO policy aligns with a generalized surfing strategy, exploiting coherent structures and flow timescales, and in turbulence can match the quasi-optimal baseline. The work emphasizes that algorithm choice and tuning are critical for real-world autonomous navigation in complex, partially observable flows, and points to memory-enabled and off-policy extensions as promising future directions.

Abstract

Navigating in a fluid flow while being carried by it, using only information accessible from on-board sensors, is a problem commonly faced by small planktonic organisms. It is also directly relevant to autonomous robots deployed in the oceans. In the last ten years, the fluid mechanics community has widely adopted reinforcement learning, often in the form of its simplest implementations, to address this challenge. But it is unclear how good are the strategies learned by these algorithms. In this paper, we perform a quantitative assessment of reinforcement learning methods applied to navigation in partially observable flows. We first introduce a well-posed problem of directional navigation for which a quasi-optimal policy is known analytically. We then report on the poor performance and robustness of commonly used algorithms (Q-Learning, Advantage Actor Critic) in flows regularly encountered in the literature: Taylor-Green vortices, Arnold-Beltrami-Childress flow, and two-dimensional turbulence. We show that they are vastly surpassed by PPO (Proximal Policy Optimization), a more advanced algorithm that has established dominance across a wide range of benchmarks in the reinforcement learning community. In particular, our custom implementation of PPO matches the theoretical quasi-optimal performance in turbulent flow and does so in a robust manner. Reaching this result required the use of several additional techniques, such as vectorized environments and generalized advantage estimation, as well as hyperparameter optimization. This study demonstrates the importance of algorithm selection, implementation details, and fine-tuning for discovering truly smart autonomous navigation strategies in complex flows.

Figures (5)

• Figure 1: Navigation in Taylor-Green vortices (TGV). The flow is represented in the upper left corner by showing the (out-of-plane component of the) vorticity ($\bm{\omega} = \boldsymbol{\nabla} \times \bm{u}$), along with streamlines. The dashed line represents the initialization of particles whose trajectories are shown in the bottom panel, for particles following the naive strategy (left) and the learned PPO strategy (right). Unlike naive agents which can be trapped on periodic orbits, PPO agents have learned to escape such trapping and all converge to a single trajectory that yields the largest vertical displacement, independently of their initial location. The PPO policy is compared to the surfing policy in the upper right corner. While both tend to diverge from downflow regions (violet) and converge to upflow regions (orange), PPO does it more aggressively, with steeper changes of direction.
• Figure 2: Navigation in Arnold-Beltrami-Childress flow (ABC). The ABC flow is represented in the upper left corner by showing the total distance travelled by passive tracers advected by the flow (image generated with https://github.com/auroreloisy/ldflow based on the LDDS package LDDS). Such quantity, called Lagrangian descriptor LagrangianDescriptors, highlights flow regions with qualitatively different dynamics. This flow contains many coherent tube-like structure (light yellow) where tracers tend to cover large distances, these areas are also associated with preferential directions. They are separated by a chaotic region (dark red). In the bottom panels where trajectories are shown, agents are initialized at the $z=0$ plane, following the naive strategy (left) and the PPO strategy (right). PPO agents have learned to converge to a particular flow structure, characterized by large upward transport. In the upper right corner, the surfing and PPO policies are projected onto a horizontal plane. The beneficial (detrimental) coherent structures are visible in the background: the orange (purple) one is associated to large upward (downward) displacement of passive tracers. Compared to surfing, PPO orients more aggressively toward the orange structure, which explains its overall superior performance.
• Figure 3: Navigation in a two-dimensional turbulent flow (TURB). A snapshot of the time-dependent turbulence simulation is represented in the upper left: the (out-of-plane) vorticity is shown in the background, along with the streamlines. The snapshot corresponds to a randomly chosen time $t=t_0$ at which agents are initialized on the dashed line, their trajectories are shown in the bottom panel. Compared to the naive strategy (bottom left), the PPO strategy (bottom right) yields trajectories that tend to clump together to benefit from upward flow. This is visible in the policy representation (top right), where PPO is compared to surfing: both tend to diverge from downflow regions (violet) and converge to upflow regions (orange). While not strictly identical, these two policies are very similar to each other.
• Figure 4: Learning curves for each algorithm (columns) in each environment (rows): performance (distance traveled in the vertical direction normalized by $vT$) as a function of the number of episodes used for training. The performance of the naive policy is shown by a black line. To smooth out large episode-to-episode fluctuations and enhance readability, we use a moving averaging window of 1000 episodes. Only the first $10^5$ episodes are shown, training is continued over a total of $10^6$ episodes before evaluating performance in Table \ref{['tab:performance']}.
• Figure 5: Surfing policy: optimization over $\tau$ in the different flows. In TGV $\tau^{*} = 2.0$, in ABC $\tau^{*} = 0.72$, and in TURB $\tau^{*} = 0.23$