Obstacle-aware navigation of smart microswimmers in a turbulent flow

Abstract

Microswimmers in turbulent flows often navigate complex, heterogeneous, and obstacle-rich environments, where they exhibit intricate behaviors such as trapping at and escape from obstacles. We generalize recent $\mathcal{Q}-$learning methods of J.K. Alageshan \textit{et al.} [Phys.Rev.E \textbf{101}, 043110 (2020)] and A. Gupta \textit{et al.} [Physics of Fluids \textbf{37}, 045107 (2025)] developed for non-interacting microswimmers that aim to move optimally from an initial position to a target, to account for the additional complication of an obstacle in the flow. We begin by considering one circular obstacle in forced two-dimensional (2D) Navier-Stokes turbulence in which the energy spectrum displays a forward cascade. We employ the volume-penalization method to introduce this obstacle within our doubly periodic simulation domain. We augment our adversarial $\mathcal{Q}-$learning Refs.~\cite{Alageshan_2020,Akanksha_2025} by suppressing the tendency of microswimmers to get trapped in stagnation points in the vicinity of the obstacle. We demonstrate that smart microswimmers ($SS$), which adopt our obstacle-aware adversarial $\mathcal{Q}-$learning strategy, outperform both naïve swimmers ($NS$) and surfers ($SuS$).

Figures (8)

• Figure 1: (a) - (b) Plots showing the fluid velocity along lines intersecting the obstacle. Vertical dashed lines indicate the obstacle boundaries.(a) Velocity profile along the line $y = \pi + 1$. The velocity of fluid goes to vanishes where the obstacle is present. (b) Velocity profile along the line $x = \pi$. In both plots, different colors represent the velocity field $\bm {u}$ sampled every 10,000 time steps. (c) Schematic diagram of microswimmer interaction with the obstacle. Red dots mark swimmers near the obstacle. This illustrates typical reorientation and detachment behavior, consistent with the results below.
• Figure 2: (a) provides a visual snapshot of microswimmers within a turbulent fluid. Here, $\Omega_s$ represents the solid, volume-penalized domain, while $\Omega_f$ denotes the fluid domain. The root mean squared vorticity, $\omega_0$, serves as a threshold used to determine the state of the microswimmer. The naïve swimmer is depicted as a white dot, while the smart swimmer is shown as a black dot. The distances between the naïve and smart swimmers and the target are denoted by $r_1$ (white dotted line) and $r_2$ (black dotted line), respectively. The symbol $\hat{T}$ denotes the swimmer’s direction relative to the target and acts as the control direction. The naïve swimmer constantly reorients itself to point directly toward the target and moves along that line, with the purple arrow indicating this orientation, as opposed to to a smart swimmer that learns to orient itself to the target through the reinforcement process. The target is positioned at $(\tilde{x}, \tilde{y}) = (0.1704, 0.5)$, while the swimmers begin at $(\tilde{x}, \tilde{y}) = (0.5795, 0.5)$. The coordinates $(\tilde{x}, \tilde{y})$ are non-dimensionalized positions as $(x/L_x, y/L_y)$ respectively. This specific arrangement was selected to create a scenario where the distance between the starting point and the target is shorter when navigating through the obstacle, but longer if the swimmers attempt to bypass the boundary. This setup provides an opportunity to observe the path-planning strategies employed by the smart swimmers. (b) shows the process of state construction. The current state is defined based on the combination of two main parameters: the fluid vorticity (three states, indicated by the red, yellow, and blue colorbar in subfigure (a)) and the angle relative to the target (denoted as $S_\theta$ represented by dark red, light blue, orange, and green). This approach results in a total of 12 possible states. (c) depicts the Q-learning process as stated in \ref{['machine_learning']}.
• Figure 3: Flow chart of sequence of processes involved in adversarial Q learning scheme.
• Figure 4: Pseudocolor plots of the vorticity $\omega$ with superimposed microswimmer positions. The arrows denote the directions of their velocities at nine representative times, showing the spatiotemporal evolution [see the Video V1 in the Supplemental Information] of the flow and the microswimmers. In the time interval $t = 61 ~\text{to}~ 63$, the swimmers interact with the obstacle and adjusting their alignment to move away from it. The sequence from $t=64 ~\text{to}~68$ illustrates how the swimmers glide along the obstacle, gradually realigning until they fully detach. In the snapshot at $t=69$, new microswimmers are shown attaching to the obstacle, thereby continuing the interaction cycle.
• Figure 5: Plots vs the nondimensionalized time $t/\tau_{\Omega}$; (a) Cumulative sum of the difference between naïve and smart microswimmers reaching the target; (b) the cumulative sum of the rewards obtained.