HydroGym: A Reinforcement Learning Platform for Fluid Dynamics

TL;DR

HydroGym tackles the bottlenecks hindering RL in fluid dynamics by delivering a solver-agnostic platform with 42 validated flow-control environments spanning 2D to 3D regimes and variable Reynolds numbers. It integrates differentiable and non-differentiable solvers, enabling gradient-enhanced optimization and scalable multi-agent control, and provides comprehensive benchmarking against standard RL algorithms. The work demonstrates robust transfer learning across configurations, accelerates learning with gradient information, and showcases 3D flow control capabilities including wake manipulation and acoustic disruption. By democratizing access to standardized, high-fidelity fluid benchmarks, HydroGym paves the way for rapid algorithmic advances and broader adoption of AI-driven flow control in engineering practice.

Abstract

Modeling and controlling fluid flows is critical for several fields of science and engineering, including transportation, energy, and medicine. Effective flow control can lead to, e.g., lift increase, drag reduction, mixing enhancement, and noise reduction. However, controlling a fluid faces several significant challenges, including high-dimensional, nonlinear, and multiscale interactions in space and time. Reinforcement learning (RL) has recently shown great success in complex domains, such as robotics and protein folding, but its application to flow control is hindered by a lack of standardized benchmark platforms and the computational demands of fluid simulations. To address these challenges, we introduce HydroGym, a solver-independent RL platform for flow control research. HydroGym integrates sophisticated flow control benchmarks, scalable runtime infrastructure, and state-of-the-art RL algorithms. Our platform includes 42 validated environments spanning from canonical laminar flows to complex three-dimensional turbulent scenarios, validated over a wide range of Reynolds numbers. We provide non-differentiable solvers for traditional RL and differentiable solvers that dramatically improve sample efficiency through gradient-enhanced optimization. Comprehensive evaluation reveals that RL agents consistently discover robust control principles across configurations, such as boundary layer manipulation, acoustic feedback disruption, and wake reorganization. Transfer learning studies demonstrate that controllers learned at one Reynolds number or geometry adapt efficiently to new conditions, requiring approximately 50% fewer training episodes. The HydroGym platform is highly extensible and scalable, providing a framework for researchers in fluid dynamics, machine learning, and control to add environments, surrogate models, and control algorithms to advance science and technology.

Figures (21)

• Figure 1: HydroGym reinforcement learning platform architecture for fluid dynamics control. The platform features a unified interface connecting flow environments with RL controllers through standardized observation-action loops (top), exemplified here by jet-based actuation control of an open cavity flow. HydroGym's key capabilities include: scalable environments from 2D to 3D configurations with systematic Reynolds number variations; transfer learning pathways enabling knowledge transfer across geometric configurations, Reynolds numbers, and dimensionality; and advanced RL methods including fully differentiable physics environments for gradient-enhanced optimization and multi-agent frameworks for spatially distributed control. All environments maintain compatibility with standard RL algorithms while supporting diverse computational backends.
• Figure 2: Comprehensive benchmark suite of HydroGym's 42+ fluid dynamics environments. The collection spans fundamental flow configurations in or around generic and real-world inspired geometries such as cylinders or spheres, airfoils, cavities, and pipes across both two-dimensional and three-dimensional domains. Each environment category includes systematic Reynolds number variations to capture different flow regimes from laminar to turbulent conditions. Special environments include differentiable physics implementations that enable gradient-based optimization methods. This standardized suite provides consistent benchmarking capabilities for systematic evaluation and comparison of reinforcement learning algorithms across diverse active flow control challenges.
• Figure 3: Reinforcement learning performance across two- and three-dimensional flow control environments. Training curves (left) show convergence behavior for PPO, DDPG, and TD3 algorithms across six benchmark scenarios: three 2D environments---fluidic pinball ($Re=100$), open cavity ($Re=4,200$), and cylinder ($Re=3,900$)---and three 3D environments---fluidic pinball ($Re=150$), open cavity ($Re=7,500$), and gust-airfoil interaction ($Re=1,000$). Test rollouts (right) demonstrate control effectiveness through drag reduction metrics, revealing the agents' ability to manage both two-dimensional flow reorganization and complex three-dimensional phenomena including spanwise instabilities, acoustic feedback loops, and turbulent structures. Flow visualizations highlight the sophisticated wake manipulation and flow control achieved through learned policies across increasing geometric complexity. All shown training results leverage the m-AIA solver.
• Figure 4: Transfer learning capabilities across flow configurations and conditions. Four transfer scenarios demonstrate knowledge generalization: Reynolds number scaling ($Re=200 \rightarrow 1,000$, $Re=1000 \rightarrow 3,900$), geometric transfer (circular to square cylinder), and dimensional scaling (2D to 3D cylinder). Training curves compare standard PPO training with transfer learning (finetuned and zeroshot). Finetuned policies consistently achieve faster convergence, requiring approximately half the training episodes of baseline methods. Test rollouts confirm that improved training efficiency translates to effective control performance, with drag reduction across all transfer scenarios.
• Figure 5: Advanced Reinforcement Learning. (Top) Gradient-enhanced policy optimization through differentiable environments substantially improves sample efficiency and the overall reward compared to gradient-free methods. JAX-based solvers enable automatic differentiation through complete simulation trajectories, providing exact policy gradients ($\partial s / \partial \alpha$). Training curves compare gradient-enhanced PPO (GPPO) against standard PPO for 2D Kolmogorov flow and 3D channel flow. (Bottom) Multi-agent reinforcement learning for spatially distributed control. Architecture decomposes a 3D cylinder environment at $Re=3,900$ into multiple cooperative agents, each managing local actuator pairs while sharing gradient information and experience buffers. This distributed approach scales efficiently with actuator count and enables parallel exploration of high-dimensional control spaces. Training performance and test rollouts demonstrate effective learning and substantial drag reduction through coordinated spanwise actuation.