Plug-and-Play Benchmarking of Reinforcement Learning Algorithms for Large-Scale Flow Control

TL;DR

FluidGym addresses fragmentation in reinforcement learning for active flow control by providing the first standalone, fully differentiable benchmark suite built in PyTorch on GPU-accelerated solvers. It unifies CFD simulation and RL through the FluidEnv interface, enabling SARL, MARL, and gradient-based control across diverse 2D/3D environments with standardized train/val/test protocols. The paper demonstrates baseline performance using PPO and SAC (and a differentiable MPC) across four environments (Cylinder, RBC, Airfoil, and Turbulent Channel) at multiple difficulty levels, highlighting SAC’s robustness and the potential for transfer learning across dimensionalities and domain sizes. By releasing all environments, data, and models, FluidGym aims to accelerate reproducibility, fair comparison, and progress toward scalable, learning-based flow control in real-world applications.

Abstract

Reinforcement learning (RL) has shown promising results in active flow control (AFC), yet progress in the field remains difficult to assess as existing studies rely on heterogeneous observation and actuation schemes, numerical setups, and evaluation protocols. Current AFC benchmarks attempt to address these issues but heavily rely on external computational fluid dynamics (CFD) solvers, are not fully differentiable, and provide limited 3D and multi-agent support. To overcome these limitations, we introduce FluidGym, the first standalone, fully differentiable benchmark suite for RL in AFC. Built entirely in PyTorch on top of the GPU-accelerated PICT solver, FluidGym runs in a single Python stack, requires no external CFD software, and provides standardized evaluation protocols. We present baseline results with PPO and SAC and release all environments, datasets, and trained models as public resources. FluidGym enables systematic comparison of control methods, establishes a scalable foundation for future research in learning-based flow control, and is available at https://github.com/safe-autonomous-systems/fluidgym.

Figures (39)

• Figure 1: The four uncontrolled environment classes in FluidGym.
• Figure 2: Overview of FluidGym using the 2D Rayleigh–Bénard Convection (RBC) environment. The framework provides three modes of interaction: single-agent RL (SARL), multi-agent RL (MARL), and gradient-based methods. The action space consists of 12 heater actuators along the lower boundary. In SARL, a single agent outputs the full action vector, whereas in MARL, each agent controls one actuator via a local action. Local actions are internally aggregated and mapped to boundary actuation values via the transformation function $\Gamma$. Gray dots indicate virtual sensor locations: in SARL, the agent receives all measurements, while in MARL, each agent observes only the local subset around its assigned actuator (denoted by the window framed in purple).
• Figure 3: Final 3D flow fields at the end of test episodes for uncontrolled and controlled cases across four FluidGym environments using PPO, SAC, or multi-agent variants. Transfer cases use policies trained on corresponding 2D or smaller domains.
• Figure 4: Left: Performance profiles as proposed by agarwall-neurips21 summarizing scores over all FluidGym environments. Error bars indicate pointwise 95% confidence intervals based on $2\,\text{k}$ stratified bootstrap replications across random seeds. Right: Interquartile mean (IQM) scores over environment classes (middle) and difficulty levels (right). For all panels, scores are computed as min–max normalized relative improvements over the baseflow, with normalization performed independently for each environment–difficulty pair.
• Figure 5: Time evolution of the control action $a_t$ and drag coefficient $C_D$ for the uncontrolled baseflow, the final PPO and SAC policies, and the differentiable model predictive control (D-MPC, see Algorithm \ref{['alg:d-mpc_opt']} in Appendix \ref{['sec:appendix:app_experimental_setup']}) controller evaluated on the CylinderJet2D-easy-v0 test environment.