Attention on flow control: transformer-based reinforcement learning for lift regulation in highly disturbed flows

TL;DR

This work tackles lift regulation of a flat plate in highly disturbed gust environments where disturbances reset the flow and measurements are limited to surface pressures. It proposes a transformer-based reinforcement learning framework (PPO) to infer a belief state from a window of observations and to select pitching accelerations, with the policy pretrained from a proportional controller and further improved by task-level transfer learning. The results show that the RL controller can outperform proportional control, particularly in multi-gust scenarios, and that quarter-chord pivoting provides superior lift regulation with reduced actuation due to the added-mass term $C_{L_{\ddot{\alpha}}}$. The combination of transformer-based belief-state estimation, pretraining, and transfer learning enables rapid convergence and generalization to arbitrarily long gust sequences, offering a practical path toward robust gust-resilient lift control in highly disturbed flows.

Abstract

A linear flow control strategy designed for weak disturbances may not remain effective in sequences of strong disturbances due to nonlinear interactions, but it is sensible to leverage it for developing a better strategy. In the present study, we propose a transformer-based reinforcement learning (RL) framework to learn an effective control strategy for regulating aerodynamic lift in arbitrarily long gust sequences via pitch control. The random gusts produce intermittent, high-variance flows observed only through limited surface pressure sensors, making this control problem inherently challenging compared to stationary flows. The transformer addresses the challenge of partial observability from the limited surface pressures. We demonstrate that the training can be accelerated with two techniques -- pretraining with an expert policy (here, linear control) and task-level transfer learning (here, extending a policy trained on isolated gusts to multiple gusts). We show that the learned strategy outperforms the best proportional control, with the performance gap widening as the number of gusts increases. The control strategy learned in an environment with a small number of successive gusts is shown to effectively generalize to an environment with an arbitrarily long sequence of gusts. We investigate the pivot configuration and show that quarter-chord pitching control can achieve superior lift regulation with substantially less control effort compared to mid-chord pitching control. Through a decomposition of the lift, we attribute this advantage to the dominant added-mass contribution accessible via quarter-chord pitching.

Figures (22)

• Figure 1: Control problem description. We only show the pivot point located at the mid chord for simplicity, shown as grey dot, but a quarter chord pivot point configuration is also investigated in this study. The pressure sensors are denoted as the red dots. The colorbar shown here is used throughout this study.
• Figure 2: An example of environment with three successive gust under no control. $t^*$ is the convective time. The specific gust parameters, for this example, by the order of $(D_y,y_0/c,\sigma/c,U_\infty t_0/c)$ is: $(-1.0,0.25,0.1,0.5)$, $(-0.5,0.25,0.05,1.0)$, $(1.0,-0.1,0.075,1.5)$.
• Figure 3: (a) Comparison between MDP and POMDP. (b) The schematic figure of transformer and the subsequent policy and value networks.
• Figure 4: Overview of the transformer-based RL framework
• Figure 5: Overview of the transfer learning.