Transfer learning strategies for accelerating reinforcement-learning-based flow control

TL;DR

This work tackles the computational bottleneck of DRL-based flow control by leveraging transfer learning across fidelity levels in a chaotic KS setting. It systematically benchmarks conventional fine-tuning against Progressive Neural Networks (PNNs), showing that fine-tuning can hasten convergence but is fragile to overfitting and fidelity gaps, while PNNs provide robust, knowledge-preserving transfer with steady performance gains. APS analysis reveals how PNNs reuse low-level features from the source while progressively adapting higher-level representations to the target task, even under regime shifts or inconsistent objectives. The findings suggest that structured transfer learning via PNNs can enable robust, scalable, and efficient flow control applicable to more complex or turbulent flows. The work also highlights practical guidelines for selecting transfer strategies and pretraining durations to maximize speed and reliability in multifidelity DRL for fluid dynamics.

Abstract

This work investigates transfer learning strategies to accelerate deep reinforcement learning (DRL) for multifidelity control of chaotic fluid flows. Progressive neural networks (PNNs), a modular architecture designed to preserve and reuse knowledge across tasks, are employed for the first time in the context of DRL-based flow control. In addition, a comprehensive benchmarking of conventional fine-tuning strategies is conducted, evaluating their performance, convergence behavior, and ability to retain transferred knowledge. The Kuramoto-Sivashinsky (KS) system is employed as a benchmark to examine how knowledge encoded in control policies, trained in low-fidelity environments, can be effectively transferred to high-fidelity settings. Systematic evaluations show that while fine-tuning can accelerate convergence, it is highly sensitive to pretraining duration and prone to catastrophic forgetting. In contrast, PNNs enable stable and efficient transfer by preserving prior knowledge and providing consistent performance gains, and are notably robust to overfitting during the pretraining phase. Layer-wise sensitivity analysis further reveals how PNNs dynamically reuse intermediate representations from the source policy while progressively adapting deeper layers to the target task. Moreover, PNNs remain effective even when the source and target environments differ substantially, such as in cases with mismatched physical regimes or control objectives, where fine-tuning strategies often result in suboptimal adaptation or complete failure of knowledge transfer. The results highlight the potential of novel transfer learning frameworks for robust, scalable, and computationally efficient flow control that can potentially be applied to more complex flow configurations.

Figures (32)

• Figure 1: Schematic of a single neuron as the fundamental unit of a feedforward neural network. Each input $h_{(ij)}$ is multiplied by a weight $w{ij}$, combined with a single bias $b_{i}$, and passed through a nonlinear activation function $\sigma(\cdot)$ to produce the output $h_{i+1}$. Here, the output of the neuron is therefore $h_{21} = \sigma \left( w_{11} h_{11} + w_{12} h_{12} +w_{13} h_{13} + b_1 \right)$.
• Figure 2: Simplified schematic of the DRL framework. A policy network observes the environment state, selects actions, and updates its parameters based on reward feedback.
• Figure 3: Conceptual illustration of the multifidelity DRL framework and its potential for computational acceleration. The top diagram shows knowledge transfer between low- and high-fidelity environments. The bottom panel depicts potential savings in training time through transfer learning. The CFD environments are represented by solutions of the Kuramoto–Sivashinsky (KS) equation, described in Section \ref{['sec:KS']}, using different spatial discretizations.
• Figure 4: Multifidelity fine-tuning-based transfer learning strategies evaluated in this study. Each box represents a layer of the neural network architecture, corresponding to an input, hidden, or output layer with a specified number of neurons. (\ref{['fig:MF_Strategies_scratch']}) Training from scratch on the high-fidelity target environment. (\ref{['fig:MF_Strategies_finetune_all']}) Fine-tuning all layers of the pretrained low-fidelity model. (\ref{['fig:MF_Strategies_finetune_last']}) Fine-tuning only the final output layer. (\ref{['fig:MF_Strategies_finetune_lastTwo']}) Fine-tuning the final two layers. (\ref{['fig:MF_Strategies_finetune_newLayer_1_freeze']}) Adding a new hidden layer and fine-tuning only the new and output layers, with the original layers frozen. (\ref{['fig:MF_Strategies_finetune_newLayer_1']}) Adding a new hidden layer and fine-tuning all layers. (\ref{['fig:MF_Strategies_finetune_newLayer_2']}) Adding two new layers and fine-tuning all layers. (\ref{['fig:MF_Strategies_finetune_newLayer_3']}) Adding three new layers and fine-tuning all layers. Trainable layers are outlined in blue and frozen layers in red. Layers are either initialized from the low-fidelity model or randomly.
• Figure 5: Simplified illustration of the progressive neural network mechanism. Features from a frozen previous column (red) are passed through lateral connections into the trainable new column (blue), enabling transfer of previously learned representations without modifying the original parameters.