Transfer Learning in Physics-Informed Neural Networks: Full Fine-Tuning, Lightweight Fine-Tuning, and Low-Rank Adaptation

TL;DR

This paper addresses the limited generalization of Physics-Informed Neural Networks (PINNs) when problem conditions change, by systematically evaluating transfer learning strategies—full fine-tuning, lightweight fine-tuning, and Low-Rank Adaptation (LoRA)—in both strong-form and energy-form PINNs across boundary conditions, materials, and geometries. It demonstrates that full fine-tuning and LoRA typically accelerate convergence and can slightly improve accuracy, while lightweight fine-tuning often underperforms due to insufficient adaptation capacity and computation-graph considerations in PINNs. The study analyzes rank selection in LoRA, showing that higher domain mismatch requires larger rank, and introduces a similarity-based heuristic for rank guidance, along with scenario fusion as a pathway to improved accuracy. Overall, the results suggest transfer learning as a practical tool to adapt PINNs to related PDE problems efficiently, with promising directions including automatic rank determination, adaptive per-layer LoRA, and integration with operator-learning frameworks.

Abstract

AI for PDEs has garnered significant attention, particularly Physics-Informed Neural Networks (PINNs). However, PINNs are typically limited to solving specific problems, and any changes in problem conditions necessitate retraining. Therefore, we explore the generalization capability of transfer learning in the strong and energy form of PINNs across different boundary conditions, materials, and geometries. The transfer learning methods we employ include full finetuning, lightweight finetuning, and Low-Rank Adaptation (LoRA). The results demonstrate that full finetuning and LoRA can significantly improve convergence speed while providing a slight enhancement in accuracy.

Figures (25)

• Figure 1: The strong form of Physics-informed neural networks (PINNs). The inputs $x_{1}$, $x_{2}$, $\cdots$, $t$ are typically spatial coordinates, while the outputs $y_{1}$, $\cdots$, $y_{n}$ are the network's outputs, usually representing the variables of interest.
• Figure 2: The energy form of Physics-informed neural networks (PINNs). The inputs $x$, $y$, $z$ are typically spatial coordinates, while the outputs $U_{x}$, $U_{y}$, and $U_{z}$ represent the displacement field $\boldsymbol{u}$. The admissible displacement $\tilde{\boldsymbol{u}}(\boldsymbol{x};\boldsymbol{\theta})$ is the displacement field $\boldsymbol{u}$ that satisfies the essential boundary conditions in advance.
• Figure 3: Categories of Transfer Learning zhuang2020comprehensive
• Figure 4: Three common methods for Parameter-based transfer learning: (a) Full fine-tuning: fine-tune all parameters of the model, with the red arrows indicating the parameters that need fine-tuning. (b) Lightweight fine-tuning: fine-tune a subset of the model’s parameters, with the red arrows indicating the parameters that need fine-tuning. (c) LoRA: the blue matrix $\boldsymbol{W}$ represents fixed parameters (pretrained parameters from the previous task). The yellow matrices are the trainable parameters $\boldsymbol{A}$ and $\boldsymbol{B}$, where $\boldsymbol{AB}$ is a low-rank matrix with rank $r$. During training on the new dataset, only $\boldsymbol{A}$ and $\boldsymbol{B}$ are adjusted. $\boldsymbol{W}^{*} = \boldsymbol{W} + \boldsymbol{A} \boldsymbol{B}$ represents the neural network weights during testing.
• Figure 5: Introduction to the Taylor-Green vortex: the velocity vector field at different values of frequency $w$. The flow pattern of the Taylor-Green vortex typically involves multiple vortex structures, where the size and distribution of the vortices exhibit high symmetry. Over time, the vortices decay and eventually disappear.