CoLoRA: Continuous low-rank adaptation for reduced implicit neural modeling of parameterized partial differential equations

TL;DR

CoLoRA introduces Continuous Low-Rank Adaptation to reduce implicit neural approximations of parameterized PDEs by continuously modulating a small set of online low-rank weights through time. By treating time as a separate online latent state and leveraging a hyper-network or Neural Galerkin variational approach, CoLoRA delivers nonlinear, causally consistent reduced models that overcome linear Kolmogorov barrier limitations while remaining data-efficient. The framework achieves orders-of-magnitude speedups over full-order models and outperforms competing nonlinear surrogates in accuracy and parameter efficiency, with the option to preserve physical quantities via equation-driven online predictions. These properties make CoLoRA particularly well-suited for data-scarce regimes, rapid parameter sweeps, and physics-informed forecasting in transport-dominated PDEs.

Abstract

This work introduces reduced models based on Continuous Low Rank Adaptation (CoLoRA) that pre-train neural networks for a given partial differential equation and then continuously adapt low-rank weights in time to rapidly predict the evolution of solution fields at new physics parameters and new initial conditions. The adaptation can be either purely data-driven or via an equation-driven variational approach that provides Galerkin-optimal approximations. Because CoLoRA approximates solution fields locally in time, the rank of the weights can be kept small, which means that only few training trajectories are required offline so that CoLoRA is well suited for data-scarce regimes. Predictions with CoLoRA are orders of magnitude faster than with classical methods and their accuracy and parameter efficiency is higher compared to other neural network approaches.

Figures (7)

• Figure 1: LoRA fine-tunes networks to downstream tasks by adapting low-rank matrices $\boldsymbol A \boldsymbol B$. Our CoLoRA introduces a scaling $\alpha(t, \boldsymbol {\mu})$ on the low-rank matrix $\boldsymbol A \boldsymbol B$ to adapt networks continuously to predict PDE solution trajectories.
• Figure 2: Left: Shows that CoLoRA's latent states $\boldsymbol {\phi}(t; \boldsymbol {\mu})$ adapt smoothly over time (RDE example). Middle: Training a CoLoRA model with a $q = 2$-dimensional latent state on the Burgers' example gives the first latent component corresponding to translation in time and the second one to the viscosity $\mu$. Right: CoLoRA learns a continuous region of low PDE residual along which the latent trajectories evolve (Vlasov example); see Appendix \ref{['appx:num']}.
• Figure 3: CoLoRA models achieve orders of magnitude lower errors than the best-approximation error of linear model reduction methods, which is in agreement with Section \ref{['sec:CoLoRANWidth']} that states that CoLoRA parameterizations circumvent the Kolmogorov barrier.
• Figure 4: Left: Purely data-driven CoLoRA (CoLoRA-D) is more than four orders of magnitude faster than traditional numerical models. If the governing equations are solved with Neural Galerkin bruna_neural_2024 in a Galerkin-optimal variational sense in the CoLoRA parameterization (CoLoRA-EQ), we still obtain about two orders of magnitude speedups while maintaining causality in the solution. Right: CoLoRA is more data efficient than operator learning and thus well suited for low-data regimes (Burgers', Vlasov).
• Figure 5: Solving the governing equations in a variational sense with Neural Galerkin bruna_neural_2024 and CoLoRA parameterizations (CoLoRA-EQ) leads to causal solutions and allows conserving quantities schwerdtner_nonlinear_2023 such as mass in the Vlasov problem, which is key for building trust in physics predictions and for interpretability.