FreqMoE: Dynamic Frequency Enhancement for Neural PDE Solvers

TL;DR

This work tackles the high-frequency signal loss inherent to Fourier Neural Operators (FNO) caused by fixed spectral truncation. It introduces FreqMoE, a sparse frequency-domain mixture-of-experts framework that dynamically processes high-frequency components while reusing low-frequency patterns through a LoRA-inspired, low-rank upcycling of pre-trained weights, enabling parameter-efficient adaptation. The Low-frequency Pretraining, High-frequency Fine-tuning (LPHF) paradigm, coupled with sparse gating and TopK activation, yields up to 16.6% accuracy gains with orders-of-magnitude fewer parameters across both regular and irregular grids and enhances long-term stability. The approach generalizes across FNO variants, supporting high-resolution PDE solving with reduced computational cost and providing a practical pathway toward frequency-aware neural PDE solvers.

Abstract

Fourier Neural Operators (FNO) have emerged as promising solutions for efficiently solving partial differential equations (PDEs) by learning infinite-dimensional function mappings through frequency domain transformations. However, the sparsity of high-frequency signals limits computational efficiency for high-dimensional inputs, and fixed-pattern truncation often causes high-frequency signal loss, reducing performance in scenarios such as high-resolution inputs or long-term predictions. To address these challenges, we propose FreqMoE, an efficient and progressive training framework that exploits the dependency of high-frequency signals on low-frequency components. The model first learns low-frequency weights and then applies a sparse upward-cycling strategy to construct a mixture of experts (MoE) in the frequency domain, effectively extending the learned weights to high-frequency regions. Experiments on both regular and irregular grid PDEs demonstrate that FreqMoE achieves up to 16.6% accuracy improvement while using merely 2.1% parameters (47.32x reduction) compared to dense FNO. Furthermore, the approach demonstrates remarkable stability in long-term predictions and generalizes seamlessly to various FNO variants and grid structures, establishing a new ``Low frequency Pretraining, High frequency Fine-tuning'' paradigm for solving PDEs.

Figures (6)

• Figure 1: Motivation of FreqMoE. Traditional FNO directly truncates high-frequency components (left), while FreqMoE(ours) efficiently preserves them through sparse dynamic experts (right). This design enables high-frequency modeling with negligible computational overhead.
• Figure 2: Methods overview of FreqMoE.(a) The standard Fourier Neural Operator (FNO) architecture consisting of input lifting (P), a sequence of Fourier layers, and output projection (Q). (b) Our modified Fourier layer design with a mixture-of-experts mechanism, where the gating networker dynamically assigns frequency components to specialized experts after FFT decomposition. High-frequency components (lighter shades) are processed by high-frequency experts, while low-frequency components are handled by the base expert. (c) Our expert initialization strategy, where pre-trained weights $R$ are used as a shared base component $R_\text{base}$ and expert-specific delta weights $\Delta R$ are initialized with LoRA trick, enabling efficient parameter sharing and specialized frequency processing.
• Figure 3: Visualization of prediction errors.Left Column: Irregular Grid Results from AirFoil. Right Column: Regular Grid Results from CFD-Turb 512. Red circles highlight regions with high-frequency components, where our FreqMoE demonstrates better capability in capturing fine-grained spatial details compared to FNO.
• Figure 4: Visualization of Experts. (a) Distribution of frequency signals after FFT transformation. (b) Activation patterns of experts in FreqMoE, where each grid cell represents a frequency mode chunk. Beyond capturing low-frequency signals in the top-left corner, FreqMoE dynamically activates experts to capture surrounding high-frequency components.
• Figure 5: Long-term Prediction Performance on Different CFD Datasets. The plots show the L2 relative error evolution during rollout prediction across three datasets of varying complexity. FreqMoE demonstrates superior stability in long-term predictions compared to baseline FNO models with different mode configurations. This advantage becomes particularly pronounced in high-resolution scenarios (CFD-Rand 512 and CFD-Turb 512), where the error growth is significantly moderated.