Mitigating Spectral Bias in Neural Operators via High-Frequency Scaling for Physical Systems

TL;DR

This work tackles spectral bias in neural operators used as PDE surrogates by introducing high-frequency scaling (HFS), a latent-space patch-based method that separately scales low- and high-frequency components to preserve sharp features in multiscale physical systems. Integrated with ResUNet variants, HFS improves prediction accuracy and energy spectra alignment for two-phase boiling and Kolmogorov flow, with larger gains in localized high-frequency regions like bubble interfaces. The study also explores a diffusion-model refinement conditioned on neural-operator outputs, showing substantial spectral improvements when priors are reliable, albeit at higher training cost. Together, HFS and diffusion-model conditioning offer a practical pathway to accurate, low-cost surrogates for turbulent and multiphase flows, enabling better capture of sharp gradients and small-scale structures in complex physics.

Abstract

Neural operators have emerged as powerful surrogates for modeling complex physical problems. However, they suffer from spectral bias making them oblivious to high-frequency modes, which are present in multiscale physical systems. Therefore, they tend to produce over-smoothed solutions, which is particularly problematic in modeling turbulence and for systems with intricate patterns and sharp gradients such as multi-phase flow systems. In this work, we introduce a new approach named high-frequency scaling (HFS) to mitigate spectral bias in convolutional-based neural operators. By integrating HFS with proper variants of UNet neural operators, we demonstrate a higher prediction accuracy by mitigating spectral bias in single and two-phase flow problems. Unlike Fourier-based techniques, HFS is directly applied to the latent space, thus eliminating the computational cost associated with the Fourier transform. Additionally, we investigate alternative spectral bias mitigation through diffusion models conditioned on neural operators. While the diffusion model integrated with the standard neural operator may still suffer from significant errors, these errors are substantially reduced when the diffusion model is integrated with a HFS-enhanced neural operator.

Figures (17)

• Figure 1: Structure of the HFS-enhanced NO. (a) Schematic of the HFS module (right) integrated with the residual block (left). (b) Structure of the ResUNet with the HFS modules (blocks in front of conv layers). (c) An example of a learned latent space feature from the first layer of the encoder trained with and without HFS. The most similar feature maps of the models in the first encoder level are shown. (d) An example of a learned latent space feature from the last layer of the decoder trained with and without HFS. The most similar feature maps of the two models at the last decoder level are shown. (e-f) Examples of temperature prediction with NO and HFS-enhanced NO at two different time-steps. A region with high-frequency features (top right corner) is zoomed in for better visualization.
• Figure 2: Mitigating Spectral Bias with Diffusion Model. The states estimated by the NO exhibit over-smoothing. They serve as the prior that conditions the DM, which in turn reconstructs the missing frequencies iteratively through conditional sampling. The results are based on a NO with 2 million parameters.
• Figure 3: Temperature prediction errors of NO and HFS-enhanced NO varying with NO size. (a) Root mean square error (RMSE), (b) Boundary RMSE (BRMSE), (c) Bubble RMSE, (d) Mean maximum error. All the errors are calculated over the 5 time-step temperature predictions. The legends in (a) are applicable to (b - d) as well. All the results are based on test dataset in subcooled pool boiling.
• Figure 4: Subcooled pool boiling transient temperature prediction. (a) Ground truth (GT) temperatures for 5 consecutive time steps (from left to right) ($\Delta$t = 8 ms). (b) NO prediction results. (c) HFS-enhanced NO prediction results. (d) The corresponding energy spectra (p(k)) for each time step. For better visualization, the subplots in (d) show the energy spectra only for the high wavenumbers. The legends in first plot are applicable to other plots as well. All the results are based on a $\sim$ 3.5 M parameter NO.
• Figure 5: Latent space features in HFS-enhanced NO. (a, b) Example of latent space feature in the first layer of encoder and the corresponding normalized energy spectra ($p(k)$) in the $\sim$ 3.5 million parameter models. (c, d) Example of latent feature in the last layer of decoder and the corresponding normalized energy spectra for the model with $\sim$ 3.5 million parameters. (e) Example of latent feature in the first layer of encoder and the corresponding normalized energy spectra in the $\sim$ 16 million parameter models. (g) Example of latent feature in the last layer of decoder and the corresponding normalized energy spectra in the $\sim$ 16 million parameter models. (i-j) Average ratio of high- frequency energy to total energy at each layer in encoder (i) and decoder (j). Note that the low-frequency cutoff is set to the first 12.5 %, 18.75 %, 25 %, 37.5 %, and 50 % of the wavenumbers, from highest to lowest spatial resolutions (384 to 24 pixels), respectively