Neural Gaussian Scale-Space Fields

TL;DR

This work introduces Neural Gaussian Scale-Space Fields, a self-supervised framework to learn a fully continuous, anisotropic Gaussian scale space for arbitrary signals via a neural field $F(\mathbf{x}, \hat{\Sigma})$. By combining dampened Fourier feature encodings with a Lipschitz-bounded MLP, the model learns Gaussian-like smoothing without explicit filtering during training, followed by a lightweight calibration $h(\Sigma)$ to map learned modulation to a target covariance. The approach supports spatially varying filtering, works across modalities (images, SDFs, textures, and light-stage data), and enables applications in texture anti-aliasing, relighting, and continuous multiscale optimization. Empirical results on multiple modalities show competitive or superior performance to baselines, with robust anisotropic filtering and favorable efficiency, while ablations clarify the contribution of each component.

Abstract

Gaussian scale spaces are a cornerstone of signal representation and processing, with applications in filtering, multiscale analysis, anti-aliasing, and many more. However, obtaining such a scale space is costly and cumbersome, in particular for continuous representations such as neural fields. We present an efficient and lightweight method to learn the fully continuous, anisotropic Gaussian scale space of an arbitrary signal. Based on Fourier feature modulation and Lipschitz bounding, our approach is trained self-supervised, i.e., training does not require any manual filtering. Our neural Gaussian scale-space fields faithfully capture multiscale representations across a broad range of modalities, and support a diverse set of applications. These include images, geometry, light-stage data, texture anti-aliasing, and multiscale optimization.

Figures (17)

• Figure 1: An original 2D signal $f$f (a) alongside samples from its Gaussian scale space ${\TextOrMath{$f$\xspace}{f}_\TextOrMath{$Σ$\xspace}{\Sigma}}$$f$f_$\Sigma$Σ (b-d). In b), isotropic smoothing is applied, while c) and d) demonstrate examples of anisotropic filtering. Insets show isolines of the corresponding Gaussian kernels. We consider scale spaces that are continuous both in signal coordinates and in full Gaussian covariance.
• Figure 2: Four different strategies to learn a neural field $F$F from a signal $f$f. $F$F takes a continuous coordinate $\mathbf{x}$x as input, which is fed into a positional encoding $\gamma$γ (Eq. \ref{['eq:positional_encoding']}) that produces a set of Fourier features using cosine (blue curves) and sine (red curves) functions of different frequencies. The resulting features serve as input to an MLP $\Psi_\TextOrMath{$θ$\xspace}{\theta}$Ψ_$\theta$θ that regresses $f$f. (a) The basic setup learns a faithful reconstruction of $f$f (the curves for $F$F and $f$f overlay completely), but does not allow any smoothing. (b) Modulating the Fourier features using custom weights $\TextOrMath{$λ$\xspace}{\lambda}_i$ (yellow bars) tends to remove some high frequencies, but distorts the reconstruction in an unpredictable way (orange rectangles mark incoherent spikes in $F$F). (c) Employing a Lipschitz-bounded MLP ${\overline\TextOrMath{$Ψ_$\theta$θ$\xspace}{\Psi_\TextOrMath{$θ$\xspace}{\theta}}}$$\Psi_\TextOrMath{$θ$\xspace}{\theta}$Ψ_$\theta$θ leads to smoothing, but it requires choosing a single fixed bound for training, lacking flexibility. (d) Our approach combines Fourier feature modulation with Lipschitz bounding to enable controllable smoothing.
• Figure 3: Given a training signal $f$f (bottom row), progressive dampening of Fourier features in combination with a Lipschitz-bounded MLP allows a neural field $F$F learn Gaussian-smoothed versions ${\TextOrMath{$f$\xspace}{f}_\TextOrMath{$Σ$\xspace}{\Sigma}}$$f$f_$\Sigma$Σ of $f$f. In the three upper rows, differently smoothed $\TextOrMath{$F$\xspace}{F}_i$ (dashed colored curves) and their respective closest $\TextOrMath{$$f$f_$\Sigma$Σ$\xspace}{{\TextOrMath{$f$\xspace}{f}_\TextOrMath{$Σ$\xspace}{\Sigma}}}_i$ (solid grey curves) are overlayed, revealing that our solution provides a faithful approximation of Gaussian filtering.
• Figure 4: Distribution of Fourier frequencies $\mathbf{a}$a (colored dots). Axis-aligned frequencies (a) cannot capture anisotropies. Uncorrelated sampling from a Gaussian distribution (b) leads to clusters and holes, impeding filtering quality. Our approach starts with a low-discrepancy sequence (c) and warps samples radially such that radial density follows a zero-mean Gaussian distribution (d). The grey insets in b)-d) show the radial density distributions of the respective point sets. In e), we visualize an example of dampening the frequencies in d) with a matrix ${\hat{\Sigma}}$Σ̂ (an isoline of its inverse is shown). Here, point size corresponds to dampening factors $\TextOrMath{$λ$\xspace}{\lambda}_i ( \TextOrMath{$Σ̂$\xspace}{{\hat{\Sigma}}} )$. Our carefully distributed Fourier frequencies facilitate highly selective anisotropic filtering.
• Figure 5: Distances (black lines) before (top) and after (bottom) positional encoding.