Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robustifying Fourier Features Embeddings for Implicit Neural Representations

Mingze Ma, Qingtian Zhu, Yifan Zhan, Zhengwei Yin, Hongjun Wang, Yinqiang Zheng

TL;DR

The paper tackles spectral bias in implicit neural representations by coupling Fourier feature embeddings with bias-free MLPs configured as adaptive linear filters and pairing this with a line-search optimization to balance learning dynamics. A simple theorem and NTK-inspired analysis motivate adaptive filtering to suppress unnecessary high-frequency components while enriching input frequencies, addressing noisy outputs without oversmoothing. Empirical results across image regression, 3D shape regression, and neural radiance fields show substantial improvements over state-of-the-art baselines, including clearer high-frequency details and more faithful reconstructions. This approach enhances the robustness and applicability of Fourier-feature embeddings in INRs, with potential impact on inverse graphics, 3D reconstruction, and novel view synthesis.

Abstract

Implicit Neural Representations (INRs) employ neural networks to represent continuous functions by mapping coordinates to the corresponding values of the target function, with applications e.g., inverse graphics. However, INRs face a challenge known as spectral bias when dealing with scenes containing varying frequencies. To overcome spectral bias, the most common approach is the Fourier features-based methods such as positional encoding. However, Fourier features-based methods will introduce noise to output, which degrades their performances when applied to downstream tasks. In response, this paper initially hypothesizes that combining multi-layer perceptrons (MLPs) with Fourier feature embeddings mutually enhances their strengths, yet simultaneously introduces limitations inherent in Fourier feature embeddings. By presenting a simple theorem, we validate our hypothesis, which serves as a foundation for the design of our solution. Leveraging these insights, we propose the use of multi-layer perceptrons (MLPs) without additive

Robustifying Fourier Features Embeddings for Implicit Neural Representations

TL;DR

The paper tackles spectral bias in implicit neural representations by coupling Fourier feature embeddings with bias-free MLPs configured as adaptive linear filters and pairing this with a line-search optimization to balance learning dynamics. A simple theorem and NTK-inspired analysis motivate adaptive filtering to suppress unnecessary high-frequency components while enriching input frequencies, addressing noisy outputs without oversmoothing. Empirical results across image regression, 3D shape regression, and neural radiance fields show substantial improvements over state-of-the-art baselines, including clearer high-frequency details and more faithful reconstructions. This approach enhances the robustness and applicability of Fourier-feature embeddings in INRs, with potential impact on inverse graphics, 3D reconstruction, and novel view synthesis.

Abstract

Implicit Neural Representations (INRs) employ neural networks to represent continuous functions by mapping coordinates to the corresponding values of the target function, with applications e.g., inverse graphics. However, INRs face a challenge known as spectral bias when dealing with scenes containing varying frequencies. To overcome spectral bias, the most common approach is the Fourier features-based methods such as positional encoding. However, Fourier features-based methods will introduce noise to output, which degrades their performances when applied to downstream tasks. In response, this paper initially hypothesizes that combining multi-layer perceptrons (MLPs) with Fourier feature embeddings mutually enhances their strengths, yet simultaneously introduces limitations inherent in Fourier feature embeddings. By presenting a simple theorem, we validate our hypothesis, which serves as a foundation for the design of our solution. Leveraging these insights, we propose the use of multi-layer perceptrons (MLPs) without additive

Paper Structure

This paper contains 27 sections, 8 theorems, 36 equations, 16 figures, 6 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related works
  3. Preliminary
  4. Fourier Features Embeddings
  5. Bias-free ReLU MLPs
  6. Analysis and Motivations
  7. Methods
  8. Bias-Free MLPs as Adaptive Linear Filter
  9. Line-searched based optimization
  10. Experiments
  11. Image Regression on Kodak Dataset
  12. 3D-Shape Regression
  13. Neural Radiance Field Experiments
  14. Ablation Study
  15. Conclusion
  16. ...and 12 more sections

Key Result

Lemma 3.2

(mohan2019robust) For a Bias-free ReLU activation function ($\sigma(\cdot)$) MLPs $f_{BF}(\cdot): \mathbb{R}^N\rightarrow\mathbb{R}^N$ with L layers, matrix at each layer is denoted as $W^l$ for $l=1,\cdots,L$. Then, the MLPs can be written as $f_{BF}(\mathbf{x})=\mathbf{A}_\mathbf{x}\mathbf{x}$.

Figures (16)

  • Figure 1: As illustrated at the circled blue regions and green regions, it can be observed that even with well-chosen standard deviation/scale, as experimented in \ref{['figure:combined']}, the results are still unsatisfactory. However, using our proposed method, the noise is significantly alleviated while further enhancing the high-frequency details.
  • Figure 2: We test the performance of MLPs with Random Fourier Features (RFF) and MLPs with Positional Encoding (PE) on a 1024-resolution image to better distinguish between high- and low-frequency regions, as demonstrated on the left-hand side of this figure. We find that the performance of MLPs+RFF degrades rapidly with increasing standard deviation compared with MLPs+PE. Since positional encoding is deterministic, scale=512 can be considered to have standard deviation around 121.
  • Figure 3: We demonstrate our hypothesis by using three models (MLPs, Fourier features with one linear layer and their combination (i.e. MLPs with Fourier features embeddings)) to fit two kinds of functions. The result demonstrate that combining MLPs with Fourier Features can actually combine their representation capability. These highlighted red boxes demonstrate that MLPs with Fourier features also involve the representation capability of the Fourier features where there are high-frequency fluctuations in the flat regions due to the non-differentiable point in spike function.
  • Figure 4: The pipeline of our method introduces two additional modules compared to the original approach. The first module, an adaptive linear filter, removes unnecessary frequency components at the pixel level, reducing high-frequency noise during regression. The second module dynamically adjusts the learning rate during training to optimize the approximated loss for the next step, achieving dynamical balance. Together, these modules result in cleaner and more detailed images.
  • Figure 5: The black boxes highlight frequency bands with varying phases or frequencies, supporting our derivation that the adaptive linear filter can enrich the embeddings.
  • ...and 11 more figures

Theorems & Definitions (13)

  • Definition 3.1: $\mathbf{Fourier\, features}$
  • Lemma 3.2
  • Lemma 3.3
  • Lemma 9.1
  • Lemma 9.2
  • Lemma 9.3
  • proof
  • Lemma 9.4
  • proof
  • Theorem 9.5: Theorem 4.1 in arora2019fine
  • ...and 3 more