Inductive Gradient Adjustment For Spectral Bias In Implicit Neural Representations

TL;DR

The paper addresses spectral bias in implicit neural representations by tying it to training dynamics through the Neural Tangent Kernel. It introduces Inductive Gradient Adjustment (IGA), which uses an eNTK-based gradient transformation and inductive generalization from sampled data to the population to tailor spectral bias. Theoretical results show that eNTK adjustments approximate NTK behavior as network width grows and that population training dynamics can be estimated from sampling, enabling scalable improvements. Empirically, IGA yields consistent high-frequency detail improvements across 2D/3D/NeRF INR tasks with manageable overhead, outperforming prior training-dynamics approaches and demonstrating practical impact for richer INRs.

Abstract

Implicit Neural Representations (INRs), as a versatile representation paradigm, have achieved success in various computer vision tasks. Due to the spectral bias of the vanilla multi-layer perceptrons (MLPs), existing methods focus on designing MLPs with sophisticated architectures or repurposing training techniques for highly accurate INRs. In this paper, we delve into the linear dynamics model of MLPs and theoretically identify the empirical Neural Tangent Kernel (eNTK) matrix as a reliable link between spectral bias and training dynamics. Based on this insight, we propose a practical Inductive Gradient Adjustment (IGA) method, which could purposefully improve the spectral bias via inductive generalization of eNTK-based gradient transformation matrix. Theoretical and empirical analyses validate impacts of IGA on spectral bias. Further, we evaluate our method on different INRs tasks with various INR architectures and compare to existing training techniques. The superior and consistent improvements clearly validate the advantage of our IGA. Armed with our gradient adjustment method, better INRs with more enhanced texture details and sharpened edges can be learned from data by tailored impacts on spectral bias.

Figures (13)

• Figure 1: Evolution of approximation error with training iterations in time and Fourier domain. Line plots visualize MSE loss curves of MLPs with $1024$ and $4096$ neurons optimized by four gradient adjustments, i.e., ${\bm{I}},{\bm{S}},\tilde{{\bm{S}}},\tilde{{\bm{S}}}_e$. The shaded area of lines indicates training fluctuations. Heatmaps show the relative error $\Delta_k$ in Eq. \ref{['delta_k']} on four frequency bands. Details are in the experiment 1 of Sec. \ref{['empirical analysiss']}.
• Figure 2: Progressively amplified impacts on spectral bias of ReLU and SIREN by increasing the number of balanced eigenvalues via $\tilde{{\bm{S}}}_e$ when $p=8$, i.e., IGA. ReLU denotes the MLP with ReLU optimized via vanilla gradients; ReLU-$\tilde{{\bm{S}}}_e$-2 denotes the ReLU optimized via gradients adjusted by $\tilde{{\bm{S}}}_e$ with $2$ balanced eigenvalues. Details and more results are in the experiment 2 of Sec. \ref{['empirical analysiss']} and Appendix \ref{['simple function appendix']}.
• Figure 3: Relative error $\Delta_k$ curves of ReLU at 20Hz and SIREN at 40Hz with varying balanced eigenvalues and group size $p$. IGA-1 denotes that $p=1$, i.e., $\tilde{{\bm{K}}}$-based gradient adjustment; Baseline denotes vanilla gradients. The shaded area of lines indicates training fluctuations, represented by twice the standard deviation. Details and more results are in the experiment 2 of Sec. \ref{['empirical analysiss']} and Appendix \ref{['simple function appendix']}.
• Figure 4: Visual examples of 2D color image approximation results by different training dynamics methods. Enlarged views of regions labeled by red boxes are provided. The residuals of these regions in the Fourier domain are visualized through heatmaps in the top right corner. The increase in error corresponds to the transition of colors in the heatmaps from blue to red. Detailed settings are in Sec. \ref{['2D color image']}.
• Figure 5: Visual examples of 3D shape representation results by different training dynamics methods. Five images on the right correspond to the enlarged views of the red-boxed area of ground truth and four models, respectively. More results can be found in our Appendix \ref{['3d shape appendix']}.

Theorems & Definitions (13)

• Theorem 3.1
• Theorem 3.2
• Lemma 1.1
• Lemma 1.2
• proof
• Lemma 1.3
• proof
• Lemma 1.4
• proof
• Theorem 1.5