Towards the Spectral bias Alleviation by Normalizations in Coordinate Networks

TL;DR

This work analyzes spectral bias in coordinate networks through the neural tangent kernel (NTK) lens and proves that classical normalization techniques, specifically batch normalization and layer normalization, can shift NTK eigenvalues by reducing the maximum and variance with minimal mean change. It introduces two novel normalizations, global normalization (GN) and cross normalization (CN), that further modulate the NTK spectrum and mitigate spectral bias. Theoretical results are complemented by extensive experiments across image compression, shape representation, CT/MRI reconstruction, and advanced tasks like 5D novel view synthesis and 3D multi-view stereo, where CN consistently achieves state-of-the-art performance. The findings suggest normalization-based coordinate networks offer a practical, principled route to robustly learn high-frequency components in diverse inverse problems and signal representations.

Abstract

Representing signals using coordinate networks dominates the area of inverse problems recently, and is widely applied in various scientific computing tasks. Still, there exists an issue of spectral bias in coordinate networks, limiting the capacity to learn high-frequency components. This problem is caused by the pathological distribution of the neural tangent kernel's (NTK's) eigenvalues of coordinate networks. We find that, this pathological distribution could be improved using classical normalization techniques (batch normalization and layer normalization), which are commonly used in convolutional neural networks but rarely used in coordinate networks. We prove that normalization techniques greatly reduces the maximum and variance of NTK's eigenvalues while slightly modifies the mean value, considering the max eigenvalue is much larger than the most, this variance change results in a shift of eigenvalues' distribution from a lower one to a higher one, therefore the spectral bias could be alleviated. Furthermore, we propose two new normalization techniques by combining these two techniques in different ways. The efficacy of these normalization techniques is substantiated by the significant improvements and new state-of-the-arts achieved by applying normalization-based coordinate networks to various tasks, including the image compression, computed tomography reconstruction, shape representation, magnetic resonance imaging, novel view synthesis and multi-view stereo reconstruction.

Figures (10)

• Figure 1: Normalization techniques significantly alleviate the spectral bias of coordinate networks. (a) Normalization techniques shift the NTK's eigenvalues distribution from a lower one to a higher one. From top to bottom, each row refers to the results of coordinate networks with ReLU activations, and positional encoding tancik2020fourier with 1 and 5 Fourier bases, respectively. Note that the values in horizontal-axis are the exponents with a base of 10. (b) The spectral bias is alleviated and better performance is achieved compared with the one without normalization (e.g., the texture on the lion's left paw).
• Figure 2: Schematic diagrams of different normalization techniques.
• Figure 3: Evolution of frequency-specific approximation error with training iterations of five different methods (x-axis for training iteration, y-axis for frequency and colormap for relative approximation error). Deeper color represents larger frequency-error.
• Figure 4: (a) Rate-distortion curves measured in PSNR of various coordinate networks under different bpps trained on the Kodak dataset. (b) Rate-distortion curves measured in SSIM of various coordinate networks under different bpps trained on the Kodak dataset. (c) exhibits the training curves of these methods on 2D image.
• Figure 5: Comparisons of different methods for representing the 2D Image Tower. The corresponding Fourier spectra are also visualized.

Theorems & Definitions (1)

• Proposition 1