Spectral Bottleneck in Sinusoidal Representation Networks: Noise is All You Need

TL;DR

The paper identifies a spectral bottleneck in SIREN-based implicit neural representations, where high-frequency components are hard to learn due to initialization and spectral energy distribution. It analyzes training dynamics via activation spectra and the empirical NTK to explain why high-frequency targets are difficult to fit and introduces WINNER, a target-aware Gaussian weight perturbation that broadens frequency support without adding trainable parameters. SIREN$^2$ (SIREN with WINNER) demonstrates state-of-the-art results in 1D audio fitting and substantial gains in 2D image fitting, as well as improved performance in image denoising, audio denoising, and audio inpainting, across diverse datasets. The approach highlights the critical impact of initialization on INR performance and provides a practical, robust method for practitioners to better capture fine-scale details in high-frequency domains. Limitations include reliance on knowledge (or estimates) of the target spectral content to set perturbation scales, and sensitivity to learning-rate schedules, suggesting avenues for adaptive or online tuning in future work.

Abstract

This work identifies and attempts to address a fundamental limitation of implicit neural representations with sinusoidal activation. The fitting error of SIRENs is highly sensitive to the target frequency content and to the choice of initialization. In extreme cases, this sensitivity leads to a spectral bottleneck that can result in a zero-valued output. This phenomenon is characterized by analyzing the evolution of activation spectra and the empirical neural tangent kernel (NTK) during the training process. An unfavorable distribution of energy across frequency modes was noted to give rise to this failure mode. Furthermore, the effect of Gaussian perturbations applied to the baseline uniformly initialized weights is examined, showing how these perturbations influence activation spectra and the NTK eigenbasis of SIREN. Overall, initialization emerges as a central factor governing the evolution of SIRENs, indicating the need for adaptive, target-aware strategies as the target length increases and fine-scale detail becomes essential. The proposed weight initialization scheme (WINNER) represents a simple ad hoc step in this direction and demonstrates that fitting accuracy can be significantly improved by modifying the spectral profile of network activations through a target-aware initialization. The approach achieves state-of-the-art performance on audio fitting tasks and yields notable improvements in image fitting tasks.

Figures (12)

• Figure 1: Spectral bottleneck issue shown using SIREN and overview of the weight perturbation scheme. (a) We attempt to fit eight discretely sampled broadband 1D signals (S1–S8) with decreasing low-frequency content. As shown in the right panel, the PSNR of SIREN sitzmann2020implicit progressively decreases from S1 to S4, eventually encountering a spectral bottleneck after S4. SIREN fails to capture nearly all the frequencies of signals S4–S8, even though its frequency support should in principle allow it to represent a portion of the spectrum. In contrast, SIREN$^2$ initialized with WINNER maintains higher PSNR across all signals. (b) Schematic of a feedforward neural network with periodic activations, illustrating the statistical distributions of layer-1 outputs $X_{ij}$, weight matrix $W_{jk}$, and noise matrix $\eta_{jk}$. The effect of Gaussian noise ($\eta_{jk}$) on the spectrum of layer 2 pre-activations is shown: WINNER enhances the receptivity of high-frequencies.
• Figure 2: An example case where the standard weight initialization scheme of SIREN fails to reconstruct an audio clip. (a) Spectrograms of ground truth (tetris.wav) (top), SIREN reconstruction (middle), and the error map (bottom) computed using first layer $\omega_0=30000$ and hidden layer width 128. (b,c) PSNR histories of SIREN for different input scalings and network sizes. A five-layer MLP was used with a learning rate scheduler reducing the rate by 2% every 20 epochs from an initial value of $10^{-4}$.
• Figure 3: Contrasting learning dynamics of SIREN for low and high frequency-dominant targets. (a) Output evolution when fitting the low-frequency signal of Eqn. \ref{['eqn:signals']}. The right subplot shows the NTK spectral energy, $\log(1 + \mathcal{S}(k))$, across frequency eigenmodes $k$ during training, exhibiting a steady increase that indicates effective learning. (b) Output and NTK spectral energy evolution for a high-frequency signal. The left subplot shows that SIREN fails to match the ground truth, while the right subplot reveals a suppression of spectral energy over training, indicating difficulty in representing high-frequency-dominant signals.
• Figure 4: Poor frequency support of SIREN for fitting high-frequency dominant targets. Distributions (column-1) and cumulative power spectra (column-2) of hidden-layer pre-activation and network outputs for a four-layer SIREN at epoch $0$ and $10^{4}$ when fitting tetris.wav. Across all layers, the spectral content is concentrated far below that of the high-frequency target, indicating insufficient frequency support.
• Figure 5: Distributions of the dot product between inputs $X_{ij} \sim \text{Arcsin}(-1,1)$ and weights initialized under two initialization schemes: (a) standard uniform weights $W_{jk} = \sim \mathcal{U}\left(- \omega_0^{-1}\sqrt{6/d}, \omega_0^{-1}\sqrt{6/d} \right)$, and (b) WINNER, in which uniform weights are perturbed with Gaussian noise, $W_{jk} + \eta_{jk}$. The noise addition increases the standard deviation of the dot product from $1$ to $\sqrt{1 + \frac{d s^2}{2}}$, where $d$ is the input dimension (fan_in). This increase closely matches the analytically predicted value (Theorem \ref{['thr:theorem1']}) shown by the dashed black line.

Theorems & Definitions (4)

• Proposition 1
• Theorem 4.1
• proof
• Proposition 2