VI3NR: Variance Informed Initialization for Implicit Neural Representations

TL;DR

VI3NR addresses the initialization bottleneck in implicit neural representations by deriving a variance-preserving initialization that remains valid for arbitrary activations. The method jointly considers forward and backward variance, uses Monte Carlo estimates for activation statistics when needed, and provides a practical workflow to choose the target preactivation variance $\sigma_p^2$ for a given task. It unifies and extends classical initializations (Xavier, Kaiming) under a principled variance framework and demonstrates improved convergence and reconstruction quality for images, 3D surfaces, and audio, especially for challenging INR activations like Gaussian and sinc. The work offers a general, activation-agnostic scheme that enhances INR stability and performance with broad practical impact on high-frequency signal reconstruction and neural representations.

Abstract

Implicit Neural Representations (INRs) are a versatile and powerful tool for encoding various forms of data, including images, videos, sound, and 3D shapes. A critical factor in the success of INRs is the initialization of the network, which can significantly impact the convergence and accuracy of the learned model. Unfortunately, commonly used neural network initializations are not widely applicable for many activation functions, especially those used by INRs. In this paper, we improve upon previous initialization methods by deriving an initialization that has stable variance across layers, and applies to any activation function. We show that this generalizes many previous initialization methods, and has even better stability for well studied activations. We also show that our initialization leads to improved results with INR activation functions in multiple signal modalities. Our approach is particularly effective for Gaussian INRs, where we demonstrate that the theory of our initialization matches with task performance in multiple experiments, allowing us to achieve improvements in image, audio, and 3D surface reconstruction.

Figures (5)

• Figure 1: Notation diagram. For clarity, we illustrate a general MLP based INR architecture (layers $0$, $i$ and $n$ shown). Layers are of the form $\phi_i(x_i)=f(W_ix_i)$ where $f$ is the activation function and $W_i$ are the weights, the preactivations are $z_0,...,z_{n-1}$, the postactivations are $x_1,...,x_n$ and the input and output of the network are $x_0$ and $z_n$.
• Figure 2: Heatmaps of the condition in \ref{['eq:backwards-cond-final']} and task performance when using our initialization with Gaussian activations. In each heatmap the Gaussian activation function variance ($\sigma_a^2$) and preactivation variance ($\sigma_p^2$) are varied, revealing linear trends.
• Figure 3: Parameter grid search for images with differing number of layers. We plot PSNR as a function of the Gaussian activation standard deviation ($\sigma_a$) and the weight distribution standard deviation ($\sigma_p$). As the number of layers increases, the slope converges towards the theoretical slope, and the linear trend sharpens. This matches our hypothesis that the backward condition \ref{['eq:backwards-cond-final']} is more important as the number of layers increase.
• Figure 4: Image comparison. Left to right: random normal init, our MC init, our init.
• Figure 5: Performance gap vs. $\sigma_a$. As $\sigma_a$ decreases, performance drops for both inits, but a significant gap remains.

Theorems & Definitions (2)

• Proposition 3.1
• proof