The Inlet Rank Collapse in Implicit Neural Representations: Diagnosis and Unified Remedy

TL;DR

Implicit Neural Representations often struggle to recover fine details within finite budgets due to spectral bias. The authors propose a Structural Diagnostic Framework that decomposes the Neural Tangent Kernel (NTK) layerwise and identifies an Inlet Rank Collapse: a low-rank inlet representation that bottlenecks gradient flow from the input coordinates. They reinterpret PE, SIREN, and BN as rank-restoration mechanisms and introduce a minimalist Rank-Expanding Initialization that raises inlet rank without architectural changes, enabling high-fidelity reconstructions. Across 2-D and 3-D tasks and spectral analyses, restoring inlet rank proves sufficient to unlock the full expressive capacity of standard MLPs, suggesting a broad principle for optimizing information flow in neural architectures.

Abstract

Implicit Neural Representations (INRs) have revolutionized continuous signal modeling, yet they struggle to recover fine-grained details within finite training budgets. While empirical techniques, such as positional encoding (PE), sinusoidal activations (SIREN), and batch normalization (BN), effectively mitigate this, their theoretical justifications are predominantly post hoc, focusing on the global NTK spectrum only after modifications are applied. In this work, we reverse this paradigm by introducing a structural diagnostic framework. By performing a layer-wise decomposition of the NTK, we mathematically identify the ``Inlet Rank Collapse'': a phenomenon where the low-dimensional input coordinates fail to span the high-dimensional embedding space, creating a fundamental rank deficiency at the first layer that acts as an expressive bottleneck for the entire network. This framework provides a unified perspective to re-interpret PE, SIREN, and BN as different forms of rank restoration. Guided by this diagnosis, we derive a Rank-Expanding Initialization, a minimalist remedy that ensures the representation rank scales with the layer width without architectural modifications or computational overhead. Our results demonstrate that this principled remedy enables standard MLPs to achieve high-fidelity reconstructions, proving that the key to empowering INRs lies in the structural optimization of the initial rank propagation to effectively populate the latent space.

Figures (8)

• Figure 1: Inlet Rank Restoration Unlocks ReLU MLP Capacity. Reconstruction results of a standard ReLU-activated MLP. By modifying only the initialization of the first layer to expand representation rank, the PSNR improves significantly from $24.14$ (left) to $27.69$ (right). This pure initialization-based gain, without any architectural changes, identifies the Inlet Rank Collapse as the primary bottleneck to the capacity of vanilla INRs.
• Figure 2: The contrast between (a) weight matrices $\mathbf{W}_i$ and (b) intermediate representations $\mathbf{Z}_i$ in a 3-layer vanilla MLP (width 256). While weights remain full-rank and well-conditioned, the representations suffer from severe Inlet Rank Collapse, effectively starving the parameter gradients.
• Figure 3: Reconstruction quality vs. BN position. The experiment confirms that addressing rank deficiency at the inlet (Layer 1) is the only way to fully recover the network's effective capacity. Interventions in deeper layers cannot recover the gradient diversity of preceding stages, as those earlier weights remain trapped in the low-rank subspace imposed by the inlet.
• Figure 4: Conceptual illustration of capacity. While parameter dimension (gray) is fixed, effective capacity is governed by representation rank (colored). Case 3: Persistent low rank at the inlet starves the entire network. Case 2: Mid-stage expansion only recovers capacity for deeper layers. Case 1: Early rank expansion at the inlet unlocks the capacity of all parameters.
• Figure 5: Illustration of the proposed initialization in the 1-D case. Carefully chosen biases separate activation thresholds across input coordinates, yielding a full-rank, triangular representation matrix.

Theorems & Definitions (15)

• Definition 1
• Definition 2
• Proposition 1
• Proposition 2
• Lemma 1
• Corollary 1
• Remark 1: Healthy Weights: The Capacity
• Remark 2: Inlet Rank Collapse: The Starvation
• Proposition 3
• Proposition 4