Split-Layer: Enhancing Implicit Neural Representation by Maximizing the Dimensionality of Feature Space

TL;DR

This work tackles the limited representational capacity of implicit neural representations (INRs) arising from the linear feature space of vanilla MLPs. It introduces the split-layer, which partitions each layer into $N$ branches and fuses their outputs via Hadamard product, creating high-degree polynomial feature interactions that expand the feature space to $\binom{\frac{C}{\sqrt{N}} + N - 1}{N}$ without increasing parameters. The approach is theoretically analyzed through feature-space expansion and Neural Tangent Kernel perspectives and empirically validated across 2D image fitting, 2D CT reconstruction, 3D shape representation, and 5D novel-view synthesis, showing consistent and substantial gains across multiple INR backbones. The results suggest that split-layer can broadly enhance INR performance on inverse problems and rendering tasks, with potential for further extensions using kernel-inspired reformulations.

Abstract

Implicit neural representation (INR) models signals as continuous functions using neural networks, offering efficient and differentiable optimization for inverse problems across diverse disciplines. However, the representational capacity of INR defined by the range of functions the neural network can characterize, is inherently limited by the low-dimensional feature space in conventional multilayer perceptron (MLP) architectures. While widening the MLP can linearly increase feature space dimensionality, it also leads to a quadratic growth in computational and memory costs. To address this limitation, we propose the split-layer, a novel reformulation of MLP construction. The split-layer divides each layer into multiple parallel branches and integrates their outputs via Hadamard product, effectively constructing a high-degree polynomial space. This approach significantly enhances INR's representational capacity by expanding the feature space dimensionality without incurring prohibitive computational overhead. Extensive experiments demonstrate that the split-layer substantially improves INR performance, surpassing existing methods across multiple tasks, including 2D image fitting, 2D CT reconstruction, 3D shape representation, and 5D novel view synthesis.

Figures (6)

• Figure 1: The diagram of split-layer. Solid lines represent learnable weights, and dashed lines represent the Hadamard product.
• Figure 2: We visualize the features of SIREN with 9 hidden neurons (a), and the corresponding Split-SIREN with 2 splits (b), and corresponding Split-SIREN with 3 splits (c) on a 2D image fitting task. As can be observed, split-layer introduces more diverse feature basis, significantly enlarging the feature space of the original model.
• Figure 3: Left: Comparisons of the distribution of NTK's eigenvalues. Most of the MLP's eigenvalues are smaller than 1, while the eigenvalues of Split-MLP are increased to the range of $[10^{-2},10^2]$, meaning better performance of Split-MLP for representing a signal with high-frequency components. Right: Verification of the Eqn. \ref{['eopt-n']} for obtaining the optimal split. The curved surface visualizes the Eqn. \ref{['eopt-n']} and four curves plots the quality of image fitting with different splits. It is observed that the optimal result of each curve appears near the surface, verifying the robustness of Eqn. \ref{['eopt-n']}.
• Figure 4: Comparisons of different methods for representing the 2D Image Santorini.
• Figure 5: Comparisons of different methods for CT reconstruction. The corresponding error maps are also visualized.