Adaptive Training of INRs via Pruning and Densification

TL;DR

Implicit neural representations encode signals as continuous mappings from coordinates to values using sinusoidal encodings, but choosing frequencies $\omega$ and architectures is challenging. AIRe introduces an adaptive INR training framework that alternates pruning with targeted weight decay and input-frequency densification to align model capacity with signal content. The authors provide a harmonic-expansion justification and a pruning stability bound, and demonstrate that AIRe achieves comparable or better reconstruction with substantially fewer parameters across images, SDFs, and NeRFs. This approach reduces redundancy and training instability while reducing hyperparameter tuning, with public code and pretrained models available.

Abstract

Encoding input coordinates with sinusoidal functions into multilayer perceptrons (MLPs) has proven effective for implicit neural representations (INRs) of low-dimensional signals, enabling the modeling of high-frequency details. However, selecting appropriate input frequencies and architectures while managing parameter redundancy remains an open challenge, often addressed through heuristics and heavy hyperparameter optimization schemes. In this paper, we introduce AIRe ($\textbf{A}$daptive $\textbf{I}$mplicit neural $\textbf{Re}$presentation), an adaptive training scheme that refines the INR architecture over the course of optimization. Our method uses a neuron pruning mechanism to avoid redundancy and input frequency densification to improve representation capacity, leading to an improved trade-off between network size and reconstruction quality. For pruning, we first identify less-contributory neurons and apply a targeted weight decay to transfer their information to the remaining neurons, followed by structured pruning. Next, the densification stage adds input frequencies to spectrum regions where the signal underfits, expanding the representational basis. Through experiments on images and SDFs, we show that AIRe reduces model size while preserving, or even improving, reconstruction quality. Code and pretrained models will be released for public use.

Figures (12)

• Figure 1: We present AIRe, a robust training method that adaptively fits the INR architecture to the target signal through two complementary mechanisms: (i) pruning with targeted weight decay (TWD) which mitigates parameter redundancy and fine tuning (FT) dependence by transferring information prior to structured neuron removal (see birds), and (ii) input frequency densification, which augments the representation basis, enhancing convergence and details fidelity (see hand). We compare three strategies: (i) an overparameterized SIREN model with standard training (large network), (ii) a model adapted with AIRe (Ours), and (iii) a small network fitted with standard training. AIRe improves reconstruction accuracy while producing more compact networks (blue box), and enhances training convergence in settings where overparameterization leads to divergence (see statue box).
• Figure 2: We present AIRe, a training framework that adapts network architecture through two theoretically grounded strategies: densification and pruning. For signals with rich frequency content, densification selects the most relevant input frequencies $\omega_j$ and expands the spectrum by augmenting $\omega$ with $2 \cdot \omega_j$. To reduce network size, pruning identifies candidate neurons via magnitude criterion, transfers information during training with a novel targeted weight decay (TWD) regularization, and removes neurons whose norm falls below a threshold $\epsilon$. The function $\text{TopK}(v)$ selects the $K$ largest entries of $v$.
• Figure 3: Qualitative comparison of SDF reconstructions on the Armadillo, Buddha, and Lucy models using a SIREN with $\omega_0=60$ and small network size $[64, 64, 256]$. Left: results of training the final small network directly. Right: results of AIRe. Colors indicate the distance from the ground-truth surface, from dark blue (0) to dark red ($\geq 0.01$). AIRe produces reconstructions that are consistently closer to the ground truth than those obtained by training the small network from scratch.
• Figure 4: Qualitative comparison on the Armadillo. Left: ground truth. Middle: standard training of the large network, which diverges and produces noisy artifacts. Right: AIRe, which avoids divergence and yields a cleaner reconstruction with fewer parameters.
• Figure 5: TWD reduces the dependence of finetune (FT) when pruning. TWD effectively transfers information before pruning. Left: qualitative results with $28\%, 52\%, 72\%$, and $88\%$ of parameters pruned. The first row shows results without TWD, and the second row with TWD. Right: Table with the PSNR values for each case.

Theorems & Definitions (6)

• Theorem 1
• Theorem 2
• Theorem 3
• proof
• Theorem 4
• proof