Learning Transferable Features for Implicit Neural Representations

TL;DR

This work tackles the limited transferability of implicit neural representations by introducing STRAINER, which learns a shared encoder across multiple INRs and uses it to initialize new INRs for unseen signals from the same domain. The method yields faster convergence and higher reconstruction quality, with reported gains up to $+10\ \mathrm{dB}$ in PSNR for in-domain image fitting, and strong cross-domain generalization in both fitting and inverse problems. Through analyses of learned input-space partitions and training dynamics, the authors show that the shared encoder captures transferable, data-driven priors that improve performance over random initializations and several meta-learning baselines. STRAINER thus enables data-driven priors in INRs, accelerating high-quality reconstructions in real-time and low-resource settings while highlighting avenues for deeper understanding of what makes transfer possible in implicit representations.

Abstract

Implicit neural representations (INRs) have demonstrated success in a variety of applications, including inverse problems and neural rendering. An INR is typically trained to capture one signal of interest, resulting in learned neural features that are highly attuned to that signal. Assumed to be less generalizable, we explore the aspect of transferability of such learned neural features for fitting similar signals. We introduce a new INR training framework, STRAINER that learns transferrable features for fitting INRs to new signals from a given distribution, faster and with better reconstruction quality. Owing to the sequential layer-wise affine operations in an INR, we propose to learn transferable representations by sharing initial encoder layers across multiple INRs with independent decoder layers. At test time, the learned encoder representations are transferred as initialization for an otherwise randomly initialized INR. We find STRAINER to yield extremely powerful initialization for fitting images from the same domain and allow for $\approx +10dB$ gain in signal quality early on compared to an untrained INR itself. STRAINER also provides a simple way to encode data-driven priors in INRs. We evaluate STRAINER on multiple in-domain and out-of-domain signal fitting tasks and inverse problems and further provide detailed analysis and discussion on the transferability of STRAINER's features. Our demo can be accessed at https://kushalvyas.github.io/strainer.html .

Figures (15)

• Figure 1: strainer- Learning Transferable Features for Implicit Neural Representations. During training time (a), strainer divides an INR into encoder and decoder layers. strainer fits similar signals while sharing the encoder layers, capturing a rich set of transferrable features. At test-time, strainer serves as powerful initialization for fitting a new signal (b). An INR initialized with strainer's learned encoder features achieves (c) faster convergence and better quality reconstruction compared to baseline siren models.
• Figure 2: strainer learns faster. We show the reconstruction quality (PSNR) of different initialization schemes for in-domain image fitting on CelebA-HQ celebahq. We compare siren sitzmann2020implicit model initialized by (1) random weights ( siren), (2) fitting on another face image ( siren finetuned), (3) strainer -1 (trained using one face image), and (4) strainer-10 (trained using ten face images). We also evaluate against multiple baselines such as Meta-Learned 5K tancik2021learned, TransINRtransinr, and IPCipc
• Figure 3: Visualization of learned features in strainer and baseline siren model. We visualize (a) the first principal component of the learned encoder features for strainer and corresponding layer for siren . At iteration 0, strainer's features already capture a low dimensional structure allowing it to quickly adapt to the cat image. High frequency detail emerges in strainer's learned features by iteration 50, whereas siren is lacking at iteration 100. The inset showing the power spectrum of the reconstructed image further confirms that strainer learns high frequency faster. We also show the (b) reconstructed images and remark that strainer fits high frequencies faster.
• Figure 4: strainerconverges to low and high frequencies fast. We plot the histogram of absolute gradients of layers 1,5 and last over 1000 iterations while fitting an unseen signal. At test time, strainer's initialization quickly learns low frequency, receiving large gradients update at the start in its initial layers and reaching convergence. The Decoder layer in strainer also fits high frequency faster. Large gradients from corresponding siren layers show it learning significant features as late as 1000 iterations.
• Figure 5: Fitting MRI images from OASIS-MRI dataset. At just 100 iterations, strainer is able to represent medical images with high quality. strainer's initialization allows for fast recovery for sparse and delicate structures, showing applicability in low-resource medical domains as well.