InsertNeRF: Instilling Generalizability into NeRF with HyperNet Modules

TL;DR

This work introduces InsertNeRF, a method for INStilling gEneRalizabiliTy into NeRF that dynamically tailors NeRF's weights to specific reference scenes, transforming multi-scale sampling-aware features into scene-specific representations.

Abstract

Generalizing Neural Radiance Fields (NeRF) to new scenes is a significant challenge that existing approaches struggle to address without extensive modifications to vanilla NeRF framework. We introduce InsertNeRF, a method for INStilling gEneRalizabiliTy into NeRF. By utilizing multiple plug-and-play HyperNet modules, InsertNeRF dynamically tailors NeRF's weights to specific reference scenes, transforming multi-scale sampling-aware features into scene-specific representations. This novel design allows for more accurate and efficient representations of complex appearances and geometries. Experiments show that this method not only achieves superior generalization performance but also provides a flexible pathway for integration with other NeRF-like systems, even in sparse input settings. Code will be available https://github.com/bbbbby-99/InsertNeRF.

Figures (13)

• Figure 1: Overview of motivation. (a) We instill generalizability into NeRF-like systems, including vanilla NeRF, mip-NeRF, and NeRF++ frameworks, to achieve consistent performance across scenes without modifying the base framework or requiring scene-specific retraining. (b) InsertNeRF significantly improves depth estimation compared to its original counterpart.
• Figure 2: Overview of InsertNeRF. (a) Within the NeRF framework, two types of HyperNet modules are inserted into $\mathcal{F}_{geo}$ and $\mathcal{F}_{app}$. The HyperNet modules begin by (b) extracting features among multiple ($N$) reference images, and (c) using a multi-layer dynamic-static aggregation strategy to aggregate the scene representations. Based on these scene representations and specially designed sampling-aware filters, (d) we develop dynamic MLPs and activation functions to guide the weights and instill generalizability into vanilla NeRF. Finally, (a)standard volume rendering is performed.
• Figure 3: (a) Qualitative comparisons of InsertNeRF against SOTA methods. (b) A t-SNE plot of the scene-specific representations from our HyperNet modules. More analysis in the \ref{['D2']}
• Figure 4: Performance and efficiency under different input-number $N$ on NeRF Synthetic.
• Figure 5: Qualitative results of Insert-mip-NeRF. Please refer to the \ref{['E4']} for more results.