NeuGen: Amplifying the 'Neural' in Neural Radiance Fields for Domain Generalization

TL;DR

NeuGen introduces a brain-inspired neural generalization layer for Neural Radiance Fields (NeRFs) to improve domain generalization. It leverages a Winner-Takes-All-like mechanism to compute a domain-invariant representation $I^{G}$ from input patches and combines it with the original image via $I^{E} = I \oplus I^{G}$, feeding this enhanced input into NeRF pipelines. Across MVSNeRF and GeoNeRF, NeuGen yields consistent gains in PSNR, SSIM, and perceptual quality (lower LPIPS) on diverse datasets (Realistic Synthetic, LLFF, DTU), under both training-from-scratch and finetuning scenarios. The work demonstrates that a brain-inspired data representation can enhance generalization without altering architectural designs, suggesting broad applicability of NeuGen to robust 3D scene synthesis and beyond.

Abstract

Neural Radiance Fields (NeRF) have significantly advanced the field of novel view synthesis, yet their generalization across diverse scenes and conditions remains challenging. Addressing this, we propose the integration of a novel brain-inspired normalization technique Neural Generalization (NeuGen) into leading NeRF architectures which include MVSNeRF and GeoNeRF. NeuGen extracts the domain-invariant features, thereby enhancing the models' generalization capabilities. It can be seamlessly integrated into NeRF architectures and cultivates a comprehensive feature set that significantly improves accuracy and robustness in image rendering. Through this integration, NeuGen shows improved performance on benchmarks on diverse datasets across state-of-the-art NeRF architectures, enabling them to generalize better across varied scenes. Our comprehensive evaluations, both quantitative and qualitative, confirm that our approach not only surpasses existing models in generalizability but also markedly improves rendering quality. Our work exemplifies the potential of merging neuroscientific principles with deep learning frameworks, setting a new precedent for enhanced generalizability and efficiency in novel view synthesis. A demo of our study is available at https://neugennerf.github.io.

Figures (6)

• Figure 1: Block diagram of the proposed methodology: The figure shows the comprehensive overview of our domain-invariant data representation pipeline. The original images $(I)$ are first partitioned into patches and then processed by each layer in the pipeline, as shown by the equations. The output after processing from all the blocks of the NeuGen layer is a NeuGen image $(I^{G})$. The inspiration for this approach is taken from the mammalian visual cortex, which similarly interprets scenes. The next step merges NeuGen images with the original images, producing NeuGen-enhanced images $(I^{E})$, contributing to better feature extraction for input to NeRF. The right part of the figure illustrates the volumetric rendering step, a fundamental process shared by all NeRF methods like MVSNeRF chen2021mvsnerf and GeoNeRF johari2022geonerf. However, it is essential to note that the specific details and implementations of feature volume vary with each architecture, tailored to their unique enhancements and optimization strategies.
• Figure 2: Feature enhancement with NeuGen across datasets. The top images (A) from left to right illustrate samples from the Realistic Syntheticnerfsynthetic, DTUDTU, and LLFFmildenhall2019local datasets, respectively, paired with their NeuGen versions. The NeuGen images result in notably higher SSIM scores and a greater count of SIFT feature matches, signifying improved feature detection. The graphs below (B,C,D) detail these improvements: each dot represents the average SSIM score between one reference image and the rest within a category, while the overlaying lines trace the general pattern of quality enhancement—yellow for the original and purple for NeuGen mages — demonstrating NeuGen's consistent efficacy in feature emphasis across all 3 datasets.
• Figure 3: Qualitative comparison on Realistic Synthetic for MVSNeRF chen2021mvsnerf and GeoNeRF johari2022geonerf along with our approach. We show results on Ficus and Drums rendered with 3 source images. Our method synthesizes homogenous regions and complex geometries better.
• Figure 4: Qualitative comparison on LLFF Dataset for MVSNeRF chen2021mvsnerf and GeoNeRF johari2022geonerf along with our approach. While our method performs better on wiry structures in plants, it performs significantly well on reflective/transparent surfaces as well.
• Figure 5: Qualitative comparison on DTU Dataset for MVSNeRF chen2021mvsnerf and GeoNeRF johari2022geonerf along with our approach. Our method performs better regardless of light conditions while also preventing cloudy artefacts or noise in the background.