Slot-guided Volumetric Object Radiance Fields

TL;DR

sVORF introduces slot-guided volumetric object radiance fields to achieve unsupervised 3D object-centric scene decomposition from a single image. It connects 2D-derived object slots to per-object NeRFs via a hypernetwork and composes these NeRFs with a slot-guided attention mechanism for 3D-consistent novel view synthesis. The method achieves state-of-the-art 3D segmentation and high-quality novel views on synthetic datasets, while also showing promising results on real-world LLFF data, and it significantly reduces training memory and time compared to previous volumetric methods. The approach fuses 3D geometric bias with 2D priors, enabling robust object separation, controllable scene editing, and efficient training, advancing practical 3D object-centric representation learning.

Abstract

We present a novel framework for 3D object-centric representation learning. Our approach effectively decomposes complex scenes into individual objects from a single image in an unsupervised fashion. This method, called slot-guided Volumetric Object Radiance Fields (sVORF), composes volumetric object radiance fields with object slots as a guidance to implement unsupervised 3D scene decomposition. Specifically, sVORF obtains object slots from a single image via a transformer module, maps these slots to volumetric object radiance fields with a hypernetwork and composes object radiance fields with the guidance of object slots at a 3D location. Moreover, sVORF significantly reduces memory requirement due to small-sized pixel rendering during training. We demonstrate the effectiveness of our approach by showing top results in scene decomposition and generation tasks of complex synthetic datasets (e.g., Room-Diverse). Furthermore, we also confirm the potential of sVORF to segment objects in real-world scenes (e.g., the LLFF dataset). We hope our approach can provide preliminary understanding of the physical world and help ease future research in 3D object-centric representation learning.

Figures (11)

• Figure 1: sVORF overview. The image encoder $E_{\theta}(\mathbf{I})$ processes the source view of a scene to generate 2D image features that serve as a prior. Next, these features are fed into the Scene Decomposition module to infer object and background slots. A hypernetwork then maps these slots to volumetric object radiance fields. Finally, the object slots provide guidance for the recombination of object radiance fields to render arbitrary views with 3D-consistent object decomposition.
• Figure 2: Qualitative results of sVORF on MSN.
• Figure 3: Qualitative Comparison. We compare the reconstructions of the input view, a novel view, as well as the novel view segmentation using the uORF yu2021unsupervised, COLF smith2022unsupervised, OSRT sajjadi2022object, and our method on four datasets. Our method produces a finer segmentation and more precise shapes.
• Figure 4: 3D scene manipulation for moving object and changing background.
• Figure 5: Qualitative comparison of ablation studies on CLEVR-567 dataset. Specifically, we evaluate the impact of four techniques - Novel View Synthesis (NVS), Connectivity Regularization (CR), Composing Mechanism, and Self-Attention (SA).