LLaVA$^3$: Representing 3D Scenes like a Cubist Painter to Boost 3D Scene Understanding of VLMs

TL;DR

LLaVA$^3$ tackles the challenge of 3D scene understanding with vision-language models by eliminating the need for VLM fine-tuning and relying on multi-view 2D imagery. The approach recasts 3D scenes as a Cubist-inspired collection of per-object omni-directional visual tokens derived from a hierarchical NeRF-based feature field, separating view-invariant semantics from view-dependent spatial cues. By constructing a discrete object hierarchy and assigning object-centric token descriptions, LLaVA$^3$ enables accurate 3D VQA, grounding, and segmentation, outperforming several 2D VLM-based and NeRF-based baselines across ScanNet, ScanQA, and MSR3D datasets. The method demonstrates strong zero-shot generalization to diverse environments, offering a scalable, modular pipeline that leverages existing 2D encoders while avoiding costly fine-tuning or large-scale 3D data collection.

Abstract

Developing a multi-modal language model capable of understanding 3D scenes remains challenging due to the limited availability of 3D training data, in contrast to the abundance of 2D datasets used for vision-language models (VLM). As an alternative, we introduce LLaVA$^3$ (pronounced LLaVA-Cube), a novel method that improves the 3D scene understanding capabilities of VLM using only multi-view 2D images and without any fine-tuning. Inspired by Cubist painters, who represented multiple viewpoints of a 3D object within a single picture, we propose to describe the 3D scene for the VLM through omnidirectional visual representations of each object. These representations are derived from an intermediate multi-view 3D reconstruction of the scene. Extensive experiments on 3D VQA and 3D language grounding show that our approach outperforms previous 2D-based VLM solutions.

Figures (13)

• Figure 1: Overview of LLaVA$^3$. We first reconstruct the 3D scene as a NeRF from multi-view images with an associated LLaVA feature field. We also derive a hierarchical 3D segmentation of our NeRF. For each object, we create an omni-directional visual-description as a set of tokens. After object re-ordering, we can finally feed them to the VLM for 3D interpretation.
• Figure 2: Qualitative performance of our method on 3D VQA and Grounding. By decomposing into objects the view-dependent features, our method avoids several very common issues in 3D VQA: (a) missing objects due to insufficient sampling, (b) weak inter-object spatial relationships, (c) loss of objects details and (d) bad multi-view understanding (i.e. relations between objects from different images). Our method can also perform grounding from different types of queries ((e),(f),(g),(h)).
• Figure 3: Precise Architecture of the Feature Fields. $N$ stands for "Normalization".
• Figure 4: NeRF Semantic Segmentation Performance on Replica's room0 scene.
• Figure 5: Examples of the Impact of Filtering on our segmentation.