Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models

TL;DR

Diff2Scene introduces a two-branch open-vocabulary 3D semantic understanding framework that leverages frozen text-to-image diffusion representations to perform zero-shot 3D segmentation and visual grounding without 3D labels. The 2D branch provides semantically rich mask embeddings from a diffusion backbone, while the 3D branch learns geometry-aware masks via mask distillation and lifting of 2D masks to 3D. Inference fuses diffusion-based and discriminative grounding to assign per-point labels, achieving state-of-the-art zero-shot results on ScanNet, ScanNet200, Matterport3D, Replica, and effective visual grounding on Nr3D. The work demonstrates strong generalization and highlights diffusion models as a potent source of local, semantically rich features for 3D open-vocabulary perception, while noting limitations with small or fine-grained categories and suggesting future refinement.

Abstract

In this paper, we investigate the use of diffusion models which are pre-trained on large-scale image-caption pairs for open-vocabulary 3D semantic understanding. We propose a novel method, namely Diff2Scene, which leverages frozen representations from text-image generative models, along with salient-aware and geometric-aware masks, for open-vocabulary 3D semantic segmentation and visual grounding tasks. Diff2Scene gets rid of any labeled 3D data and effectively identifies objects, appearances, materials, locations and their compositions in 3D scenes. We show that it outperforms competitive baselines and achieves significant improvements over state-of-the-art methods. In particular, Diff2Scene improves the state-of-the-art method on ScanNet200 by 12%.

Figures (5)

• Figure 1: Illustration of open-vocabulary 3D semantic scene understanding. We propose Diff2Scene, a 3D model that performs open-vocabulary semantic segmentation and visual grounding tasks given novel text prompts, without relying on any annotated 3D data. By leveraging discriminative-based and generative-based 2D foundation models, Diff2Scene can handle a wide variety of novel text queries for both common and rare classes, like "desk" and "soap dispenser". It can also handle compositional queries, such as "find the white sneakers that are closer to the desk chair."
• Figure 2: Illustration of open-vocabulary 3D perception methods.$L_{PD}$ and $L_{MD}$ denote point-based distillation loss and mask-based distillation loss. $M_{3D}$ denote a set of predicted 3D masks; $M_{2D}$ and $Z_{mf}$ denote a set of predicted 2D masks and their semantic embeddings; $Z_{gf}$ denote the high-resolution 3D feature map. (a) Directly minimizing the per-point feature distance between the CLIP-based model and the tuned 3D model openscene. (b) Directly using a 3D mask proposal network trained on labeled 3D data to produce class-agnostic masks, and then pool corresponding representations from the CLIP feature map takmaz2023openmask3d. (c) The proposed mask distillation approach, namely Diff2Scene, that uses Stable Diffusion and performs mask-based distillation. Diff2Scene leverages the semantically-rich mask embeddings from 2D foundation models and geometrically accurate masks from the tuned 3D model, and thus achieves superior performance compared to previous methods.
• Figure 3: Overview of our method. We propose Diff2Scene, an open-vocabulary 3D semantic understanding model. Diff2Scene contains two branches. The 2D branch is designed to be a diffusion-based 2D semantic segmentation model. It accepts a 2D image as input and predicts a set of 2D probabilistic masks with corresponding semantically-rich mask embeddings. The 3D branch utilizes the point cloud and 2D mask embeddings as input. The 2D mask embeddings are used as "semantic queries" to generate corresponding 3D probabilistic masks. The model learns salient patterns from the RGB images and geometric information from the point clouds.
• Figure 4: Qualitative results from our model and OpenScene on zero-shot semantic segmentation. We visualize the segmentation results on the validation set of ScanNet200 rozenberszki2022language. We observe that our model can predict coherent masks with accurate semantic labels compared to OpenScene for both head and tail categories.
• Figure 5: Qualitative results from our model and OpenScene on zero-shot visual grounding. Our open-vocabulary semantic understanding model is capable of handling different types of novel and compositional queries. Novel object classes as well as objects described by colors, shapes, appearances, locations, and usages are successfully retrieved by our method. Note that the located points are colored in yellow.