Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging 2D-VLM for Label-Free 3D Segmentation in Large-Scale Outdoor Scene Understanding

Toshihiko Nishimura, Hirofumi Abe, Kazuhiko Murasaki, Taiga Yoshida, Ryuichi Tanida

TL;DR

The paper tackles the challenge of 3D semantic segmentation for large-scale outdoor point clouds without relying on annotated 3D data or paired RGB imagery. It introduces a training-free approach that renders a LiDAR-scene along a virtual camera path, uses a 2D vision–language model with natural-language prompts to obtain semantic labels in each view, and fuses the labels back into 3D via weighted voting. A bird's-eye refinement option further enhances recognition for large or elongated objects. The method achieves competitive performance to supervised methods and enables open-vocabulary detection, offering scalable, label-free 3D scene understanding for applications like autonomous driving and digital twins.

Abstract

This paper presents a novel 3D semantic segmentation method for large-scale point cloud data that does not require annotated 3D training data or paired RGB images. The proposed approach projects 3D point clouds onto 2D images using virtual cameras and performs semantic segmentation via a foundation 2D model guided by natural language prompts. 3D segmentation is achieved by aggregating predictions from multiple viewpoints through weighted voting. Our method outperforms existing training-free approaches and achieves segmentation accuracy comparable to supervised methods. Moreover, it supports open-vocabulary recognition, enabling users to detect objects using arbitrary text queries, thus overcoming the limitations of traditional supervised approaches.

Leveraging 2D-VLM for Label-Free 3D Segmentation in Large-Scale Outdoor Scene Understanding

TL;DR

The paper tackles the challenge of 3D semantic segmentation for large-scale outdoor point clouds without relying on annotated 3D data or paired RGB imagery. It introduces a training-free approach that renders a LiDAR-scene along a virtual camera path, uses a 2D vision–language model with natural-language prompts to obtain semantic labels in each view, and fuses the labels back into 3D via weighted voting. A bird's-eye refinement option further enhances recognition for large or elongated objects. The method achieves competitive performance to supervised methods and enables open-vocabulary detection, offering scalable, label-free 3D scene understanding for applications like autonomous driving and digital twins.

Abstract

This paper presents a novel 3D semantic segmentation method for large-scale point cloud data that does not require annotated 3D training data or paired RGB images. The proposed approach projects 3D point clouds onto 2D images using virtual cameras and performs semantic segmentation via a foundation 2D model guided by natural language prompts. 3D segmentation is achieved by aggregating predictions from multiple viewpoints through weighted voting. Our method outperforms existing training-free approaches and achieves segmentation accuracy comparable to supervised methods. Moreover, it supports open-vocabulary recognition, enabling users to detect objects using arbitrary text queries, thus overcoming the limitations of traditional supervised approaches.
Paper Structure (5 sections, 3 equations, 4 figures, 1 table)

This paper contains 5 sections, 3 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Experimental Results
  5. Conclusion

Figures (4)

  • Figure 1: Test scenes used in our experiments. Scene 1 contains a tunnel, and Scene 2 contains a bridge.
  • Figure 2: Voting Merge. Point clouds from virtual cameras are weighted by confidence score and distance from the camera, then voted on. The highest-scoring label becomes the final result.
  • Figure 3: Matched Results from 2D to 3D. 2D segmentation results are transferred to 3D point clouds
  • Figure 4: Open vocabulary segmentation. Arbitrary objects can be recognized with 2D-VLM applied to 2D rendered images.