Language-Assisted 3D Scene Understanding

TL;DR

LAST-PCL addresses data scarcity in 3D point cloud understanding by injecting language priors through LLM-generated, fine-grained text descriptions and aligning them with point features via a text-contrastive framework. A training-free, statistically driven feature selection preserves textual priors while reducing dimensionality and redundancy, enabling efficient cross-modal supervision without paired triplets. Across 3D semantic segmentation, object detection, and scene classification, the method achieves state-of-the-art or competitive results, with notable gains on large or fine-grained category sets and robust generalization to new text queries. The work highlights the role of semantic-rich text in guiding 3D perception and demonstrates flexible, scalable inference through language, including potential image-query enhancements via CLIP.

Abstract

The scale and quality of point cloud datasets constrain the advancement of point cloud learning. Recently, with the development of multi-modal learning, the incorporation of domain-agnostic prior knowledge from other modalities, such as images and text, to assist in point cloud feature learning has been considered a promising avenue. Existing methods have demonstrated the effectiveness of multi-modal contrastive training and feature distillation on point clouds. However, challenges remain, including the requirement for paired triplet data, redundancy and ambiguity in supervised features, and the disruption of the original priors. In this paper, we propose a language-assisted approach to point cloud feature learning (LAST-PCL), enriching semantic concepts through LLMs-based text enrichment. We achieve de-redundancy and feature dimensionality reduction without compromising textual priors by statistical-based and training-free significant feature selection. Furthermore, we also delve into an in-depth analysis of the impact of text contrastive training on the point cloud. Extensive experiments validate that the proposed method learns semantically meaningful point cloud features and achieves state-of-the-art or comparable performance in 3D semantic segmentation, 3D object detection, and 3D scene classification tasks.

Figures (9)

• Figure 1: Different semantic segmentation pipelines. (a) Standard one-hot label supervision method. $N$ is the number of input points, $d_i$ is the input dimension, $d_o$ is the output dimension, and $k$ is the number of classes. (b) Text contrastive training, where text features keep the original dimension and point features are transformed to high dimension by a projection layer. $m$ is the number of sentences. (c) Text contrastive training, with text features transformed to low dimension via a projection layer.
• Figure 2: Model Architecture. Left: Training process. (a) Diversified fine-grained descriptions generated by LLM for GT categories, achieving text enrichment. (b) Extracting features from the generated text using a frozen text encoder and obtaining averaged features for multiple sentences of the same category. (c) Statistical-based significance feature selection for text feature dimensional reduction. (e) Point cloud backbone extracting dense point cloud features. (f) Semantic segmentation. (g) 3D object detection. (h) 3D scene type classification. Projector layers transform point features into the same dimension as text features. (d) Text-point cloud contrastive loss. Right: Inference process, illustrated by the segmentation task.
• Figure 3: PCA visualization of features. Colors depict categories. The features learned by ours exhibit continuity, unlike the discreteness of PointTransformerV2.
• Figure 4: (a) Query images generated by DALL·E ramesh2021zero. (b) Coloured regions show highly similar points to the query image. (c) Reference 3D mesh for readers.
• Figure 5: Generalization comparison on S3DIS, with models trained on ScanNet20. Project-based method shows significant decline.