Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Open-vocabulary 3D scene perception in industrial environments

Keno Moenck, Adrian Philip Florea, Julian Koch, Thorsten Schüppstuhl

TL;DR

The paper addresses the problem of open-vocabulary 3D scene perception in industrial settings where data scarcity and domain shift limit traditional methods. It proposes a training-free pipeline that replaces pre-trained class-agnostic 3D masks with a superpoint-based masking approach, projecting into 2D views and using SAM/CLIP (IndustrialCLIP) for open-vocabulary querying. Qualitative evaluation on a real workshop scene demonstrates successful segmentation of industrial objects and provides insights into the strengths and limitations of IndustrialCLIP, including overfitting to catalog-style imagery and language breadth issues. Overall, the work shows promise for domain-adapted VLFM in 3D perception and highlights practical considerations for language grounding and robust open-vocabulary industrial perception.

Abstract

Autonomous vision applications in production, intralogistics, or manufacturing environments require perception capabilities beyond a small, fixed set of classes. Recent open-vocabulary methods, leveraging 2D Vision-Language Foundation Models (VLFMs), target this task but often rely on class-agnostic segmentation models pre-trained on non-industrial datasets (e.g., household scenes). In this work, we first demonstrate that such models fail to generalize, performing poorly on common industrial objects. Therefore, we propose a training-free, open-vocabulary 3D perception pipeline that overcomes this limitation. Instead of using a pre-trained model to generate instance proposals, our method simply generates masks by merging pre-computed superpoints based on their semantic features. Following, we evaluate the domain-adapted VLFM "IndustrialCLIP" on a representative 3D industrial workshop scene for open-vocabulary querying. Our qualitative results demonstrate successful segmentation of industrial objects.

Open-vocabulary 3D scene perception in industrial environments

TL;DR

The paper addresses the problem of open-vocabulary 3D scene perception in industrial settings where data scarcity and domain shift limit traditional methods. It proposes a training-free pipeline that replaces pre-trained class-agnostic 3D masks with a superpoint-based masking approach, projecting into 2D views and using SAM/CLIP (IndustrialCLIP) for open-vocabulary querying. Qualitative evaluation on a real workshop scene demonstrates successful segmentation of industrial objects and provides insights into the strengths and limitations of IndustrialCLIP, including overfitting to catalog-style imagery and language breadth issues. Overall, the work shows promise for domain-adapted VLFM in 3D perception and highlights practical considerations for language grounding and robust open-vocabulary industrial perception.

Abstract

Autonomous vision applications in production, intralogistics, or manufacturing environments require perception capabilities beyond a small, fixed set of classes. Recent open-vocabulary methods, leveraging 2D Vision-Language Foundation Models (VLFMs), target this task but often rely on class-agnostic segmentation models pre-trained on non-industrial datasets (e.g., household scenes). In this work, we first demonstrate that such models fail to generalize, performing poorly on common industrial objects. Therefore, we propose a training-free, open-vocabulary 3D perception pipeline that overcomes this limitation. Instead of using a pre-trained model to generate instance proposals, our method simply generates masks by merging pre-computed superpoints based on their semantic features. Following, we evaluate the domain-adapted VLFM "IndustrialCLIP" on a representative 3D industrial workshop scene for open-vocabulary querying. Our qualitative results demonstrate successful segmentation of industrial objects.
Paper Structure (15 sections, 7 figures)

This paper contains 15 sections, 7 figures.

Table of Contents

  1. Introduction
  2. Fundamentals
  3. Related works
  4. Open-vocabulary 3D scene understanding
  5. 3D scene understanding in industrial environments
  6. Preliminary experiments
  7. Method
  8. Superpoints
  9. Feature Extraction
  10. Superpoint merging
  11. Open-vocabulary querying
  12. Experiments
  13. Single prompt scores
  14. Comparison to CLIP
  15. Discussion and conclusion

Figures (7)

  • Figure 1: Prompting "vise" using features from CLIP and IndustrialCLIP (yellow and blue-colored points correspond to high and low semantic similarity scores, respectively).
  • Figure 2: The industrial scene under study: (a) 3D scan cut-out, and sample images from (b, c) two different viewpoints.
  • Figure 3: Mask3D-generated mask proposals (color-coded): evaluated as high (white boxes: cabinet [3], chair [5]) and low quality masks (black boxes: table saw [1], vise [2], lathe [4]).
  • Figure 4: Superpoint-based oversegmentation of the workshop scene.
  • Figure 5: Masking leads to a more precise feature representation. Compared to the unmasked version (a), masking (b) makes it clear which specific object is being considered.
  • ...and 2 more figures