Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Semantic Enrichment of CAD-Based Industrial Environments via Scene Graphs for Simulation and Reasoning

Nathan Pascal Walus, Ranulfo Bezerra, Shotaro Kojima, Tsige Tadesse Alemayoh, Satoshi Tadokoro, Kazunori Ohno

TL;DR

The paper tackles the challenge of enriching CAD-based industrial environments with semantic, spatial, and functional information to enable advanced robotic simulation and reasoning. It introduces an offline pipeline that labels CAD meshes with a Large Visual-Language Model, clusters them with DBSCAN, and builds a multi-layered 3D scene graph that captures both geometry and functional dependencies, particularly within pipe systems. Key contributions include a vocabulary-guided LVLM labeling workflow, a scalable CAD-to-scene-graph construction, and a method to infer functional relations from the graph to support dynamic simulation and planning. The results demonstrate the feasibility of generating informative scene graphs from complex industrial CAD models and discuss practical considerations, limitations, and future improvements to enhance labeling accuracy and functional inference for real-world deployments.

Abstract

Utilizing functional elements in an industrial environment, such as displays and interactive valves, provide effective possibilities for robot training. When preparing simulations for robots or applications that involve high-level scene understanding, the simulation environment must be equally detailed. Although CAD files for such environments deliver an exact description of the geometry and visuals, they usually lack semantic, relational and functional information, thus limiting the simulation and training possibilities. A 3D scene graph can organize semantic, spatial and functional information by enriching the environment through a Large Vision-Language Model (LVLM). In this paper we present an offline approach to creating detailed 3D scene graphs from CAD environments. This will serve as a foundation to include the relations of functional and actionable elements, which then can be used for dynamic simulation and reasoning. Key results of this research include both quantitative results of the generated semantic labels as well as qualitative results of the scene graph, especially in hindsight of pipe structures and identified functional relations. All code, results and the environment will be made available at https://cad-scenegraph.github.io

Semantic Enrichment of CAD-Based Industrial Environments via Scene Graphs for Simulation and Reasoning

TL;DR

The paper tackles the challenge of enriching CAD-based industrial environments with semantic, spatial, and functional information to enable advanced robotic simulation and reasoning. It introduces an offline pipeline that labels CAD meshes with a Large Visual-Language Model, clusters them with DBSCAN, and builds a multi-layered 3D scene graph that captures both geometry and functional dependencies, particularly within pipe systems. Key contributions include a vocabulary-guided LVLM labeling workflow, a scalable CAD-to-scene-graph construction, and a method to infer functional relations from the graph to support dynamic simulation and planning. The results demonstrate the feasibility of generating informative scene graphs from complex industrial CAD models and discuss practical considerations, limitations, and future improvements to enhance labeling accuracy and functional inference for real-world deployments.

Abstract

Utilizing functional elements in an industrial environment, such as displays and interactive valves, provide effective possibilities for robot training. When preparing simulations for robots or applications that involve high-level scene understanding, the simulation environment must be equally detailed. Although CAD files for such environments deliver an exact description of the geometry and visuals, they usually lack semantic, relational and functional information, thus limiting the simulation and training possibilities. A 3D scene graph can organize semantic, spatial and functional information by enriching the environment through a Large Vision-Language Model (LVLM). In this paper we present an offline approach to creating detailed 3D scene graphs from CAD environments. This will serve as a foundation to include the relations of functional and actionable elements, which then can be used for dynamic simulation and reasoning. Key results of this research include both quantitative results of the generated semantic labels as well as qualitative results of the scene graph, especially in hindsight of pipe structures and identified functional relations. All code, results and the environment will be made available at https://cad-scenegraph.github.io
Paper Structure (16 sections, 4 figures, 1 table)

This paper contains 16 sections, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Vocabulary
  5. Scene Graph
  6. Results
  7. Clustering Results
  8. Clustering Labeling Results
  9. Functional Relation Analysis
  10. Discussion
  11. CAD Meshes vs. Point Cloud Processing
  12. Semantic Labeling
  13. Vocabulary
  14. Functional Relation Analysis
  15. Limitations
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Overview of the process of generating a 3D scene graph from a CAD environment.
  • Figure 4: Visualization of the scene graph, semantic data and functional analysis pipeline for a selected pipe structure. a) The original rendered geometry. b) The generated spatial scene graph, where nodes (spheres) represent meshes and edges indicate spatial proximity. c) Semantic 'group' labels assigned by gpt-4o. d) The manually assigned ground truth labels. e) The extracted functional graph (functional units shown in red) derived from the LVLM-generated semantics in c). f) The extracted functional graph derived from the ground truth semantics in d).
  • Figure 5: Distribution of generated semantic labels in the used CAD environment. Similar colors indicate that the semantic group label is equal. Used group labels but not separately visualized are 'Pump Unit' and 'Connection Assembly'. The amount of labeled meshes is 2068. * Valve combines all valve-related semantic labels. ** Others contain semantic labels with a quantity less than 25.
  • Figure 6: Three selected structures, where each is a clustering result of using DBSCAN. The left side shows the structures in normal sight. The right side shows the semantic 'group' label and identified functional relations.