ArtiSG: Functional 3D Scene Graph Construction via Human-demonstrated Articulated Objects Manipulation

TL;DR

ArtiSG targets the challenge of extracting functional, manipulable information from articulated objects beyond what traditional semantic scene graphs provide. It fuses static multi-view perception with human-demonstrated $6$-DoF articulation trajectories via a portable, viewpoint-robust pipeline and an interaction-augmented refinement stage to uncover inconspicuous functional elements. The framework builds a hierarchical, open-vocabulary functional scene graph that serves as robotic memory, enabling accurate articulation parameter estimation and improved open-vocabulary querying, both in simulation and real-world environments. Empirical results show ArtiSG outperforms baselines in functional element recall, articulation accuracy, and real-world language-guided manipulation tasks, demonstrating its potential to enhance robot autonomy in diverse articulated-object scenarios.

Abstract

3D scene graphs have empowered robots with semantic understanding for navigation and planning, yet they often lack the functional information required for physical manipulation, particularly regarding articulated objects. Existing approaches for inferring articulation mechanisms from static observations are prone to visual ambiguity, while methods that estimate parameters from state changes typically rely on constrained settings such as fixed cameras and unobstructed views. Furthermore, fine-grained functional elements like small handles are frequently missed by general object detectors. To bridge this gap, we present ArtiSG, a framework that constructs functional 3D scene graphs by encoding human demonstrations into structured robotic memory. Our approach leverages a robust articulation data collection pipeline utilizing a portable setup to accurately estimate 6-DoF articulation trajectories and axes even under camera ego-motion. We integrate these kinematic priors into a hierarchical and open-vocabulary graph while utilizing interaction data to discover inconspicuous functional elements missed by visual perception. Extensive real-world experiments demonstrate that ArtiSG significantly outperforms baselines in functional element recall and articulation estimation precision. Moreover, we show that the constructed graph serves as a reliable functional memory that effectively guides robots to perform language-directed manipulation tasks in real-world environments containing diverse articulated objects.

Figures (6)

• Figure 1: Constructing Functional Scene Graphs via Human Demonstration. The bottom film strips show our manipulation sequences using a custom UMI gripper. From these sequences, we extract articulation trajectories and estimate axes, registering them to the corresponding element nodes in the graph. This structured representation enables open-vocabulary queries to locate functional elements and provides actionable priors for robot manipulation.
• Figure 2: System Overview. Our approach to building the functional scene graph for an indoor room unfolds in three stages. Firstly, the construction begins with the initialization of an element-aware scene representation, where we aggregate multi-view semantics to detect and generate object and functional element nodes that are explicitly visible. Secondly, we leverage a portable setup to track human manipulation, enabling the extraction of precise motion trajectories and the estimation of articulation axes for articulated objects. Finally, we perform interaction-augmented graph refinement, utilizing these human demonstrations to recover inconspicuous functional elements missed in the initial phase and enrich element nodes with articulation kinematic attributes.
• Figure 3: Hardware setup for articulation data collection. The handheld UMI gripper is equipped with a custom 26-sided ArUco tracking sphere, enabling robust 6-DoF pose estimation via a head-mounted camera. OptiTrack retro-reflective markers are attached to the cabinet door to provide ground truth poses for the quantitative evaluation in \ref{['sec: eval_tracking']}.
• Figure 4: Qualitative comparison of open-vocabulary querying performance. We compare the retrieval results of ArtiSG against baselines Lost&Found and OpenFunGraph in both real-world (left) and simulated (right) scenes. Green dots indicate the ground truth functional elements. As shown, our method accurately localizes target elements with high recall, whereas baselines often suffer from missed detections or imprecise localization.
• Figure 5: Visualization of the viewpoint-robust articulation tracking process. Subfigures (a) and (b) depict the start and end phases of manipulating a prismatic joint, while (c) and (d) show the manipulation of a revolute joint. The distinct coordinate frames for the World, Camera, and Sphere are highlighted to illustrate our decoupled tracking setup. The recovered gripper trajectory is visualized as an orange curve, demonstrating smooth and precise tracking performance.