G3Flow: Generative 3D Semantic Flow for Pose-aware and Generalizable Object Manipulation

TL;DR

G3Flow tackles the challenge of integrating semantic understanding with geometric precision in 3D robotic manipulation. It introduces a two-phase framework that builds a real-time semantic flow from digital twins and vision foundation models, then maintains it during manipulation via robust pose tracking to guide diffusion-based policies. The approach yields notable improvements in pose-aware manipulation and cross-object generalization, validated on RoboTwin tasks with strong performance gains over baselines. By coupling 3D generation, semantic feature extraction, and real-time tracking, G3Flow demonstrates the practical viability of semantic-aware manipulation in occluded and dynamic scenarios.

Abstract

Recent advances in imitation learning for 3D robotic manipulation have shown promising results with diffusion-based policies. However, achieving human-level dexterity requires seamless integration of geometric precision and semantic understanding. We present G3Flow, a novel framework that constructs real-time semantic flow, a dynamic, object-centric 3D semantic representation by leveraging foundation models. Our approach uniquely combines 3D generative models for digital twin creation, vision foundation models for semantic feature extraction, and robust pose tracking for continuous semantic flow updates. This integration enables complete semantic understanding even under occlusions while eliminating manual annotation requirements. By incorporating semantic flow into diffusion policies, we demonstrate significant improvements in both terminal-constrained manipulation and cross-object generalization. Extensive experiments across five simulation tasks show that G3Flow consistently outperforms existing approaches, achieving up to 68.3% and 50.1% average success rates on terminal-constrained manipulation and cross-object generalization tasks respectively. Our results demonstrate the effectiveness of G3Flow in enhancing real-time dynamic semantic feature understanding for robotic manipulation policies.

Figures (10)

• Figure 1: Motivation of G3Flow. Our approach leverages 3D generative model and language-guided detection model to generate 3D semantic flow (top). Through continuous field tracking, G3Flow enables pose-aware and generalizable manipulation, demonstrating superior performance across terminal constraint control and cross-object generalization tasks over multiple baselines (DP, DP3, and DP3 w/ color) (bottom).
• Figure 2: Pipeline of G3Flow. Our framework consists of (top) an initialization phase that generates comprehensive 3D representation (surface normals, wireframe, and geometry) through object-centric exploration and digital twin generation, which enables rich semantic field extraction, and (bottom) a control execution phase where real-time pose tracking maintains dynamic semantic fields to guide diffusion-based manipulation actions for pose-aware and generalizable manipulation.
• Figure 3: Failure mode of single-view 3D generation. When using a single view for 3D generation, certain geometric details may be inaccurately reconstructed due to occlusion, even if the result appears plausible from a commonsense perspective.
• Figure 4: Spatial alignment via object tracking. We achieve alignment between the semantic flow and the physical object in real world by synchronizing the relative transformations of the object coordinate system to the world coordinate system.
• Figure 5: G3Flow-enhanced diffusion policy.