Depth-PC: A Visual Servo Framework Integrated with Cross-Modality Fusion for Sim2Real Transfer

Abstract

Visual servoing techniques guide robotic motion using visual information to accomplish manipulation tasks, requiring high precision and robustness against noise. Traditional methods often require prior knowledge and are susceptible to external disturbances. Learning-driven alternatives, while promising, frequently struggle with the scarcity of training data and fall short in generalization. To address these challenges, we propose Depth-PC, a novel visual servoing framework that leverages decoupled simulation-based training from real-world inference, achieving zero-shot Sim2Real transfer for servo tasks. To exploit spatial and geometric information of depth and point cloud features, we introduce cross-modal feature fusion, a first in servo tasks, followed by a dedicated Graph Neural Network to establish keypoint correspondences. Through simulation and real-world experiments, our approach demonstrates superior convergence basin and accuracy compared to SOTA methods, fulfilling the requirements for robotic servo tasks while enabling zero-shot Sim2Real transfer. In addition to the enhancements achieved with our proposed framework, we have also demonstrated the effectiveness of cross-modality feature fusion within the realm of servo tasks. Code is available at https://github.com/3nnui/Depth-PC.

Figures (10)

• Figure 1: Comparative visualization between visual servoing methods based on 2D correspondence and ours. In the inference stage, apart from the feature detector, We utilize the off-shelf depth estimator to inject spatial and geometric cues into the system.
• Figure 2: Overview of our framework. The proposed framework is separated into training and inference components. During training, processed object point clouds are fed into a Graph Neural Network (GNN)28. In the inference phase, the inputs are derived from point clouds and simulation-aligned depth features generated by a relative depth estimator. The fused features are then used to construct graph relationships, which produce 6-DOF velocity to guide the robot in executing servo tasks.
• Figure 3: The illustration shows the process of data generation. Point clouds of each object are colored to distinguish their differences, with average number of points in the middle. It indicates that our data itself contains certain spatial and geometric information.
• Figure 4: Our feature fusion module are mainly made of Feature Alignment Layer and Cluster Cross Attention. The former aligns two feature spaces, while the latter fuses the positional features of each cluster with the depth features of all points. Followed by GNN and Velocity Head, we finally get the 6-DOF velocity.
• Figure 5: Visualization of depth results. The figure on the left illustrates the depth maps rendered within the simulation environment, along with keypoint clouds detected; the figure on the right presents frames and keypoint clouds processed by the depth estimator. It is obvious that depth distributions of point clouds between them share a high degree of similarity.