VisuoTactile 6D Pose Estimation of an In-Hand Object using Vision and Tactile Sensor Data

TL;DR

This work tackles 6D pose estimation for objects grasped in a robot hand under heavy occlusion, where vision alone struggles. It introduces a visuo-tactile network with pixel-wise fusion and a tactile-point-cloud representation that is sensor-agnostic, augmented by synthetic data generated via an extended NDDS pipeline. The approach yields significant improvements over vision-only baselines, demonstrates sim-to-real transfer on multiple hardware setups, and analyzes robustness to occlusion and tactile point density. The findings highlight the practical potential of combining vision with tactile sensing for robust in-hand manipulation tasks, while outlining future work on temporal coherence and geometric priors to further enhance orientation accuracy.

Abstract

Knowledge of the 6D pose of an object can benefit in-hand object manipulation. In-hand 6D object pose estimation is challenging because of heavy occlusion produced by the robot's grippers, which can have an adverse effect on methods that rely on vision data only. Many robots are equipped with tactile sensors at their fingertips that could be used to complement vision data. In this paper, we present a method that uses both tactile and vision data to estimate the pose of an object grasped in a robot's hand. To address challenges like lack of standard representation for tactile data and sensor fusion, we propose the use of point clouds to represent object surfaces in contact with the tactile sensor and present a network architecture based on pixel-wise dense fusion. We also extend NVIDIA's Deep Learning Dataset Synthesizer to produce synthetic photo-realistic vision data and corresponding tactile point clouds. Results suggest that using tactile data in addition to vision data improves the 6D pose estimate, and our network generalizes successfully from synthetic training to real physical robots.

Figures (11)

• Figure 1: An illustration of the proposed method. The input to our model is color and depth images from a camera, and tactile contact points in the form of a point cloud. The output is the estimated 6D pose of the object of interest.
• Figure 2: The proposed network architecture: given a color image, depth image and the object-surface point cloud at finger-object contact location, the network estimates the $6D$ pose of the object. See Fig. \ref{['fig:pose_estimator']} for details of pose estimation network.
• Figure 3: The network architecture for the pose estimator. The colors blue, green and yellow denote visual-channel features, tactile-channel features, and global feature, respectively.
• Figure 4: An illustration of tactile data to object surface point cloud generation: a) Optical-based tactile sensor and the corresponding point-cloud, b) Pressure-based tactile sensor and the corresponding tactile point-cloud.
• Figure 5: a) Unreal Engine data collection setup with domain randomization b) The Allegro Hand with tactile sensors. The yellow ellipses show the location of virtual depth cameras used to generate tactile data in simulation.