Simultaneous Extrinsic Contact and In-Hand Pose Estimation via Distributed Tactile Sensing

TL;DR

TacGraph presents a factor-graph based framework for simultaneous in-hand pose and extrinsic contact estimation using distributed tactile sensing. By learning object-agnostic tactile models and enforcing geometric, non-penetration, contact-kinematics, and force-balance constraints, it achieves robust joint inference via MAP optimization with iSAM2 and multi-particle initialization. The approach demonstrates superior performance over baselines, particularly in tactile-only settings, and enables precise tactile-only peg insertion, highlighting the value of integrating tactile geometry and contact forces in prehensile manipulation. The work advances practical in-hand perception by coupling tactile feedback with physical laws to reduce ambiguity in pose and contact estimation, with potential extensions to online control and multi-contact scenarios.

Abstract

Prehensile autonomous manipulation, such as peg insertion, tool use, or assembly, require precise in-hand understanding of the object pose and the extrinsic contacts made during interactions. Providing accurate estimation of pose and contacts is challenging. Tactile sensors can provide local geometry at the sensor and force information about the grasp, but the locality of sensing means resolving poses and contacts from tactile alone is often an ill-posed problem, as multiple configurations can be consistent with the observations. Adding visual feedback can help resolve ambiguities, but can suffer from noise and occlusions. In this work, we propose a method that pairs local observations from sensing with the physical constraints of contact. We propose a set of factors that ensure local consistency with tactile observations as well as enforcing physical plausibility, namely, that the estimated pose and contacts must respect the kinematic and force constraints of quasi-static rigid body interactions. We formalize our problem as a factor graph, allowing for efficient estimation. In our experiments, we demonstrate that our method outperforms existing geometric and contact-informed estimation pipelines, especially when only tactile information is available. Video results can be found at https://tacgraph.github.io/.

Figures (5)

• Figure 1: We propose TacGraph, an estimator that exploits geometric consistency, force balance, non-penetration, and contact kinematics to jointly estimate the object pose and extrinsic contacts.
• Figure 2: Overview of our proposed methodology. First, we propose a set of tactile models which process raw distributed tactile observations into geometric and force feedback terms. Second, these terms are utilized, along with known geometries and optionally with visual feedback, in a factor-graph based estimator, TacGraph, which estimates the object pose and extrinsic contacts. Observations, variables, and factors that are fixed (non time-varying) are double circled. Factors only active when contact is detected are connected with dashed lines.
• Figure 3: Our experimental setup. On the left ATI Gamma is an example object fixture. On the right ATI Gamma is the sensorized press surface we use for data collection/experiments.
• Figure 4: Train/Test objects used in our experiments.
• Figure 5: We show comparison of predicted and ground truth object pose, and predicted and ground truth extrinsic contact. (a-b) final qualitative TacGraph estimates for two different objects. (c) progression of particle weights within TacGraph for a tactile-only inference. Two highlighted particles indicate how initial orientations can be filtered based on the contacts made to correctly select particle solutions. Figure best viewed in color.