Active Tactile Exploration for Rigid Body Pose and Shape Estimation

TL;DR

The paper addresses tactile-only learning of the pose $x_T$ and geometry $\theta$ of unseen convex rigid objects, even as contacts perturb the object. It combines a violation-implicit loss that enforces physical constraints with an Expected Information Gain (EIG)-driven exploration strategy to select informative contact actions, using a friction-cone $QP$ and differentiable dynamics for gradient-based learning. The contributions are (i) a loss enabling simultaneous estimation of pose and geometry from tactile data and (ii) an active exploration scheme that accelerates learning, achieving cuboid and convex-polytope reconstruction with under $<10\mathrm{s}$ of tactile data in both simulation and real robots. This work demonstrates data-efficient, tactile-driven model-building for unseen objects and sets the stage for multi-modal extensions with vision.

Abstract

General robot manipulation requires the handling of previously unseen objects. Learning a physically accurate model at test time can provide significant benefits in data efficiency, predictability, and reuse between tasks. Tactile sensing can compliment vision with its robustness to occlusion, but its temporal sparsity necessitates careful online exploration to maintain data efficiency. Direct contact can also cause an unrestrained object to move, requiring both shape and location estimation. In this work, we propose a learning and exploration framework that uses only tactile data to simultaneously determine the shape and location of rigid objects with minimal robot motion. We build on recent advances in contact-rich system identification to formulate a loss function that penalizes physical constraint violation without introducing the numerical stiffness inherent in rigid-body contact. Optimizing this loss, we can learn cuboid and convex polyhedral geometries with less than 10s of randomly collected data after first contact. Our exploration scheme seeks to maximize Expected Information Gain and results in significantly faster learning in both simulated and real-robot experiments. More information can be found at https://dairlab.github.io/activetactile

Figures (6)

• Figure 1: Only tactile data is used to find the pose and geometry of an arbitrary convex object (e.g., parameterized as a 20-vertex polytope as shown here). Actions are chosen to maximize Expected Information Gain, related to the info we expect to gain from contact divided by the info previously observed. As parts of the ground-truth object (e.g. a cuboid, shown in gray) are contacted, the corresponding areas of the learned estimate will be assigned a high observed info (blue $\rightarrow$ red).
• Figure 2: Block diagram of the interaction protocol. An estimate of object pose and geometry emerges from tactile and proprioceptive data (top left) by minimizing a violation-implicit loss. In this example with a cuboid parameterization, the top, left, and bottom faces can be directly inferred from data while the right face is unobserved. The expected information gain (EIG) of the next action is optimized using a Gaussian cross-entropy method: robot trajectories are sampled, simulated forward to time horizon $H$, and scored. Pictured here, $\mathcal{D}_{obs}$ includes contact with left and top, leading to high observed info $\mathcal{I}$ (red), so the upper sample contacting these same sides yields low EIG. The lower sample contacts the right side with low $\mathcal{I}$ (blue), maximizing EIG.
• Figure 3: Hardware setup: a Trifinger wuthrich_trifinger_2021 modified to have 2 fingers directly opposing each other, each instrumented with a Densetact do_densetact_2023 sensor. Densetact images (green) can be used to locate the center-of-pressure (yellow), which in turn can be projected onto the nominal model to compute the measured normal vector $\hat{n}_{m,t}$ (pink). A single RealSense running FoundationPose (red) is used for evaluation only to calculate bCH.
• Figure 4: bCH between the ground truth and estimated shapes for cuboid (top) and convex polytope (bottom) parameterizations in simulation. Shaded region is the 95% confidence interval for the mean of a normal distribution ($N=10$). One outlier trial for random (marked with $\times$) does not factor into pictured 95% CI.
• Figure 5: bCH between the ground truth and estimated shapes on the real robot. Shaded region is the 95% confidence interval for the mean of a normal distribution ($N=6$).