Tactile Neural De-rendering

TL;DR

This work introduces Tactile Neural De-rendering, a novel approach that leverages a generative model to reconstruct a local 3D representation of an object based solely on its tactile signature, providing a robust framework for tactile-based perception in robotics.

Abstract

Tactile sensing has proven to be an invaluable tool for enhancing robotic perception, particularly in scenarios where visual data is limited or unavailable. However, traditional methods for pose estimation using tactile data often rely on intricate modeling of sensor mechanics or estimation of contact patches, which can be cumbersome and inherently deterministic. In this work, we introduce Tactile Neural De-rendering, a novel approach that leverages a generative model to reconstruct a local 3D representation of an object based solely on its tactile signature. By rendering the object as though perceived by a virtual camera embedded at the fingertip, our method provides a more intuitive and flexible representation of the tactile data. This 3D reconstruction not only facilitates precise pose estimation but also allows for the quantification of uncertainty, providing a robust framework for tactile-based perception in robotics.

Figures (5)

• Figure 1: Method overview: A tactile signature is measured from the contact between a visio-tactile sensor and an object, and then De-rendered with our method to produce local 3D in-contact object geometry. The resulting geometry can be used for downstream applications. Here, we focus on pose estimation.
• Figure 2: A summary of Tactile Neural De-rendering training. (a) A tactile signature is obtained from the contact between a visio-tactile sensor and an object, with known poses. (b) The four key steps to simulate the in-contact object's depth. (c) Tactile Neural De-rendering training process, we train a diffusion model conditioned on the tactile imprint to generate the in-contact object's depth. For training, we make use of the simulated in-contact object's depth computed in (b) as $X_0$.
• Figure 3: Designed objects to train and test our method. Among them are different sizes of cylinders, triangles, trapezoids, bullets, spheres, and crosses. The objects used for training have a light-blue background, and the ones used for testing unseen objects have a light-red background.
• Figure 4: Depth generations examples for the model with data configuration of real-data + synthetic 24k. Each row represents a single object, and each column displays the tactile imprint, the simulated depth used to test the model, and the generated in-contact object depth image, respectively. Each depth image is colored using the JET colormap.
• Figure 5: Kernel density estimate (KDE) plot of the pose estimation error using our method, for a specific contact configuration of the Soft Bubble sensor and the Round Cross object. Dots represent the pose estimation error from the baseline as they are deterministic.