Tactile Functasets: Neural Implicit Representations of Tactile Datasets

TL;DR

This work proposes neural implicit functions trained to reconstruct the tactile dataset, producing compact representations that capture the underlying structure of the sensory inputs that enable probabilistically interpretable inference, and facilitate generalization across different sensors.

Abstract

Modern incarnations of tactile sensors produce high-dimensional raw sensory feedback such as images, making it challenging to efficiently store, process, and generalize across sensors. To address these concerns, we introduce a novel implicit function representation for tactile sensor feedback. Rather than directly using raw tactile images, we propose neural implicit functions trained to reconstruct the tactile dataset, producing compact representations that capture the underlying structure of the sensory inputs. These representations offer several advantages over their raw counterparts: they are compact, enable probabilistically interpretable inference, and facilitate generalization across different sensors. We demonstrate the efficacy of this representation on the downstream task of in-hand object pose estimation, achieving improved performance over image-based methods while simplifying downstream models. We release code, demos and datasets at https://www.mmintlab.com/tactile-functasets.

Figures (5)

• Figure 1: Modern tactile sensors are capable of streaming high-resolution feedback from the contact between the robot and objects. These high-resolution data streams pose significant challenges for processing and storage. We ask whether there is a more compact and efficient method to store data and perform inference for tactile data? This question is inspired by the observation that tactile signals are lower in information density when compared to natural image.
• Figure 2: The architecture of our tactile functaset framework. Orange layers represent the trunk model, with blue layers being the modulations mapped from red latent vectors. Each red latent vector is stored as one tactile functa point for a specific data point. The entire model takes pixel locations of tactile images as input and output corresponding pixel values.
• Figure 3: Comparison of three key stages in the image reconstruction process: the initial meta-learned state, the reconstructed image through gradient descent steps, and the desired outcome image.
• Figure 4: Visualization of VAE and T3 reconstructed images versus their target images.
• Figure 5: Example distributions for pose estimation, illustrating uncertainty quantification with the functaset representation. The distributions are most dense over the ground truth which indicates the fidelity of the model.