ACROSS: A Deformation-Based Cross-Modal Representation for Robotic Tactile Perception

TL;DR

ACROSS tackles the problem of obsolescence of tactile datasets by introducing a deformation-based cross-modal transfer that converts source sensor signals into 3D deformation meshes, maps those deformations to a target sensor, and renders the target output. The approach uses three β-VAE–driven modules and two latent-mapping networks to bridge BioTac and DIGIT modalities, followed by a Taxim-based rendering step to produce DIGIT images from predicted mesh deformations. A large open dataset of over 155K paired BioTac–DIGIT deformation meshes is released to facilitate cross-sensor data reuse and collaboration. Results show strong low-level accuracy for deformation reconstruction (MVB/MVD) and reasonable cross-modal translation, with performance limited by BioTac’s low electrode count and alignment challenges, highlighting both the potential and the current limitations of deformation-space data transfer. The work advances practical data exchange in tactile robotics and points to future expansion across sensors and tasks, with improved rendering and broader modality coverage as key directions.

Abstract

Tactile perception is essential for human interaction with the environment and is becoming increasingly crucial in robotics. Tactile sensors like the BioTac mimic human fingertips and provide detailed interaction data. Despite its utility in applications like slip detection and object identification, this sensor is now deprecated, making many valuable datasets obsolete. However, recreating similar datasets with newer sensor technologies is both tedious and time-consuming. Therefore, adapting these existing datasets for use with new setups and modalities is crucial. In response, we introduce ACROSS, a novel framework for translating data between tactile sensors by exploiting sensor deformation information. We demonstrate the approach by translating BioTac signals into the DIGIT sensor. Our framework consists of first converting the input signals into 3D deformation meshes. We then transition from the 3D deformation mesh of one sensor to the mesh of another, and finally convert the generated 3D deformation mesh into the corresponding output space. We demonstrate our approach to the most challenging problem of going from a low-dimensional tactile representation to a high-dimensional one. In particular, we transfer the tactile signals of a BioTac sensor to DIGIT tactile images. Our approach enables the continued use of valuable datasets and data exchange between groups with different setups.

Figures (5)

• Figure 1: An example of the ACROSS framework applied to translate BioTac signals into DIGIT images. Step 1: Convert BioTac input signals to BioTac surface deformation. Step 2: Convert BioTac surface mesh deformation to DIGIT surface mesh deformation. Step 3: Generate DIGIT's output from the surface mesh deformation.
• Figure 2: The nine indenters used to collect the BioTac-DIGIT deformation dataset.
• Figure 3: Transferred BioTac sensor (green) to align it with the DIGIT sensor surface (gold), in order to collect paired 3D deformation meshes.
• Figure 4: Comparison of artifacts in the generated image before (left) and after (right) applying the additional pyramid Gaussian blur.
• Figure 5: Converted samples. First row: Real electrode values. Second row: Ground-truth BioTac mesh deformations. The outer frame represents the "unfolded" BioTac surface. Third row: Converted DIGIT mesh deformations. Fourth row: DIGIT output images. The third and fourth rows were generated using the first row as input.