AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors

TL;DR

This work tackles the lack of standardized cross-sensor tactile understanding by introducing TacQuad, a large aligned multi-modal tactile dataset across four sensors, and AnyTouch, a two-stage framework that learns unified static-dynamic tactile representations. The approach combines pixel-level masked modeling for fine-grained details with semantic-level multi-modal alignment and cross-sensor matching to produce sensor-agnostic features, aided by universal sensor tokens for transfer to unseen sensors. Extensive experiments across nine tactile datasets and a real-world pouring task demonstrate improved cross-sensor transfer, robust static and dynamic perception, and meaningful clustering by object content rather than sensor. The work provides a practical pathway toward standardized, cross-sensor tactile perception and transfer in robotics.

Abstract

Visuo-tactile sensors aim to emulate human tactile perception, enabling robots to precisely understand and manipulate objects. Over time, numerous meticulously designed visuo-tactile sensors have been integrated into robotic systems, aiding in completing various tasks. However, the distinct data characteristics of these low-standardized visuo-tactile sensors hinder the establishment of a powerful tactile perception system. We consider that the key to addressing this issue lies in learning unified multi-sensor representations, thereby integrating the sensors and promoting tactile knowledge transfer between them. To achieve unified representation of this nature, we introduce TacQuad, an aligned multi-modal multi-sensor tactile dataset from four different visuo-tactile sensors, which enables the explicit integration of various sensors. Recognizing that humans perceive the physical environment by acquiring diverse tactile information such as texture and pressure changes, we further propose to learn unified multi-sensor representations from both static and dynamic perspectives. By integrating tactile images and videos, we present AnyTouch, a unified static-dynamic multi-sensor representation learning framework with a multi-level structure, aimed at both enhancing comprehensive perceptual abilities and enabling effective cross-sensor transfer. This multi-level architecture captures pixel-level details from tactile data via masked modeling and enhances perception and transferability by learning semantic-level sensor-agnostic features through multi-modal alignment and cross-sensor matching. We provide a comprehensive analysis of multi-sensor transferability, and validate our method on various datasets and in the real-world pouring task. Experimental results show that our method outperforms existing methods, exhibits outstanding static and dynamic perception capabilities across various sensors.

Figures (9)

• Figure 1: TacQuad: an aligned multi-modal multi-sensor tactile dataset from four visuo-tactile sensors. We select GelSight Minigelsight and DIGITlambeta2020digit from publicly available sensors, DuraGelzhang2024duragel from self-made sensors, and Tac3Dzhang2022tac3d from force field sensors for diversity. There is a noticeable gap between the data from these sensors. We use the four sensors to touch the same position on the same object to obtain aligned data. To maximize aligned data collection, we use two methods to gather subsets with different alignment accuracy. We collect fine-grained spatio-temporal aligned data on a calibration platform, while larger-scale coarse-grained spatial aligned data is acquired through handheld collection.
• Figure 2: Overview of AnyTouch. Our framework integrates static tactile images and dynamic tactile videos, aiming to learn a unified multi-sensor representation suitable for various tasks. Through a multi-level architecture, we employ masked modeling to learn pixel-level tactile details, and use multi-modal aligning and cross-sensor matching to understand semantic-level sensor-agnostic tactile properties. We also use universal sensor tokens to integrate and transfer sensor information.
• Figure 3: Comparison with existing multi-modal aligning methods. Our method not only uses multi-modal data to bridge the gap between sensors, but also explicitly clusters representations of the same position on the same object from different sensors together.
• Figure 4: The impact of components in AnyTouch on the multi-sensor representation space. We use t-SNE to visualize the representations on the unused fine-grained subset of TacQuad, starting with CLIP and sequentially incorporating the modules. Each color represents a single touch, and samples from three sensors that touch the same position are connected by dashed lines.
• Figure 5: Evaluation on the real-world pouring task using GelSight Mini.