Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalities

TL;DR

TLV-CoRe tackles sensor heterogeneity in tactile sensing and limited tri-modal integration with vision and language. It introduces a Sensor-Aware Modulator to unify tactile representations across sensors, tactile-irrelevant decoupled learning to remove sensor artifacts, and a Unified Bridging Adapter to align tactile, language, and vision in a shared space, built on CLIP foundations. The authors propose the RSS evaluation framework to assess Robustness, Synergy, and Stability across modalities and sensor settings. Empirical results show sensor-agnostic tactile representations and improved cross-modal alignment, demonstrating TLV-CoRe as a robust path toward unified multimodal tactile perception.

Abstract

Tactile sensing offers rich and complementary information to vision and language, enabling robots to perceive fine-grained object properties. However, existing tactile sensors lack standardization, leading to redundant features that hinder cross-sensor generalization. Moreover, existing methods fail to fully integrate the intermediate communication among tactile, language, and vision modalities. To address this, we propose TLV-CoRe, a CLIP-based Tactile-Language-Vision Collaborative Representation learning method. TLV-CoRe introduces a Sensor-Aware Modulator to unify tactile features across different sensors and employs tactile-irrelevant decoupled learning to disentangle irrelevant tactile features. Additionally, a Unified Bridging Adapter is introduced to enhance tri-modal interaction within the shared representation space. To fairly evaluate the effectiveness of tactile models, we further propose the RSS evaluation framework, focusing on Robustness, Synergy, and Stability across different methods. Experimental results demonstrate that TLV-CoRe significantly improves sensor-agnostic representation learning and cross-modal alignment, offering a new direction for multimodal tactile representation.

Figures (5)

• Figure 1: Three properties of heterogeneous sensors are identified: (i). Tactile sensors lack full standardization, leading to significant tactile images variation yang2024binding. (ii). Tactile images from the identical touch object can differ inconsistently (e.g., ⓐ and ⓑ are similar, both differing greatly from ⓒ). (iii). Despite different touch objects, tactile images may share a consistent style (e.g., ⓐ and ⓑ resemble ⓓ in a dark tone tinged with red).
• Figure 2: Overview of TLV-CoRe, which consists of modality-specific encoders for tactile (T), visual (V), and language (L) modalities inputs, a Sensor-Aware Modulator (SAM) in the tactile branch to remove sensor-specific biases, and a Unified Bridging Adapter (UBA) that projects features into a shared parameter space for alignment.
• Figure 3: Performance (%) comparison of different methods across various batch sizes.
• Figure 4: Ablation experiments on various components.
• Figure 5: Convergence comparison of TLV-CoRe versus state-of-the-art baselines on the TAG dataset. Each line represents the test loss. TLV-CoRe (green) exhibits faster convergence and more stability compared to TLV-Link (blue) and AnyTouch (red), empirically validating our theoretical analysis in Theorem 3.1 and Proposition 3.3. Notably, while TLV-Link suffers from instability after epoch 8, TLV-CoRe maintains stable performance throughout training.

Theorems & Definitions (14)

• Theorem 3.1: Convergence Rate
• Lemma 3.2: Gradient Variance Reduction
• Proposition 3.3: Optimization Robustness
• Theorem 3.4: Cross-Modal Information Transfer
• Corollary 3.5: Cross-Modal Performance
• Theorem 3.6: Batch-Size Stability
• Proposition 3.7: Representation Enhancement
• proof : Proof of Theorem 3.1
• proof : Proof of Lemma 3.2
• proof : Proof of Proposition 3.3