ViTaS: Visual Tactile Soft Fusion Contrastive Learning for Visuomotor Learning

TL;DR

ViTaS presents a visuo-tactile fusion framework for visuomotor learning that leverages Soft Fusion Contrastive Learning to align visual and tactile embeddings and a CVAE module to exploit their complementarity. The framework optimizes a joint objective $\mathcal{L} = \lambda\mathcal{L}_{\text{CON}} + \mu\mathcal{L}_{\text{VAE}} + \mathcal{L}_{\text{policy}}$, with $\mathcal{L}_{\text{CON}}$ promoting cross-modal alignment and $\mathcal{L}_{\text{VAE}}$ enforcing cross-modal consistency. It is evaluated on 12 simulated tasks across 5 environments and 3 real-world tasks under RL and IL, showing state-of-the-art performance, particularly in self-occluded and cluttered settings. Real-world experiments demonstrate robustness to occlusion and reduced visual input thanks to tactile cues. The results highlight the importance of preserving both alignment and complementarity in multimodal representations for reliable visuomotor control.

Abstract

Tactile information plays a crucial role in human manipulation tasks and has recently garnered increasing attention in robotic manipulation. However, existing approaches mostly focus on the alignment of visual and tactile features and the integration mechanism tends to be direct concatenation. Consequently, they struggle to effectively cope with occluded scenarios due to neglecting the inherent complementary nature of both modalities and the alignment may not be exploited enough, limiting the potential of their real-world deployment. In this paper, we present ViTaS, a simple yet effective framework that incorporates both visual and tactile information to guide the behavior of an agent. We introduce Soft Fusion Contrastive Learning, an advanced version of conventional contrastive learning method and a CVAE module to utilize the alignment and complementarity within visuo-tactile representations. We demonstrate the effectiveness of our method in 12 simulated and 3 real-world environments, and our experiments show that ViTaS significantly outperforms existing baselines. Project page: https://skyrainwind.github.io/ViTaS/index.html.

Figures (6)

• Figure 1: ViTaS is capable of handling various simulation and real-world manipulation tasks including transparent objects and self-occluded scenarios, by fusing visual and tactile features for effective policy learning.
• Figure 2: Method overview. ViTaS takes vision and touch as inputs, which are then processed through separate CNN encoders. Encoded embeddings are utilized by soft fusion contrastive approach, yielding fused feature representation for policy network. A CVAE-based reconstruction framework is also applied for cross-modal integration.
• Figure 3: Tasks. Our method is evaluated on $12$ simulation tasks and $3$ real-world tasks, with various embodiment types.
• Figure 4: ViTaS with Imitation Learning. The imitation learning paradigm is adopted to further test feature extraction ability of ViTaS in different settings, which we use for both simulation and real world tasks.
• Figure 5: Real-World Robot Setting.