Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Training

Vedant Dave, Fotios Lygerakis, Elmar Rueckert

TL;DR

This work addresses robust visuo-tactile perception for manipulation by introducing MViTac, a self-supervised multimodal contrastive framework that learns intra- and inter-modal representations using dual encoders and momentum counterparts. The method optimizes multiple InfoNCE losses, combining them into $L_{mm} = L_{vv} + L_{tt} + \lambda_{inter}(L_{vt} + L_{tv})$ to align both modality-specific and cross-modal features. Evaluations on material property identification (Touch-and-Go) and grasping prediction (Calandra) show that visual-tactile fusion yields clear gains over single-modality and prior SSL approaches, with MViTac often surpassing self-supervised baselines and narrowing the gap to supervised methods in data-constrained regimes. The results demonstrate the potential of scalable, annotation-free multimodal learning for robust robotic perception and control, while highlighting the need for real-world robot validation and broader task coverage in future work.

Abstract

The rapidly evolving field of robotics necessitates methods that can facilitate the fusion of multiple modalities. Specifically, when it comes to interacting with tangible objects, effectively combining visual and tactile sensory data is key to understanding and navigating the complex dynamics of the physical world, enabling a more nuanced and adaptable response to changing environments. Nevertheless, much of the earlier work in merging these two sensory modalities has relied on supervised methods utilizing datasets labeled by humans.This paper introduces MViTac, a novel methodology that leverages contrastive learning to integrate vision and touch sensations in a self-supervised fashion. By availing both sensory inputs, MViTac leverages intra and inter-modality losses for learning representations, resulting in enhanced material property classification and more adept grasping prediction. Through a series of experiments, we showcase the effectiveness of our method and its superiority over existing state-of-the-art self-supervised and supervised techniques. In evaluating our methodology, we focus on two distinct tasks: material classification and grasping success prediction. Our results indicate that MViTac facilitates the development of improved modality encoders, yielding more robust representations as evidenced by linear probing assessments.

Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Training

TL;DR

This work addresses robust visuo-tactile perception for manipulation by introducing MViTac, a self-supervised multimodal contrastive framework that learns intra- and inter-modal representations using dual encoders and momentum counterparts. The method optimizes multiple InfoNCE losses, combining them into to align both modality-specific and cross-modal features. Evaluations on material property identification (Touch-and-Go) and grasping prediction (Calandra) show that visual-tactile fusion yields clear gains over single-modality and prior SSL approaches, with MViTac often surpassing self-supervised baselines and narrowing the gap to supervised methods in data-constrained regimes. The results demonstrate the potential of scalable, annotation-free multimodal learning for robust robotic perception and control, while highlighting the need for real-world robot validation and broader task coverage in future work.

Abstract

The rapidly evolving field of robotics necessitates methods that can facilitate the fusion of multiple modalities. Specifically, when it comes to interacting with tangible objects, effectively combining visual and tactile sensory data is key to understanding and navigating the complex dynamics of the physical world, enabling a more nuanced and adaptable response to changing environments. Nevertheless, much of the earlier work in merging these two sensory modalities has relied on supervised methods utilizing datasets labeled by humans.This paper introduces MViTac, a novel methodology that leverages contrastive learning to integrate vision and touch sensations in a self-supervised fashion. By availing both sensory inputs, MViTac leverages intra and inter-modality losses for learning representations, resulting in enhanced material property classification and more adept grasping prediction. Through a series of experiments, we showcase the effectiveness of our method and its superiority over existing state-of-the-art self-supervised and supervised techniques. In evaluating our methodology, we focus on two distinct tasks: material classification and grasping success prediction. Our results indicate that MViTac facilitates the development of improved modality encoders, yielding more robust representations as evidenced by linear probing assessments.
Paper Structure (26 sections, 8 equations, 1 figure, 2 tables)

This paper contains 26 sections, 8 equations, 1 figure, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Tactile Sensing in Robotics
  4. Vision and Touch in Robotic Manipulation
  5. Visual-Tactile Joint Representation learning
  6. MULTIMODAL SELF-SUPERVISED LEARNING
  7. Problem Statement
  8. Multimodal Learning
  9. Architecture
  10. Intra-modal learning
  11. Inter-modal learning
  12. Image-to-Tactile representation learning
  13. Tactile-to-Image representation learning
  14. Combined Loss
  15. EXPERIMENTS
  16. ...and 11 more sections

Figures (1)

  • Figure 1: Left: Architecture of MViTaC that consists of two training mechanisms, the inter-modal (green and pink) and intra-modal contrastive learning (blue and cyan), which collectively contribute to the self-supervised learning loss. Right: The SSL pre-trained model is subsequently evaluated on downstream tasks via linear probing. The task we consider are material property classification (top) and grasp success prediction (bottom).