MOSAIC: Learning Unified Multi-Sensory Object Property Representations for Robot Learning via Interactive Perception

TL;DR

This work addresses the challenge of robust object property understanding for robots across vision, audio, and haptics by introducing MOSAIC, a framework that distills language grounding from a pre-trained CLIP text encoder into unified multimodal representations. Through a two-stage process, MOSAIC freezes vision, audio, and text encoders while training a haptic encoder, self-attention fusion, and an MLP to align multimodal embeddings with CLIP text space via a symmetric contrastive loss, resulting in embeddings of dimension $D_{\mathcal{Z}}=512$. Evaluated on 100 objects across 20 categories and 10 exploratory behaviors, MOSAIC demonstrates competitive object category recognition with a linear probe and strong zero-shot fetch-object performance, driven by interactive perception that leverages all three modalities. The findings highlight the potential of grounding robotic perception in foundation-model language representations, enabling data-efficient, cross-modal understanding with practical impact on manipulation and retrieval tasks.

Abstract

A holistic understanding of object properties across diverse sensory modalities (e.g., visual, audio, and haptic) is essential for tasks ranging from object categorization to complex manipulation. Drawing inspiration from cognitive science studies that emphasize the significance of multi-sensory integration in human perception, we introduce MOSAIC (Multimodal Object property learning with Self-Attention and Interactive Comprehension), a novel framework designed to facilitate the learning of unified multi-sensory object property representations. While it is undeniable that visual information plays a prominent role, we acknowledge that many fundamental object properties extend beyond the visual domain to encompass attributes like texture, mass distribution, or sounds, which significantly influence how we interact with objects. In MOSAIC, we leverage this profound insight by distilling knowledge from multimodal foundation models and aligning these representations not only across vision but also haptic and auditory sensory modalities. Through extensive experiments on a dataset where a humanoid robot interacts with 100 objects across 10 exploratory behaviors, we demonstrate the versatility of MOSAIC in two task families: object categorization and object-fetching tasks. Our results underscore the efficacy of MOSAIC's unified representations, showing competitive performance in category recognition through a simple linear probe setup and excelling in the fetch object task under zero-shot transfer conditions. This work pioneers the application of sensory grounding in foundation models for robotics, promising a significant leap in multi-sensory perception capabilities for autonomous systems. We have released the code, datasets, and additional results: https://github.com/gtatiya/MOSAIC.

Figures (4)

• Figure 1: Overview of the MOSAIC Framework: Initially, the robot collects sensory data through object exploration, which is then used to train models for distilling unified multimodal representations guided by a pre-trained text encoder. These acquired representations are subsequently applied to a variety of downstream tasks.
• Figure 2: (A) 100 objects, grouped in 20 object categories. (B) The interactive behaviors that the robot performed on the objects.
• Figure 3: 2D unified representations derived from autoencoder trained on Push behavior's data: (A) Object categories, (B) Material, (C) Deformability, and (D) Hardness properties.
• Figure : Training MOSAIC Framework