Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Modality Selection and Skill Segmentation via Cross-Modality Attention

Jiawei Jiang, Kei Ota, Devesh K. Jha, Asako Kanezaki

TL;DR

The paper tackles the curse of dimensionality when incorporating tactile and audio modalities into robotic policies by introducing Cross-Modality Attention (CMA), a transformer-based mechanism that adaptively weights modalities for action generation. It couples CMA with a 1D-conditional diffusion policy and a three-stage workflow (imitation-driven CMA training, unsupervised primitive segmentation, and hierarchical policy learning) to enable long-horizon, contact-rich manipulation. CMA not only improves efficiency but also reveals interpretable, action-specific modality usage through attention patterns, and supports unsupervised segmentation of demonstrations into primitives. The results on FurnitureSim leg-assembly tasks demonstrate improved sample efficiency and provide a viable path toward modality-aware, hierarchical robotics with better generalization in complex manipulation tasks.

Abstract

Incorporating additional sensory modalities such as tactile and audio into foundational robotic models poses significant challenges due to the curse of dimensionality. This work addresses this issue through modality selection. We propose a cross-modality attention (CMA) mechanism to identify and selectively utilize the modalities that are most informative for action generation at each timestep. Furthermore, we extend the application of CMA to segment primitive skills from expert demonstrations and leverage this segmentation to train a hierarchical policy capable of solving long-horizon, contact-rich manipulation tasks.

Modality Selection and Skill Segmentation via Cross-Modality Attention

TL;DR

The paper tackles the curse of dimensionality when incorporating tactile and audio modalities into robotic policies by introducing Cross-Modality Attention (CMA), a transformer-based mechanism that adaptively weights modalities for action generation. It couples CMA with a 1D-conditional diffusion policy and a three-stage workflow (imitation-driven CMA training, unsupervised primitive segmentation, and hierarchical policy learning) to enable long-horizon, contact-rich manipulation. CMA not only improves efficiency but also reveals interpretable, action-specific modality usage through attention patterns, and supports unsupervised segmentation of demonstrations into primitives. The results on FurnitureSim leg-assembly tasks demonstrate improved sample efficiency and provide a viable path toward modality-aware, hierarchical robotics with better generalization in complex manipulation tasks.

Abstract

Incorporating additional sensory modalities such as tactile and audio into foundational robotic models poses significant challenges due to the curse of dimensionality. This work addresses this issue through modality selection. We propose a cross-modality attention (CMA) mechanism to identify and selectively utilize the modalities that are most informative for action generation at each timestep. Furthermore, we extend the application of CMA to segment primitive skills from expert demonstrations and leverage this segmentation to train a hierarchical policy capable of solving long-horizon, contact-rich manipulation tasks.

Paper Structure

This paper contains 12 sections, 1 equation, 5 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Multi-modal Diffusion Policy
  5. Cross-Modality Attention
  6. Encoder
  7. Experiments
  8. Settings
  9. Attention characteristics of different primitive actions
  10. Training Whole Trajectory v.s. Individual Primitives
  11. Clustering action segments
  12. Conclusion

Figures (5)

  • Figure 1: Three stages design. First stage: train CMA using imitation learning. Second stage: Use CMA to cluster and segment primitive actions. Thrid stage: Train a hierarchical policy that select and execute individual primitive actions that only select useful modalities as input.
  • Figure 2: Attention weights obtained through training a multimodal diffusion model.
  • Figure 3: Primitive Actions.
  • Figure 4: Validation Loss.
  • Figure 5: T-SNE graph of primitive actions