Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation

Andrew Lee, Ian Chuang, Ling-Yuan Chen, Iman Soltani

TL;DR

This paper introduces InterACT: Inter-dependency aware Action Chunking with Hierarchical Attention Transformers, a novel imitation learning framework designed specifically for bimanual manipulation that leverages hierarchical attention mechanisms to effectively capture inter-dependencies between dual-arm joint states and visual inputs.

Abstract

Bimanual manipulation presents unique challenges compared to unimanual tasks due to the complexity of coordinating two robotic arms. In this paper, we introduce InterACT: Inter-dependency aware Action Chunking with Hierarchical Attention Transformers, a novel imitation learning framework designed specifically for bimanual manipulation. InterACT leverages hierarchical attention mechanisms to effectively capture inter-dependencies between dual-arm joint states and visual inputs. The framework comprises a Hierarchical Attention Encoder, which processes multi-modal inputs through segment-wise and cross-segment attention mechanisms, and a Multi-arm Decoder that generates each arm's action predictions in parallel, while sharing information between the arms through synchronization blocks by providing the other arm's intermediate output as context. Our experiments, conducted on various simulated and real-world bimanual manipulation tasks, demonstrate that InterACT outperforms existing methods. Detailed ablation studies further validate the significance of key components, including the impact of CLS tokens, cross-segment encoders, and synchronization blocks on task performance. We provide supplementary materials and videos on our project page.

InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation

TL;DR

This paper introduces InterACT: Inter-dependency aware Action Chunking with Hierarchical Attention Transformers, a novel imitation learning framework designed specifically for bimanual manipulation that leverages hierarchical attention mechanisms to effectively capture inter-dependencies between dual-arm joint states and visual inputs.

Abstract

Bimanual manipulation presents unique challenges compared to unimanual tasks due to the complexity of coordinating two robotic arms. In this paper, we introduce InterACT: Inter-dependency aware Action Chunking with Hierarchical Attention Transformers, a novel imitation learning framework designed specifically for bimanual manipulation. InterACT leverages hierarchical attention mechanisms to effectively capture inter-dependencies between dual-arm joint states and visual inputs. The framework comprises a Hierarchical Attention Encoder, which processes multi-modal inputs through segment-wise and cross-segment attention mechanisms, and a Multi-arm Decoder that generates each arm's action predictions in parallel, while sharing information between the arms through synchronization blocks by providing the other arm's intermediate output as context. Our experiments, conducted on various simulated and real-world bimanual manipulation tasks, demonstrate that InterACT outperforms existing methods. Detailed ablation studies further validate the significance of key components, including the impact of CLS tokens, cross-segment encoders, and synchronization blocks on task performance. We provide supplementary materials and videos on our project page.
Paper Structure (12 sections, 4 figures, 5 tables, 2 algorithms)

This paper contains 12 sections, 4 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Bimanual Manipulation
  4. Hierarchical Attention Mechanisms
  5. InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformer
  6. Hierarchical Attention Encoder
  7. Multi-arm Decoder
  8. Training and Evaluation
  9. Experiments and Results
  10. Results
  11. Ablation Studies
  12. Conclusion and Future Work

Figures (4)

  • Figure 1: Architecture of the InterACT. The Hierarchical Attention Encoder consists of multiple blocks of segment-wise encoders and cross-segment encoder. The output is passed through the Multi-arm Decoder which consists of Arm1 and Arm2 specific decoders that process the input segments independently. The synchronization block allows for information sharing between the two decoders.
  • Figure 2: Attention weights for CLS tokens at the Multi-arm Decoder over time for Peg Insertion (right) and Transfer Cube (right). The red highlighted sections correspond to specific timesteps in executing the task. Spikes in attention weights are observed during coordinated phase.
  • Figure 3: Our Real-robot Setup. We have modified the ALOHA 2 setup for our real-world experiments. Modifications include adjusting the camera height and using a tarp around the setup.
  • Figure :