Learning Bimanual Manipulation via Action Chunking and Inter-Arm Coordination with Transformers

TL;DR

This paper tackles the challenge of learning coordinated bimanual manipulation for dual-arm robots in human environments. It introduces the Inter-Arm Coordinated Transformer Encoder (IACE), with per-arm encoders and a Transformer-based decoder, and evaluates single versus split decoder configurations along with a CVAE for demonstration variability. Real-world experiments on eight bimanual tasks using the ALOHA platform show improved performance over baselines, particularly in asynchronous tasks, and highlight the critical role of IACE through ablations. The work advances data-efficient, synchronization-aware imitation learning for dual-arm robotics and suggests directions toward multimodal inputs and single-arm extensions, broadening practical applicability.

Abstract

Robots that can operate autonomously in a human living environment are necessary to have the ability to handle various tasks flexibly. One crucial element is coordinated bimanual movements that enable functions that are difficult to perform with one hand alone. In recent years, learning-based models that focus on the possibilities of bimanual movements have been proposed. However, the high degree of freedom of the robot makes it challenging to reason about control, and the left and right robot arms need to adjust their actions depending on the situation, making it difficult to realize more dexterous tasks. To address the issue, we focus on coordination and efficiency between both arms, particularly for synchronized actions. Therefore, we propose a novel imitation learning architecture that predicts cooperative actions. We differentiate the architecture for both arms and add an intermediate encoder layer, Inter-Arm Coordinated transformer Encoder (IACE), that facilitates synchronization and temporal alignment to ensure smooth and coordinated actions. To verify the effectiveness of our architectures, we perform distinctive bimanual tasks. The experimental results showed that our model demonstrated a high success rate for comparison and suggested a suitable architecture for the policy learning of bimanual manipulation.

Figures (5)

• Figure 1: Details of Our proposed architectures. (a) Single Decoder to process information from the multiple encoders, simplifying the decoding process. (b) Split Decoders get the input from each encoder, process, and output each action. It allows each decoder to handle different parts of the data, potentially improving performance on complex bimanual tasks (c) CVAE Encoder is crucial for reconstructing inputs or generating new samples to eliminate the variation of human teleoperation data.
• Figure 2: Our bimanual robot experimental setting. It has been designed for real-world applications. We have modified the ALOHA setup for these experiments, including adjusting the camera height and adding a protective tarp around the setup.
• Figure 3: The snapshot of the motion generation in Fold Towel (Asynchronous task). a), b) & c) is Fold motion, d) is Pick motion, f) is Place motion.
• Figure 4: The snapshot of the motion generation in Lift and Place Towel (Synchronous task). a), b) & c) are Lift motion, d), e) & f) are Place motion.
• Figure 5: Architectures used for verification. (Top) Our proposed models' architecture includes the architectures with split decoders or single decoder and inter-arm coordinated module. (Bottom) These models do not have the inter-arm coordinated transformer encoder to analyse and compare the model structure with the existing base model.