Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BiKC+: Bimanual Hierarchical Imitation with Keypose-Conditioned Coordination-Aware Consistency Policies

Hang Xu, Yizhou Chen, Dongjie Yu, Yi Ren, Jia PanI

TL;DR

BiKC+ tackles the challenge of robust, efficient bimanual multi-stage manipulation by introducing a hierarchical imitation-learning framework with a keypose-conditioned coordination-aware consistency policy. It combines a high-level keypose predictor (a CM) with a low-level consistency-model-based trajectory generator to provide one-step inference and handle distributional multimodality. A three-stage bimanual keypose identification pipeline, including VLM-assisted coordination-mode detection and coordination-driven merging, ensures synchronization during coordination while preserving arm autonomy otherwise. Empirical results in simulation and on real hardware demonstrate improved task success rates, faster inference, and demonstrated multi-modality handling across rigid and deformable objects. The approach offers a scalable pathway toward reliable, efficient, and flexible bimanual manipulation in industrial-like settings.

Abstract

Robots are essential in industrial manufacturing due to their reliability and efficiency. They excel in performing simple and repetitive unimanual tasks but still face challenges with bimanual manipulation. This difficulty arises from the complexities of coordinating dual arms and handling multi-stage processes. Recent integration of generative models into imitation learning (IL) has made progress in tackling specific challenges. However, few approaches explicitly consider the multi-stage nature of bimanual tasks while also emphasizing the importance of inference speed. In multi-stage tasks, failures or delays at any stage can cascade over time, impacting the success and efficiency of subsequent sub-stages and ultimately hindering overall task performance. In this paper, we propose a novel keypose-conditioned coordination-aware consistency policy tailored for bimanual manipulation. Our framework instantiates hierarchical imitation learning with a high-level keypose predictor and a low-level trajectory generator. The predicted keyposes serve as sub-goals for trajectory generation, indicating targets for individual sub-stages. The trajectory generator is formulated as a consistency model, generating action sequences based on historical observations and predicted keyposes in a single inference step. In particular, we devise an innovative approach for identifying bimanual keyposes, considering both robot-centric action features and task-centric operation styles. Simulation and real-world experiments illustrate that our approach significantly outperforms baseline methods in terms of success rates and operational efficiency. Implementation codes can be found at https://github.com/JoanaHXU/BiKC-plus.

BiKC+: Bimanual Hierarchical Imitation with Keypose-Conditioned Coordination-Aware Consistency Policies

TL;DR

BiKC+ tackles the challenge of robust, efficient bimanual multi-stage manipulation by introducing a hierarchical imitation-learning framework with a keypose-conditioned coordination-aware consistency policy. It combines a high-level keypose predictor (a CM) with a low-level consistency-model-based trajectory generator to provide one-step inference and handle distributional multimodality. A three-stage bimanual keypose identification pipeline, including VLM-assisted coordination-mode detection and coordination-driven merging, ensures synchronization during coordination while preserving arm autonomy otherwise. Empirical results in simulation and on real hardware demonstrate improved task success rates, faster inference, and demonstrated multi-modality handling across rigid and deformable objects. The approach offers a scalable pathway toward reliable, efficient, and flexible bimanual manipulation in industrial-like settings.

Abstract

Robots are essential in industrial manufacturing due to their reliability and efficiency. They excel in performing simple and repetitive unimanual tasks but still face challenges with bimanual manipulation. This difficulty arises from the complexities of coordinating dual arms and handling multi-stage processes. Recent integration of generative models into imitation learning (IL) has made progress in tackling specific challenges. However, few approaches explicitly consider the multi-stage nature of bimanual tasks while also emphasizing the importance of inference speed. In multi-stage tasks, failures or delays at any stage can cascade over time, impacting the success and efficiency of subsequent sub-stages and ultimately hindering overall task performance. In this paper, we propose a novel keypose-conditioned coordination-aware consistency policy tailored for bimanual manipulation. Our framework instantiates hierarchical imitation learning with a high-level keypose predictor and a low-level trajectory generator. The predicted keyposes serve as sub-goals for trajectory generation, indicating targets for individual sub-stages. The trajectory generator is formulated as a consistency model, generating action sequences based on historical observations and predicted keyposes in a single inference step. In particular, we devise an innovative approach for identifying bimanual keyposes, considering both robot-centric action features and task-centric operation styles. Simulation and real-world experiments illustrate that our approach significantly outperforms baseline methods in terms of success rates and operational efficiency. Implementation codes can be found at https://github.com/JoanaHXU/BiKC-plus.
Paper Structure (39 sections, 10 equations, 14 figures, 7 tables)

This paper contains 39 sections, 10 equations, 14 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Bimanual Manipulation with Imitation Learning
  4. Hierarchical Imitation for Multi-stage Tasks
  5. Preliminaries: Consistency Model
  6. Modeling Distribution with Consistency Models
  7. Training and Inference of Consistency Models
  8. Consistency Models as Visuomotor Policies
  9. Bimanual Keypose-Conditioned Coordination-Aware Consistency Policy
  10. Framework Overview and Notation Description
  11. Bimanual Keypose Identification
  12. Stage-1. Unimanual Keypose Extraction
  13. Stage-2. Coordination Mode Identification
  14. Stage-3. Coordination-Driven Merging
  15. Bimanual Keypose Predictor
  16. ...and 24 more sections

Figures (14)

  • Figure 1: Framework of bimanual keypose-conditioned consistency policy.
  • Figure 2: Three-stage pipeline for constructing bimanual keypose dataset.
  • Figure 3: Heuristics rules for identifying unimanual keypose.
  • Figure 4: Illustration of dual-arm operation categorization and coordination-mode identification principles
  • Figure 5: Keyposes and trajectories sampled from demonstrations, for training keypose predictor and trajectory generator.
  • ...and 9 more figures