Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulation

Zhengnan Sun, Zhaotai Shi, Jiayin Chen, Qingtao Liu, Yu Cui, Qi Ye, Jiming Chen

TL;DR

This work tackles the difficulty of learning bimanual dexterous manipulation by introducing VTAO-BiManip, which combines visual-tactile-action pretraining with object pose/size understanding and a two-stage curriculum RL framework. A custom VTAO capture system collects synchronized ego-centric vision, tactile signals, action, and object state, feeding a masked MAE-style transformer that learns cross-modal representations and future action predictions. The downstream policy uses frozen pre-trained features within a two-stage curriculum (fixed bottle then released bottle) to achieve cooperative dual-hand manipulation, demonstrated on a bottle-cap unscrewing task in simulation and real world. Results show that incorporating action prediction and object understanding yields significant improvements over single-modality baselines, validating multimodal pretraining as a viable path toward robust, human-like bimanual dexterity.

Abstract

Bimanual dexterous manipulation remains significant challenges in robotics due to the high DoFs of each hand and their coordination. Existing single-hand manipulation techniques often leverage human demonstrations to guide RL methods but fail to generalize to complex bimanual tasks involving multiple sub-skills. In this paper, we introduce VTAO-BiManip, a novel framework that combines visual-tactile-action pretraining with object understanding to facilitate curriculum RL to enable human-like bimanual manipulation. We improve prior learning by incorporating hand motion data, providing more effective guidance for dual-hand coordination than binary tactile feedback. Our pretraining model predicts future actions as well as object pose and size using masked multimodal inputs, facilitating cross-modal regularization. To address the multi-skill learning challenge, we introduce a two-stage curriculum RL approach to stabilize training. We evaluate our method on a bottle-cap unscrewing task, demonstrating its effectiveness in both simulated and real-world environments. Our approach achieves a success rate that surpasses existing visual-tactile pretraining methods by over 20%.

VTAO-BiManip: Masked Visual-Tactile-Action Pre-training with Object Understanding for Bimanual Dexterous Manipulation

TL;DR

This work tackles the difficulty of learning bimanual dexterous manipulation by introducing VTAO-BiManip, which combines visual-tactile-action pretraining with object pose/size understanding and a two-stage curriculum RL framework. A custom VTAO capture system collects synchronized ego-centric vision, tactile signals, action, and object state, feeding a masked MAE-style transformer that learns cross-modal representations and future action predictions. The downstream policy uses frozen pre-trained features within a two-stage curriculum (fixed bottle then released bottle) to achieve cooperative dual-hand manipulation, demonstrated on a bottle-cap unscrewing task in simulation and real world. Results show that incorporating action prediction and object understanding yields significant improvements over single-modality baselines, validating multimodal pretraining as a viable path toward robust, human-like bimanual dexterity.

Abstract

Bimanual dexterous manipulation remains significant challenges in robotics due to the high DoFs of each hand and their coordination. Existing single-hand manipulation techniques often leverage human demonstrations to guide RL methods but fail to generalize to complex bimanual tasks involving multiple sub-skills. In this paper, we introduce VTAO-BiManip, a novel framework that combines visual-tactile-action pretraining with object understanding to facilitate curriculum RL to enable human-like bimanual manipulation. We improve prior learning by incorporating hand motion data, providing more effective guidance for dual-hand coordination than binary tactile feedback. Our pretraining model predicts future actions as well as object pose and size using masked multimodal inputs, facilitating cross-modal regularization. To address the multi-skill learning challenge, we introduce a two-stage curriculum RL approach to stabilize training. We evaluate our method on a bottle-cap unscrewing task, demonstrating its effectiveness in both simulated and real-world environments. Our approach achieves a success rate that surpasses existing visual-tactile pretraining methods by over 20%.
Paper Structure (35 sections, 2 equations, 7 figures, 2 tables)

This paper contains 35 sections, 2 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multimodal Pretraining for Robotics
  4. Robotic Bimanual Manipulation
  5. Method
  6. VTAO Capture System for Human Bi-Manipulation
  7. System Overview
  8. Data Alignment and Unified Representation
  9. Retarget Human Manipulation Trajectories
  10. VTAO Dataset
  11. Masked VTAO Transformer for Pre-Training
  12. VTAO Encoder $E_\theta (V, T, A) \rightarrow (h_{\text{fs}}, h_v, h_c, h_a, h_o)$
  13. VTAO Decoder $D_\theta (h_v, h_c, h_a, h_o, m) \rightarrow (\widehat{V}, \widehat{T}, [\widehat{A}, \widehat{A_{+1}}, ..., \widehat{A_{+p}}], \widehat{O})$
  14. Reconstruct Image $\widehat{V}$ and Tactile Signals $\widehat{T}$
  15. Reconstruct Current Actions and Predict Future Actions $[\widehat{A}, \widehat{A_{+1}}, ..., \widehat{A_{+p}}]$
  16. ...and 20 more sections

Figures (7)

  • Figure 1: Left: examples of our bottle-cap turning dataset. Middle: simulation results of the bottle-cap task with Shadow Hand shadowhand. Right: the real-world deployment of the bottle-cap task with Leap Hand shaw2023leaphand.
  • Figure 2: Our VTAO capture system for dual-hand human manipulation.
  • Figure 3: Overview of our VTAO-BiManip pre-train framework and its downstream bimanual manipulation with sub-skill and collaborative curriculum reinforcement learning.
  • Figure 5: Test success rates (%) on seen and unseen bottles across different experimental settings.
  • Figure 6: Our manipulation platform.
  • ...and 2 more figures