Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bimanual Dexterity for Complex Tasks

Kenneth Shaw, Yulong Li, Jiahui Yang, Mohan Kumar Srirama, Ray Liu, Haoyu Xiong, Russell Mendonca, Deepak Pathak

TL;DR

This work introduces Bidex, an extremely dexterous, low-cost, low-latency and portable bimanual dexterous teleoperation system which relies on motion capture gloves and teacher arms and finds Bidex to produce better quality data for more complex tasks at a faster rate.

Abstract

To train generalist robot policies, machine learning methods often require a substantial amount of expert human teleoperation data. An ideal robot for humans collecting data is one that closely mimics them: bimanual arms and dexterous hands. However, creating such a bimanual teleoperation system with over 50 DoF is a significant challenge. To address this, we introduce Bidex, an extremely dexterous, low-cost, low-latency and portable bimanual dexterous teleoperation system which relies on motion capture gloves and teacher arms. We compare Bidex to a Vision Pro teleoperation system and a SteamVR system and find Bidex to produce better quality data for more complex tasks at a faster rate. Additionally, we show Bidex operating a mobile bimanual robot for in the wild tasks. The robot hands (5k USD) and teleoperation system (7k USD) is readily reproducible and can be used on many robot arms including two xArms (16k USD). Website at https://bidex-teleop.github.io/

Bimanual Dexterity for Complex Tasks

TL;DR

This work introduces Bidex, an extremely dexterous, low-cost, low-latency and portable bimanual dexterous teleoperation system which relies on motion capture gloves and teacher arms and finds Bidex to produce better quality data for more complex tasks at a faster rate.

Abstract

To train generalist robot policies, machine learning methods often require a substantial amount of expert human teleoperation data. An ideal robot for humans collecting data is one that closely mimics them: bimanual arms and dexterous hands. However, creating such a bimanual teleoperation system with over 50 DoF is a significant challenge. To address this, we introduce Bidex, an extremely dexterous, low-cost, low-latency and portable bimanual dexterous teleoperation system which relies on motion capture gloves and teacher arms. We compare Bidex to a Vision Pro teleoperation system and a SteamVR system and find Bidex to produce better quality data for more complex tasks at a faster rate. Additionally, we show Bidex operating a mobile bimanual robot for in the wild tasks. The robot hands (5k USD) and teleoperation system (7k USD) is readily reproducible and can be used on many robot arms including two xArms (16k USD). Website at https://bidex-teleop.github.io/

Paper Structure

This paper contains 23 sections, 9 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Bimanual Robot Hand and Arm System
  4. Multi-fingered Hand Tracking
  5. Arm Tracking
  6. Robot Configurations
  7. Experiment Setup
  8. Baseline Teleoperation Approaches
  9. Choice of Dexterous End-Effectors
  10. Task Descriptions
  11. Results
  12. Bimanual Dexterous Teleoperation Results
  13. Training Dexterous Visuomotor Policies with BiDex
  14. Extreme Dexterity using LEAP Hand V2
  15. Discussion and Limitations
  16. ...and 8 more sections

Figures (9)

  • Figure 1: Bimanual Dexterity: BiDex can effortlessly teleoperate various complex tasks including pouring, scooping, hammering, chopstick picking, hanger picking, picking up basket, drilling, plate pickup and pot picking to train high-quality behavior cloning policies with over 50 degrees of freedom. Our teleoperation system and two LEAP hands shaw2023leap costs around $12k in total and is readily reproducible by academic labs.
  • Figure 2: Mobile bimanual teleoperation system Left: An operator strapped into BiDex. Right: Our bimanual robot setup including two xArm robot arms, two LEAP Hands shaw2023leap and three cameras on an AgileX base.
  • Figure 3: All Tasks: Teleoperation of the mobile robot systems with BiDex. Top: Picking up trash from a table and discarding it into a bin. Bottom: Grasping a chair and moving it to align with a table.
  • Figure 4: Clearing the Dishes: In this task, we use BiDex to perform a long horizon task to place bowls and spoons into a drying rack and lift the drying rack away from the table.
  • Figure 5: Behavior Cloning Policy Architecture
  • ...and 4 more figures