Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Open TeleDex: A Hardware-Agnostic Teleoperation System for Imitation Learning based Dexterous Manipulation

Xu Chi, Chao Zhang, Yang Su, Lingfeng Dou, Fujia Yang, Jiakuo Zhao, Haoyu Zhou, Xiaoyou Jia, Yong Zhou, Shan An

TL;DR

Open TeleDex tackles the data bottleneck in robot imitation learning by delivering a hardware-agnostic teleoperation framework rooted in ROS2. It introduces a three-tier architecture with a RealEnv hardware abstraction and a generative hand pose retargeting algorithm to enable cross-robot data collection. The system emphasizes high-fidelity, multi-modal data with a robust TimeSync Manager ensuring per-frame synchronization and a structured dataset output for IL. Evaluations on Bottle Peg-in-Hole and Cube Assembly across COTS and in-house hardware demonstrate strong data synchronization (>99%) and improved temporal fidelity with in-house hardware, validating the TripleAny vision. Collectively, Open TeleDex provides a scalable, extensible platform for accelerating both academic and industrial development in dexterous manipulation and imitation learning.

Abstract

Accurate and high-fidelity demonstration data acquisition is a critical bottleneck for deploying robot Imitation Learning (IL) systems, particularly when dealing with heterogeneous robotic platforms. Existing teleoperation systems often fail to guarantee high-precision data collection across diverse types of teleoperation devices. To address this, we developed Open TeleDex, a unified teleoperation framework engineered for demonstration data collection. Open TeleDex specifically tackles the TripleAny challenge, seamlessly supporting any robotic arm, any dexterous hand, and any external input device. Furthermore, we propose a novel hand pose retargeting algorithm that significantly boosts the interoperability of Open TeleDex, enabling robust and accurate compatibility with an even wider spectrum of heterogeneous master and slave equipment. Open TeleDex establishes a foundational, high-quality, and publicly available platform for accelerating both academic research and industry development in complex robotic manipulation and IL.

Open TeleDex: A Hardware-Agnostic Teleoperation System for Imitation Learning based Dexterous Manipulation

TL;DR

Open TeleDex tackles the data bottleneck in robot imitation learning by delivering a hardware-agnostic teleoperation framework rooted in ROS2. It introduces a three-tier architecture with a RealEnv hardware abstraction and a generative hand pose retargeting algorithm to enable cross-robot data collection. The system emphasizes high-fidelity, multi-modal data with a robust TimeSync Manager ensuring per-frame synchronization and a structured dataset output for IL. Evaluations on Bottle Peg-in-Hole and Cube Assembly across COTS and in-house hardware demonstrate strong data synchronization (>99%) and improved temporal fidelity with in-house hardware, validating the TripleAny vision. Collectively, Open TeleDex provides a scalable, extensible platform for accelerating both academic and industrial development in dexterous manipulation and imitation learning.

Abstract

Accurate and high-fidelity demonstration data acquisition is a critical bottleneck for deploying robot Imitation Learning (IL) systems, particularly when dealing with heterogeneous robotic platforms. Existing teleoperation systems often fail to guarantee high-precision data collection across diverse types of teleoperation devices. To address this, we developed Open TeleDex, a unified teleoperation framework engineered for demonstration data collection. Open TeleDex specifically tackles the TripleAny challenge, seamlessly supporting any robotic arm, any dexterous hand, and any external input device. Furthermore, we propose a novel hand pose retargeting algorithm that significantly boosts the interoperability of Open TeleDex, enabling robust and accurate compatibility with an even wider spectrum of heterogeneous master and slave equipment. Open TeleDex establishes a foundational, high-quality, and publicly available platform for accelerating both academic research and industry development in complex robotic manipulation and IL.
Paper Structure (35 sections, 6 equations, 3 figures, 4 tables)

This paper contains 35 sections, 6 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Teleoperation Systems
  4. Hand Pose Retargeting
  5. Demonstration Data for Imitation Learning
  6. Framework of Open TeleDex
  7. System Architecture
  8. Perception
  9. Hand Pose Retargeting:
  10. Control and Data Collection
  11. Control Layer
  12. Data Synchronization Mechanism:
  13. Data Integration Layer
  14. Data Storage Layer
  15. Hardware
  16. ...and 20 more sections

Figures (3)

  • Figure 1: The Framework of Open TeleDex. The diagram illustrates the layered architecture of the system, which is designed to support heterogeneous robot platforms for different requirement of data collection. The system is structurally divided into three modules: Perception, Control and Data Collection, and Hardware.
  • Figure 2: Some of the Hardware Components Supported by Open TeleDex.
  • Figure 3: This shows the task of our evaluation system. The top three images are Bottle Peg in Hole, and the bottom three images are Cube Assembly