Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TeleDex: Accessible Dexterous Teleoperation

Omar Rayyan, Maximilian Gillesm, Yuchen Cui

Abstract

Despite increasing dataset scale and model capacity, robot manipulation policies still struggle to generalize beyond their training distributions. As a result, deploying state-of-the-art policies in new environments, tasks, or robot embodiments often requires collecting additional demonstrations. Enabling this in real-world deployment settings requires tools that allow users to collect demonstrations quickly, affordably, and with minimal setup. We present TeleDex, an open-source system for intuitive teleoperation of dexterous hands and robotic manipulators using any readily available phone. The system streams low-latency 6-DoF wrist poses and articulated 21-DoF hand state estimates from the phone, which are retargeted to robot arms and multi-fingered hands without requiring external tracking infrastructure. TeleDex supports both a handheld phone-only mode and an optional 3D-printable hand-mounted interface for finger-level teleoperation. By lowering the hardware and setup barriers to dexterous teleoperation, TeleDex enables users to quickly collect demonstrations during deployment to support policy fine-tuning. We evaluate the system across simulation and real-world manipulation tasks, demonstrating its effectiveness as a unified scalable interface for robot teleoperation. All software and hardware designs, along with demonstration videos, are open-source and available at orayyan.com/teledex.

TeleDex: Accessible Dexterous Teleoperation

Abstract

Despite increasing dataset scale and model capacity, robot manipulation policies still struggle to generalize beyond their training distributions. As a result, deploying state-of-the-art policies in new environments, tasks, or robot embodiments often requires collecting additional demonstrations. Enabling this in real-world deployment settings requires tools that allow users to collect demonstrations quickly, affordably, and with minimal setup. We present TeleDex, an open-source system for intuitive teleoperation of dexterous hands and robotic manipulators using any readily available phone. The system streams low-latency 6-DoF wrist poses and articulated 21-DoF hand state estimates from the phone, which are retargeted to robot arms and multi-fingered hands without requiring external tracking infrastructure. TeleDex supports both a handheld phone-only mode and an optional 3D-printable hand-mounted interface for finger-level teleoperation. By lowering the hardware and setup barriers to dexterous teleoperation, TeleDex enables users to quickly collect demonstrations during deployment to support policy fine-tuning. We evaluate the system across simulation and real-world manipulation tasks, demonstrating its effectiveness as a unified scalable interface for robot teleoperation. All software and hardware designs, along with demonstration videos, are open-source and available at orayyan.com/teledex.
Paper Structure (25 sections, 2 equations, 9 figures, 2 tables)

This paper contains 25 sections, 2 equations, 9 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Scaling Robotics Data
  4. Phone-based Teleoperation Systems
  5. Dexterous Hand Teleoperation
  6. System Overview
  7. Basic Hand-held Setting
  8. Wrist-Mounted Setting
  9. TeleDex App and Python API
  10. Experimental Evaluation
  11. Experimental Setup
  12. RQ1: Efficiency of Collecting Demonstrations
  13. Task Setup
  14. Metrics
  15. Participants and Procedures
  16. ...and 10 more sections

Figures (9)

  • Figure 2: TeleDex System Architecture. The system supports two settings: (left) a Dexterous Setting using a 3D-printable wrist-mounted interface for 21-DoF hand and 6-DoF wrist tracking, and (right) a Phone-Only Setting for 6-DoF end-effector control. Captured states are processed via Dex-Retarget to map inputs to arbitrary robotic URDF models.
  • Figure 3: Screenshots of TeleDex phone application.
  • Figure 4: Example tasks from MolmoSpaces-Bench.
  • Figure 5: MolmoSpaces-Bench user study setup: left, using SpaceMouse; right, using a keyboard.
  • Figure 6: Average time-to-success for the MS-Open and MS-Pick tasks, across users in MolmoSpaces-Bench.
  • ...and 4 more figures