TelePreview: A User-Friendly Teleoperation System with Virtual Arm Assistance for Enhanced Effectiveness

TL;DR

TelePreview tackles the core challenges of dexterous teleoperation—usability, safety, and cross-platform transfer—by presenting a low-cost system that offers real-time virtual previews of planned robot actions. The approach combines IMU-based wrist tracking, mocap glove hand pose capture, SMPL-X body representation, and a non-collision retargeting network to map human motions to high-DoF robot hands with safety guarantees. A dedicated Preview Module aligns a virtual robot with the physical robot via AprilTag calibration and renders multi-view overlays, enabling users to verify and refine motions before execution, thereby improving data quality for imitation learning. Across five real-world tasks and multiple end-effectors, TelePreview shows higher success rates and shorter execution times for new users, with the most pronounced gains on high-DoF, dexterous hands, while maintaining generalizability and straightforward deployment for diverse hardware.

Abstract

Teleoperation provides an effective way to collect robot data, which is crucial for learning from demonstrations. In this field, teleoperation faces several key challenges: user-friendliness for new users, safety assurance, and transferability across different platforms. While collecting real robot dexterous manipulation data by teleoperation to train robots has shown impressive results on diverse tasks, due to the morphological differences between human and robot hands, it is not only hard for new users to understand the action mapping but also raises potential safety concerns during operation. To address these limitations, we introduce TelePreview. This teleoperation system offers real-time visual feedback on robot actions based on human user inputs, with a total hardware cost of less than $1,000. TelePreview allows the user to see a virtual robot that represents the outcome of the user's next movement. By enabling flexible switching between command visualization and actual execution, this system helps new users learn how to demonstrate quickly and safely. We demonstrate that it outperforms other teleoperation systems across five tasks, emphasize its ease of use, and highlight its straightforward deployment across diverse robotic platforms. We release our code and a deployment document on our website https://nus-lins-lab.github.io/telepreview-web/.

Figures (16)

• Figure 1: TelePreview is a user-friendly teleoperation system enabling the real-time virtual preview before robot execution.
• Figure 2: Overview of the System Architecture (1) Various input devices for capturing human motion. Only one device from each of the two input groups (wrist pose and hand gesture) is required during operation. The VR headset is shown at the intersection of the two groups, as it can capture both; (2) A processing pipeline based on SMPL-X that performs joint mapping and collision-free retargeting; (3) Output to various robot platforms for executing the mapped motions.
• Figure 3: Pipeline of Our Modules: The system tracks user wrist and hand poses, maps them to robot joint configurations through joint-to-joint mapping and non-collision retargeting, and provides a visual preview before physical execution. We achieve precise alignment between virtual and physical robots through AprilTag calibration.
• Figure 4: Comparison of Hand Configuration Retargeting Methods: (a) shows the direct mapping between human and robot hands leading to self-collision; (b) demonstrates our collision-aware retargeting approach that maintains safe configurations.
• Figure 5: Our Transformation Relationship: The number in the circle denotes the order of transformation acquisition.