DexEMG: Towards Dexterous Teleoperation System via EMG2Pose Generalization

TL;DR

DexEMG is presented, a lightweight and cost-effective teleoperation system leveraging surface electromyography (sEMG) to bridge the gap between human intent and robotic execution and offers a scalable and intuitive interface for both general-purpose robotic manipulation and assistive technologies.

Abstract

High-fidelity teleoperation of dexterous robotic hands is essential for bringing robots into unstructured domestic environments. However, existing teleoperation systems often face a trade-off between performance and portability: vision-based capture systems are constrained by costs and line-of-sight requirements, while mechanical exoskeletons are bulky and physically restrictive. In this paper, we present DexEMG, a lightweight and cost-effective teleoperation system leveraging surface electromyography (sEMG) to bridge the gap between human intent and robotic execution. We first collect a synchronized dataset of sEMG signals and hand poses via a MoCap glove to train EMG2Pose, a neural network capable of continuously predicting hand kinematics directly from muscle activity. To ensure seamless control, we develop a robust hand retargeting algorithm that maps the predicted poses onto a multi-fingered dexterous hand in real-time. Experimental results demonstrate that DexEMG achieves high precision in diverse teleoperation tasks. Notably, our system exhibits strong generalization capabilities across novel objects and complex environments without the need for intensive individual-specific recalibration. This work offers a scalable and intuitive interface for both general-purpose robotic manipulation and assistive technologies.

Figures (6)

• Figure 1: Overview of the DexEMG system. (Top-Left) The teleoperation interface leverages sEMG signals for portable control. (Top-Right) The EMG2Pose architecture utilizes a TDS-based encoder and a feedback-enabled LSTM decoder for continuous pose regression. (Bottom) DexEMG demonstrates robust generalization across diverse object categories and successfully completes long-horizon manipulation tasks including desktop packaging and wiping.
• Figure 2: Overview of the DexEMG system architecture. (Top-Left) Multimodal data collection captures synchronized sEMG signals and ground-truth hand kinematics. (Bottom-Left) The EMG2Pose perception engine extracts spatial-temporal features via TDS stages and uses an LSTM-based decoder to reconstruct continuous hand poses. (Top-Right) The real-time deployment pipeline integrates the predicted hand action chunks with wrist tracking to control a multi-fingered robotic hand.
• Figure 3: Representative snapshots of continuous pose estimation via EMG2Pose. EMG2Pose model accurately tracks high-dimensional hand kinematics across various grasping and in-hand manipulation gestures. The estimated poses (rendered) show high correspondence with the user's actual hand motions.
• Figure 4: Trained and unseen object sets for evaluation. The collection encompasses diverse geometries with varying sizes, masses, and aspect ratios.
• Figure 5: Grasping performance across diverse geometries and generalization scenarios, including trained objects, unseen objects, and novel environments.