A Closed-Form Geometric Retargeting Solver for Upper Body Humanoid Robot Teleoperation

TL;DR

SEW-Mimic introduces a closed-form, orientation-based retargeting solver for upper-body humanoid robots that directly aligns human shoulder-elbow-wrist vectors to a robot’s corresponding axes, yielding provable optimality and millisecond-range inference on CPUs. The method proceeds in a three-stage sequence—upper-arm alignment, lower-arm alignment, and wrist alignment—via canonical Subproblems (SP1/SP2/SP4), enabling a body-centric, calibration-free mapping that is robust to size differences between human and robot embodiments. A fast safety filter based on XPBD uses SEW-Mimic as a drop-in IK to reduce self-collisions in bimanual teleoperation, while continuous-time collision checking minimizes unsafe poses. Empirical results show SEW-Mimic outperforms end-effector or keypoint-based baselines in speed and pose accuracy, improves pilot task success, and accelerates full-body retargeting, with demonstrations on Kinova Gen3, Rainbow RB-Y1, and Unitree G1. Limitations include reliance on kinematic assumptions, frame-syncing requirements, perception noise, and the lack of formal guarantees for the safety filter; future work aims to extend optimality to joint limits, enhance perception robustness, and further reduce latency.

Abstract

Retargeting human motion to robot poses is a practical approach for teleoperating bimanual humanoid robot arms, but existing methods can be suboptimal and slow, often causing undesirable motion or latency. This is due to optimizing to match robot end-effector to human hand position and orientation, which can also limit the robot's workspace to that of the human. Instead, this paper reframes retargeting as an orientation alignment problem, enabling a closed-form, geometric solution algorithm with an optimality guarantee. The key idea is to align a robot arm to a human's upper and lower arm orientations, as identified from shoulder, elbow, and wrist (SEW) keypoints; hence, the method is called SEW-Mimic. The method has fast inference (3 kHz) on standard commercial CPUs, leaving computational overhead for downstream applications; an example in this paper is a safety filter to avoid bimanual self-collision. The method suits most 7-degree-of-freedom robot arms and humanoids, and is agnostic to input keypoint source. Experiments show that SEW-Mimic outperforms other retargeting methods in computation time and accuracy. A pilot user study suggests that the method improves teleoperation task success. Preliminary analysis indicates that data collected with SEW-Mimic improves policy learning due to being smoother. SEW-Mimic is also shown to be a drop-in way to accelerate full-body humanoid retargeting. Finally, hardware demonstrations illustrate SEW-Mimic's practicality. The results emphasize the utility of SEW-Mimic as a fundamental building block for bimanual robot manipulation and humanoid robot teleoperation.

Figures (18)

• Figure 1: Comparison of human arm with keypoints and 7-DOF upper body robot arm showing links (boxes) and joints (cylinders show rotation axes).
• Figure 2: SEW-Mimic takes in human arm keypoints (left) and aligns the robot's upper arm, then lower arm, then wrist.
• Figure 3: Our safety filter uses capsules (right) to compute and avoid self collision, as shown on Rainbow RBY1 hardware.
• Figure 4: Retargeting alignment error and inference time of SEW-Mimic vs. baselines.
• Figure 5: The Glass Gap task requires moving a red box from between wine glasses to a large green goal on the left. We show an example pose from our user study showing SEW-Mimic on the left vs. MINK-IK on the right, which shows how a lack of explicit elbow control can cause task failure from unexpected joint self-motion.

Theorems & Definitions (12)

• Remark 1: Body-Centric Frame
• Proposition 2
• Remark 3
• Remark 4
• Lemma 5: Optimality of Solving Subproblem 4
• proof
• Lemma 6: Optimality of Solving Subproblem 2
• proof
• Lemma 7: Optimality of Solving Subproblem 1
• proof