Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MIRROR: Visual Motion Imitation via Real-time Retargeting and Teleoperation with Parallel Differential Inverse Kinematics

Junheng Li, Lizhi Yang, Aaron D. Ames

Abstract

Real-time humanoid teleoperation requires inverse kinematics (IK) solvers that are both responsive and constraint-safe under kinematic redundancy and self-collision constraints. While differential IK enables efficient online retargeting, its locally linearized updates are inherently basin-dependent and often become trapped near joint limits, singularities, or active collision boundaries, leading to unsafe or stagnant behavior. We propose a GPU-parallelized, continuation-based differential IK that improves escape from such constraint-induced local minima while preserving real-time performance, promoting safety and stability. Multiple constrained IK quadratic programs are evaluated in parallel, together with a self-collision avoidance control barrier function (CBF), and a Lyapunov-based progression criterion selects updates that reduce the final global task-space error. The method is paired with a visual skeletal pose estimation pipeline that enables robust, real-time upper-body teleoperation on the THEMIS humanoid robot hardware in real-world tasks.

MIRROR: Visual Motion Imitation via Real-time Retargeting and Teleoperation with Parallel Differential Inverse Kinematics

Abstract

Real-time humanoid teleoperation requires inverse kinematics (IK) solvers that are both responsive and constraint-safe under kinematic redundancy and self-collision constraints. While differential IK enables efficient online retargeting, its locally linearized updates are inherently basin-dependent and often become trapped near joint limits, singularities, or active collision boundaries, leading to unsafe or stagnant behavior. We propose a GPU-parallelized, continuation-based differential IK that improves escape from such constraint-induced local minima while preserving real-time performance, promoting safety and stability. Multiple constrained IK quadratic programs are evaluated in parallel, together with a self-collision avoidance control barrier function (CBF), and a Lyapunov-based progression criterion selects updates that reduce the final global task-space error. The method is paired with a visual skeletal pose estimation pipeline that enables robust, real-time upper-body teleoperation on the THEMIS humanoid robot hardware in real-world tasks.
Paper Structure (26 sections, 5 theorems, 25 equations, 7 figures, 1 table)

This paper contains 26 sections, 5 theorems, 25 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Pipeline and Approach
  3. Perception and Pose Processing
  4. Visual Skeleton
  5. Pose Filtering and Robustification
  6. Task-space Command Continuation
  7. Command Scaling
  8. Command Continuation
  9. GPU-accelerated Whole-body Kinematic Retargeting
  10. Self-collision CBFs
  11. Family of Parallel QPs
  12. Lyapunov Progress Certificate and Selection
  13. Robot Hardware
  14. Parallel Differential Inverse Kinematics
  15. Local Basin Dependence of Differential IK
  16. ...and 11 more sections

Key Result

Lemma 1

Consider the regularized nonlinear IK objective At an iterate $q_k$, we can linearize by eqn. eq:firstKin. Then it yields the convex QP eq:pc_qp_obj, which computes a Gauss--Newton step for minimizing $F(q)$ at $q_k$ and has a global minimizer $\Delta q_k$ for the local model.

Figures (7)

  • Figure 1: Real-time human-humanoid motion mirroring teleoperation. Full experiment videos https://youtu.be/LvwIwOTTu9g
  • Figure 2: MIRROR pipeline architecture.
  • Figure 3: Snapshots of real-time teleoperation. Pose mirroring and mimicking.
  • Figure 4: Snapshots of useful real-world tasks through MIRROR teleoperation.
  • Figure 5: Comparison of motion capture devices tracking hand location.
  • ...and 2 more figures

Theorems & Definitions (12)

  • Remark 1
  • Lemma 1: Differential IK as a Gauss--Newton step
  • Proposition 1: Basin dependence of differential IK
  • Remark 2
  • Theorem 1: Progress-certified escape probability increases with $K$
  • proof
  • Remark 3
  • Lemma 2: Escape probability increases with $K$
  • proof
  • Lemma 3: Volume ratio decays exponentially with dimension
  • ...and 2 more