ResPilot: Teleoperated Finger Gaiting via Gaussian Process Residual Learning

TL;DR

ResPilot addresses the challenge of teleoperating multi-fingered hands for in-hand manipulation by augmenting a base retargeter with a residual Gaussian Process learned from a small labeled calibration set. The residual GP per finger expands the robot hand's reachable workspace, enabling previously unseen finger gaiting while preserving precise control in near-tap pinch regions. Finger constraints enable stable contact during object reorientation. Experiments on six dexterous tasks with real hardware show rapid calibration (~4.5 minutes) and online performance around 8 Hz (4.5 Hz under constraint), with improved workspace relative to baselines. This approach offers a practical path to scalable teleoperation and data collection for dexterous manipulation.

Abstract

Dexterous robot hand teleoperation allows for long-range transfer of human manipulation expertise, and could simultaneously provide a way for humans to teach these skills to robots. However, current methods struggle to reproduce the functional workspace of the human hand, often limiting them to simple grasping tasks. We present a novel method for finger-gaited manipulation with multi-fingered robot hands. Our method provides the operator enhanced flexibility in making contacts by expanding the reachable workspace of the robot hand through residual Gaussian Process learning. We also assist the operator in maintaining stable contacts with the object by allowing them to constrain fingertips of the hand to move in concert. Extensive quantitative evaluations show that our method significantly increases the reachable workspace of the robot hand and enables the completion of novel dexterous finger gaiting tasks. Project website: http://respilot-hri.github.io

Figures (8)

• Figure 1: Our method retargets human hand configurations to a robot hand, enabling finger-gaited in-hand manipulation. We evaluate our method on six highly dexterous tasks with the palm facing upward and downward.
• Figure 2: Hand keypoint vectors.
• Figure 3: The full set of 24 calibration poses used to learn the residual GP. Active fingers for each calibration configuration are colored green.
• Figure 4: Task performance for two pilots on the 6 tasks with and without using the fingertip constraints. The plot shows the average scores; error bars show the relative variability ($0.2\sigma$).
• Figure 5: Qualitative comparisons between retargeters. Res-GP reproduces the operator's pinch at many distances from the palm.