Efficient and Reliable Teleoperation through Real-to-Sim-to-Real Shared Autonomy

Abstract

Fine-grained, contact-rich teleoperation remains slow, error-prone, and unreliable in real-world manipulation tasks, even for experienced operators. Shared autonomy offers a promising way to improve performance by combining human intent with automated assistance, but learning effective assistance in simulation requires a faithful model of human behavior, which is difficult to obtain in practice. We propose a real-to-sim-to-real shared autonomy framework that augments human teleoperation with learned corrective behaviors, using a simple yet effective k-nearest-neighbor (kNN) human surrogate to model operator actions in simulation. The surrogate is fit from less than five minutes of real-world teleoperation data and enables stable training of a residual copilot policy with model-free reinforcement learning. The resulting copilot is deployed to assist human operators in real-world fine-grained manipulation tasks. Through simulation experiments and a user study with sixteen participants on industry-relevant tasks, including nut threading, gear meshing, and peg insertion, we show that our system improves task success for novice operators and execution efficiency for experienced operators compared to direct teleoperation and shared-autonomy baselines that rely on expert priors or behavioral-cloning pilots. In addition, copilot-assisted teleoperation produces higher-quality demonstrations for downstream imitation learning.

Figures (7)

• Figure 1: Overview of our real-to-sim-to-real shared autonomy framework. A small amount of real teleoperation data ($<5$ minutes) is used to construct a lightweight human surrogate, which drives simulation-based training of a residual copilot policy. At deployment, the copilot provides low-level corrective actions that combine with human commands to produce reliable and efficient shared autonomy for fine-grained, contact-rich manipulation tasks, where direct teleoperation struggles with precise alignment, axis-constrained rotation, and contact regulation, including nut threading, gear meshing, and peg insertion.
• Figure 2: Method overview of residual copilot learning with a real-to-sim-to-real pipeline.Left: less than five minutes of real-world teleoperation interaction data is collected under admittance control. Center: in simulation, a lightweight kNN-based human surrogate constructed from real-world data generates base actions conditioned on the environment state, while a residual copilot policy is trained with reinforcement learning to produce corrective residual actions. Right: at deployment, the residual copilot assists the human operator in zero-shot, conditioning on the estimated environment state and human base actions. The residual formulation preserves human intent while improving alignment, contact regulation, and execution robustness.
• Figure 3: Human experiment results across three contact-rich assembly tasks. Rows correspond to Gear Meshing, Nut Threading, and Peg Insertion. Left: success rate and mean completion time across successful episodes (solid line; shaded region denotes standard deviation; dashed line indicates per-method mean). Right: subjective workload (NASA-TLX subscales; lower is better) and user satisfaction (higher is better). Across tasks, the Residual Copilot improves success rate and can effectively reduce completion time, while lowering reported workload and increasing user satisfaction compared to direct teleoperation and a residual baseline.
• Figure 4: User study qualitative results. Representative trajectory temporal overlays comparing unassisted teleoperation and our Residual Copilot on three high-precision tasks. Top row (effectiveness): our copilot enables reliable task completion in cases where teleoperation fails due to force, rotation, or grasp errors. Bottom row (efficiency): when both succeed, our copilot consistently reduces completion time and exemplifies smooth motions. Together, these results demonstrate that our method provides both reliable and efficient assistance in challenging manipulation scenarios.
• Figure 5: xArm 7 Setup.