Stiffness Copilot: An Impedance Policy for Contact-Rich Teleoperation

Abstract

In teleoperation of contact-rich manipulation tasks, selecting robot impedance is critical but difficult. The robot must be compliant to avoid damaging the environment, but stiff to remain responsive and to apply force when needed. In this paper, we present Stiffness Copilot, a vision-based policy for shared-control teleoperation in which the operator commands robot pose and the policy adjusts robot impedance online. To train Stiffness Copilot, we first infer direction-dependent stiffness matrices in simulation using privileged contact information. We then use these matrices to supervise a lightweight vision policy that predicts robot stiffness from wrist-camera images and transfers zero-shot to real images at runtime. In a human-subject study, Stiffness Copilot achieved safety comparable to using a constant low stiffness while matching the efficiency of using a constant high stiffness.

Figures (2)

• Figure 1: Overview of Stiffness Copilot execution in the human-subjective experiment. Each row shows a representative situation. (A) We first collected contact forces in simulation to infer robot stiffness, which provided training data for Stiffness Copilot. (B) Stiffness Copilot was trained using simulated wrist-camera images (upper) and was deployed using real wrist-camera images (lower) during policy rollout, illustrating the sim-to-real gap. (C) Third-person views from our user study, where a participant teleoperated the robot. (D) Stiffness produced by Stiffness Copilot is visualized as a green ellipsoid, where a longer ellipsoid axis indicates larger translation stiffness in that direction. We also annotate the stiffness magnitude along each axis. In Vase Wiping (top two rows), despite different grasp poses, the robot remained compliant perpendicular to the contact surface and stiff in other directions. In Peg-in-Hole (middle two rows), the robot was stiff while picking up the block and compliant during peg insertion. In Door Opening (bottom two rows), despite different grasp poses, the robot was stiff along the opening direction and compliant in other directions.
• Figure 2: Visualization of our study results. For box and whisker plots, the top and bottom of each box represent the first and third quartiles, and the line inside each box is the statistical median of the data. The length of the box is defined as the interquartile range (IQR). The whiskers are within a maximum of 1.5 IQR. $*/**/{*}{*}{*}$ indicate significance levels $p<.05/.01/.001$. $d$ is Cohen's $d$ for paired $t$-tests and $r$ is the Wilcoxon effect size.