Fast Explicit-Input Assistance for Teleoperation in Clutter

TL;DR

This work presents a new assistance interface for robotic manipulation where an operator can explicitly communicate a manipulation goal by pointing the end-effector, and finds that operators prefer the explicit interface, experience fewer pick failures and report lower cognitive workload.

Abstract

The performance of prediction-based assistance for robot teleoperation degrades in unseen or goal-rich environments due to incorrect or quickly-changing intent inferences. Poor predictions can confuse operators or cause them to change their control input to implicitly signal their goal. We present a new assistance interface for robotic manipulation where an operator can explicitly communicate a manipulation goal by pointing the end-effector. The pointing target specifies a region for local pose generation and optimization, providing interactive control over grasp and placement pose candidates. We compare the explicit pointing interface to an implicit inference-based assistance scheme in a within-subjects user study (N=20) where participants teleoperate a simulated robot to complete a multi-step singulation and stacking task in cluttered environments. We find that operators prefer the explicit interface, experience fewer pick failures and report lower cognitive workload. Our code is available at: https://github.com/NVlabs/fast-explicit-teleop

Figures (8)

• Figure 1: Implicit assistance (left) funnels the operator toward the goal predicted based on (for instance) the recent trajectory. The operator is not intended to change their input to influence the assistance. Explicit assistance (right) affords the operator direct control over the inferred goal by pointing the gripper toward the object of interest. A local optimization selects a feasible, collision free pose.
• Figure 2: Our realizations of explicit grasping (left) and placing assistance (right) both center on the interaction of a ray from the gripper with scene geometry. A projected anchor pose is calculated then used to select amongst a set of candidate assistance poses.
• Figure 3: The operator controls the robot while looking at two camera views displayed picture-in-picture (left). Assistance suggestions are shown as a "ghost gripper" for grasping and a "ghost shape" for placing actions (right). Ray visualizations are exaggerated for legibility in print. The experimental task involved participants extracting and stacking blue and pink blocks that were initially scattered in one of three clutter configurations (bottom).
• Figure 4: Survival analysis ($\uparrow$) of participant's completion of the task over time. Lines plot percentage of participants that completed the task at the time and Xs mark termination without completion. Differences lie within the 95% confidence interval, with a trend that the probability of having completed the task grows most quickly for the explicit input interface and reaches a higher peak.
• Figure 5: Raw data for subjective scores collected on 7--point scale with density estimates overlaid. Point and bar show estimated marginal mean with 95% confidence interval.