SPIRIT: Perceptive Shared Autonomy for Robust Robotic Manipulation under Deep Learning Uncertainty

TL;DR

This work proposes a concept termed perceptive shared autonomy, in which uncertainty estimates from DL based perception are used to regulate the level of autonomy, and develops a system, named SPIRIT, which improves both manipulation performance and system reliability.

Abstract

Deep learning (DL) has enabled impressive advances in robotic perception, yet its limited robustness and lack of interpretability hinder reliable deployment in safety critical applications. We propose a concept termed perceptive shared autonomy, in which uncertainty estimates from DL based perception are used to regulate the level of autonomy. Specifically, when the robot's perception is confident, semi-autonomous manipulation is enabled to improve performance; when uncertainty increases, control transitions to haptic teleoperation for maintaining robustness. In this way, high-performing but uninterpretable DL methods can be integrated safely into robotic systems. A key technical enabler is an uncertainty aware DL based point cloud registration approach based on the so called Neural Tangent Kernels (NTK). We evaluate perceptive shared autonomy on challenging aerial manipulation tasks through a user study of 15 participants and realization of mock-up industrial scenarios, demonstrating reliable robotic manipulation despite failures in DL based perception. The resulting system, named SPIRIT, improves both manipulation performance and system reliability. SPIRIT was selected as a finalist of a major industrial innovation award.

Figures (13)

• Figure 1: Amongst many potential applications, we examine two use cases: (a) extending the mobility of an inspection robot via pick and place, and (b) performing manipulation of industrial flange valves. The demonstration of these use cases along with the task sequence (1-4) is illustrated. Left: overview of the tasks. Right: robot performing the tasks from the left image.
• Figure 2: User interface of SPIRIT that communicates robot's notion of uncertainty to a human operator. In addition to haptic, we use XR for 2D and 3D visualization, as well as telemetry.
• Figure 3: Left: the proposed partitioned approach to point cloud registration. Matching source point cloud to the target point cloud (partitioned from the digital twin) is easier than the use of the entire point cloud from the digital twin of environments. Target point cloud is captured based on robot's position in the digital twin when performing manipulation tasks. Right: the proposed registration architecture. Lie algebra (a 6D vector) is inferred from neural networks (top). Then, Gaussian Processes (GPs) are utilized for uncertainty estimation (bottom). Uncertainty estimates are inferred over predictions in lie algebra.
• Figure 4: Results of ablation studies. Lower the better. Values from the best three baselines are shown in legend. Lower rotation and translation errors are observed for the baselines relying on partitioning. GP (our approach) produced more reliable uncertainty estimates for nominal conditions (ID) and failure cases (OOD), indicated by lower NLL.
• Figure 5: Results from the demonstration of industrial scenarios. Left: the point cloud registration is visualized with covariances. Blue ellipse represent the covariance over the translations while yellow ellipse depict the covariance over rotations for each axis. Middle: pick-and-place of heavy objects; (1) grasping the object, (a) failures in DL-based perception over 10 s, and (2) pick-and-place with over 50 N end-effector forces. Right: forceful operation of flange valve; (3) grasping of the valve bar, (b) failures in DL-based perception while grasping, and (4) closing of the valve with about 50N end-effector forces (norm).