Predictive Visuo-Tactile Interactive Perception Framework for Object Properties Inference

TL;DR

A novel predictive perception framework for identifying object properties of the diverse objects by leveraging versatile exploratory actions: non-prehensile pushing and prehensile pulling is proposed and a novel active shape perception is proposed to seamlessly initiate exploration.

Abstract

Interactive exploration of the unknown physical properties of objects such as stiffness, mass, center of mass, friction coefficient, and shape is crucial for autonomous robotic systems operating continuously in unstructured environments. Precise identification of these properties is essential to manipulate objects in a stable and controlled way, and is also required to anticipate the outcomes of (prehensile or non-prehensile) manipulation actions such as pushing, pulling, lifting, etc. Our study focuses on autonomously inferring the physical properties of a diverse set of various homogeneous, heterogeneous, and articulated objects utilizing a robotic system equipped with vision and tactile sensors. We propose a novel predictive perception framework for identifying object properties of the diverse objects by leveraging versatile exploratory actions: non-prehensile pushing and prehensile pulling. As part of the framework, we propose a novel active shape perception to seamlessly initiate exploration. Our innovative dual differentiable filtering with Graph Neural Networks learns the object-robot interaction and performs consistent inference of indirectly observable time-invariant object properties. In addition, we formulate a $N$-step information gain approach to actively select the most informative actions for efficient learning and inference. Extensive real-robot experiments with planar objects show that our predictive perception framework results in better performance than the state-of-the-art baseline and demonstrate our framework in three major applications for i) object tracking, ii) goal-driven task, and iii) change in environment detection.

Figures (26)

• Figure 1: Overview of the proposed framework for visuo-tactile based interactive perception framework for active object exploration with three main components. 1. Uses visual information to actively estimate the shape of diverse objects based on superquadrics. 2. Actively selects the most informative action affordance for interaction. 3. Utilizes dual differentiable filtering for the estimation of objects' properties using visual and tactile information.
• Figure 2: Our proposed framework is presented in detail for interactively inferring the diverse objects using visuo-tactile sensing. The framework starts in the learning phase followed by the inference phase.
• Figure 3: Illustration of a few basic superquadric shapes with the proposed non-linearity for one of the primary shapes
• Figure 4: Illustration of the shape perception approach. The entropy of each point calculated from Eq. \ref{['eq:fullentropy']} is mapped to 0-255 red channel of the sampled superquadrics, a higher red indicates higher entropy. Viewpoint entropy computed from Eq. \ref{['eq:projective_transform']} of 2 sampled points $I_{2}$ and $I_{10}$ are also presented
• Figure 5: a) Illustration of the proposed graph representation of an example articulated object with two links b) Novel graph propagation for updating the graphical model from time $t-1$ to $t$ for the example object. The support edges $e_1, e_2$, the edge $e_6$ contains contact force or tactile information [Improve]