Towards Exploratory and Focused Manipulation with Bimanual Active Perception: A New Problem, Benchmark and Strategy

TL;DR

The paper introduces Exploratory and Focused Manipulation ($EFM$), a framework to address visual occlusion by actively gathering information to complete challenging manipulation tasks. It proposes the $EFM$-10 benchmark and the Bimanual Active Perception ($BAP$) strategy, which uses a non-operating arm for eye-in-hand vision and the operating arm for force sensing, with a real-world dataset ($BAPData$) to enable imitation-learning evaluation. Experiments demonstrate that active vision involving the operating end-effector in the view improves performance, and force sensing enables neural force compliance that aids delicate manipulation, while also highlighting failure modes to guide future work. The results establish a benchmark and baseline for future $EFM$ research and suggest directions such as enhanced semantic conditioning, spatial reasoning, and active viewpoint searching to further advance bimanual manipulation under occlusion.

Abstract

Recently, active vision has reemerged as an important concept for manipulation, since visual occlusion occurs more frequently when main cameras are mounted on the robot heads. We reflect on the visual occlusion issue and identify its essence as the absence of information useful for task completion. Inspired by this, we come up with the more fundamental problem of Exploratory and Focused Manipulation (EFM). The proposed problem is about actively collecting information to complete challenging manipulation tasks that require exploration or focus. As an initial attempt to address this problem, we establish the EFM-10 benchmark that consists of 4 categories of tasks that align with our definition (10 tasks in total). We further come up with a Bimanual Active Perception (BAP) strategy, which leverages one arm to provide active vision and another arm to provide force sensing while manipulating. Based on this idea, we collect a dataset named BAPData for the tasks in EFM-10. With the dataset, we successfully verify the effectiveness of the BAP strategy in an imitation learning manner. We hope that the EFM-10 benchmark along with the BAP strategy can become a cornerstone that facilitates future research towards this direction. Project website: EFManipulation.github.io.

Figures (5)

• Figure 1: An illustration of the BAPData dataset collected for the 10 tasks in EFM-10. The left wrist-view images (left column), main-view images (middle column), and right wrist-view images (right column) are shown temporally for each example.
• Figure 2: An overview of the hardware system and the objects.
• Figure 3: A way to incorporate force sensing into the GR-MG policy, where we include Force/Torque as extra input and train the model to additionally predict future Force/Torque.
• Figure 4: Visualization of a rollout by the GR-MG policy with force sensing. In the first row, the images captured by the active view are shown. In the second row, real-time $x, y, z$ Force values of the operating end effector are visualized. In the third row, the predicted future $x, y, z$ Force values are visualized.
• Figure 5: Typical failure cases by the policies that we trained with BAPData using imitation learning.