Empowering Embodied Manipulation: A Bimanual-Mobile Robot Manipulation Dataset for Household Tasks

TL;DR

BRMData tackles the need for rich, dual-arm, mobile manipulation data in household environments by introducing a 10-task dataset with multi-view RGBD sensing and a new MES metric. It enables rigorous benchmarking of learning methods, including single-task and multi-task approaches, under progressive difficulty and human interaction scenarios. Experimental results illustrate method performance differences, the added challenge of mobility, and the dataset's capacity to reveal robustness and efficiency trade-offs. This dataset is poised to accelerate development of embodied, dexterous manipulation in real-world domestic contexts and beyond.

Abstract

The advancements in embodied AI are increasingly enabling robots to tackle complex real-world tasks, such as household manipulation. However, the deployment of robots in these environments remains constrained by the lack of comprehensive bimanual-mobile robot manipulation data that can be learned. Existing datasets predominantly focus on single-arm manipulation tasks, while the few dual-arm datasets available often lack mobility features, task diversity, comprehensive sensor data, and robust evaluation metrics; they fail to capture the intricate and dynamic nature of household manipulation tasks that bimanual-mobile robots are expected to perform. To overcome these limitations, we propose BRMData, a Bimanual-mobile Robot Manipulation Dataset specifically designed for household applications. BRMData encompasses 10 diverse household tasks, including single-arm and dual-arm tasks, as well as both tabletop and mobile manipulations, utilizing multi-view and depth-sensing data information. Moreover, BRMData features tasks of increasing difficulty, ranging from single-object to multi-object grasping, non-interactive to human-robot interactive scenarios, and rigid-object to flexible-object manipulation, closely simulating real-world household applications. Additionally, we introduce a novel Manipulation Efficiency Score (MES) metric to evaluate both the precision and efficiency of robot manipulation methods in household tasks. We thoroughly evaluate and analyze the performance of advanced robot manipulation learning methods using our BRMData, aiming to drive the development of bimanual-mobile robot manipulation technologies. The dataset is now open-sourced and available at https://embodiedrobot.github.io/.

Figures (15)

• Figure 1: Illustration of our bimanual-mobile robot manipulation tasks.
• Figure 2: Illustration of robot platform.
• Figure 3: Illustration of the multi-view data collection of BRMData. (a) Left Image: Captured from the left wrist-mounted camera, offering detailed visual input from the left-arm's side. (b) Right Image: Obtained from the right wrist-mounted camera, providing critical visual information from the robot's right side. (c) Middle Image: Sourced from the centrally mounted fixed camera, presenting a broad and central perspective of the operational area. (d) Human Perspective View: Captured from a human viewpoint, not through the robot platform, giving an external perspective on the scene.
• Figure 4: Performance comparison in single-arm and dual-arm robot manipulation tasks.
• Figure 5: Task Definition of Bottle Handoff.