DexHub and DART: Towards Internet Scale Robot Data Collection

TL;DR

The paper tackles the data bottleneck in robotic learning by introducing DART, an Augmented Reality teleoperation platform that runs in cloud-hosted simulation and enables crowd-sourced data collection with minimal hardware setup. It demonstrates that AR-enabled teleoperation yields far higher data throughput and lower fatigue than real-world teleoperation, and that policies trained on DART data transfer to the real world with improved robustness when augmented in simulation. DexHub serves as a cloud data hub to log, share, and monetize demonstrations, fostering an internet-scale dataset for robot learning. Together, DART and DexHub aim to democratize robot data collection while providing strong Sim2Real transfer, though physics/realism limits remain and real-world data continues to play a vital role.

Abstract

The quest to build a generalist robotic system is impeded by the scarcity of diverse and high-quality data. While real-world data collection effort exist, requirements for robot hardware, physical environment setups, and frequent resets significantly impede the scalability needed for modern learning frameworks. We introduce DART, a teleoperation platform designed for crowdsourcing that reimagines robotic data collection by leveraging cloud-based simulation and augmented reality (AR) to address many limitations of prior data collection efforts. Our user studies highlight that DART enables higher data collection throughput and lower physical fatigue compared to real-world teleoperation. We also demonstrate that policies trained using DART-collected datasets successfully transfer to reality and are robust to unseen visual disturbances. All data collected through DART is automatically stored in our cloud-hosted database, DexHub, which will be made publicly available upon curation, paving the path for DexHub to become an ever-growing data hub for robot learning. Videos are available at: https://dexhub.ai/project

Figures (11)

• Figure 1: 4 finger keypoints used as tracking points for robots with parallel-jaw grippers.
• Figure 2: Data throughput comparison between DART and real-world teleoperation systems. For each robot and task, five participants were asked to teleoperate the tasks as many as possible for 7 minutes. For real-world teleoperation, kinematically equivalent teacher device, i.e., kinematic double, was used as a teleoperation interface.
• Figure 3: DART allows operators to spend more time on actual data collection, rather than supplementary tasks such as resetting the environment for every task completion or dealing with hardware failures.
• Figure 4: Qualitative comparison between different teleoperation interfaces amongst user study participants. Participants reported that DART is enjoyable, physiaclly less fatiguing and allows better visual observation during teleoperation.
• Figure 5: Nominal Lab Setting