World In Your Hands: A Large-Scale and Open-source Ecosystem for Learning Human-centric Manipulation in the Wild

TL;DR

World In Your Hands (WiYH) introduces a large-scale, open-source ecosystem for learning human-centric manipulation in the wild, addressing data scarcity and misalignment with three core components: the Oracle Suite for wearable, markerless data collection; the WiYH Dataset, a 1,000+ hour multimodal resource spanning diverse real-world scenarios; and comprehensive benchmarks spanning perception to action. The authors demonstrate the ecosystem's value via HVL, HWM, and HVT evaluations, showing that human-centric data improves generalization and robustness of dexterous manipulation policies, and they explore cross-embodiment transfer and real-robot validation. Key contributions include the hardware-software stack for scalable data collection, rich multi-modal annotations (including CoT reasoning for tasks), and benchmarks that span scene understanding, world modeling, and actionable policies. This work sets a foundation for scalable embodied AI pre-training and paves the way for future simulation and privacy-preserving deployment in real-world settings.

Abstract

Large-scale pre-training is fundamental for generalization in language and vision models, but data for dexterous hand manipulation remains limited in scale and diversity, hindering policy generalization. Limited scenario diversity, misaligned modalities, and insufficient benchmarking constrain current human manipulation datasets. To address these gaps, we introduce World In Your Hands (WiYH), a large-scale open-source ecosystem for human-centric manipulation learning. WiYH includes (1) the Oracle Suite, a wearable data collection kit with an auto-labeling pipeline for accurate motion capture; (2) the WiYH Dataset, featuring over 1,000 hours of multi-modal manipulation data across hundreds of skills in diverse real-world scenarios; and (3) extensive annotations and benchmarks supporting tasks from perception to action. Furthermore, experiments based on the WiYH ecosystem show that integrating WiYH's human-centric data significantly enhances the generalization and robustness of dexterous hand policies in tabletop manipulation tasks. We believe that World In Your Hands will bring new insights into human-centric data collection and policy learning to the community.

Figures (15)

• Figure 1: Oracle Suite: Human-centric Data Collection Suite. It is primarily composed of three integrated components: (1) H-FPVHive: A first-person perception suite equipped with multiple cameras of different modalities to comprehensively record the operator's environmental context. (2) H-Glove: A hand motion capture and tactile perception module. It integrates motion capture gloves, tactile sensors, and visual trackers. The H-Glove is synchronized with the H-FPVHive, enabling precise action localization and capture in unstructured, real-world settings. (3) H-Backpack: A power supply and data storage unit.
• Figure 2: Overview of dataset statistics, including task–scene relationships, action durations, annotation distributions, and word clouds of manipulation target objects and skills. The dataset spans a wide spectrum of real-world scenarios, from industrial to daily-life (e.g., factories, hotels, apartments, supermarkets). For each scenario, it provides task and subtask annotations crucial for instruction-action alignment and task decomposition in VLA models. The chart presents multi-dimensional statistics of these annotations.
• Figure 3: Data annotation samples cross different scenes. The example of human video annotations, including depth, mask, action and task descriptions in four scenarios.
• Figure 4: Language-conditioned Video Generation. When provided with language instructions, two baseline video prediction methods exhibited significant hallucinations without WiYH fine-tuning. However, after fine-tuning on our dataset, they demonstrated a markedly enhanced ability to imagine future states.
• Figure 5: 4D Reconstruction Result. For the pouring wine task, we present the 4DGS reconstruction results across multiple timestamps. The visualizations include the rendered image, estimated depth map, and predicted 4D motion field. The results demonstrate that our WiYH dataset enables clean and accurate 4D reconstruction, even in challenging, dynamic action scenarios.