Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation

TL;DR

Humanoid Everyday addresses the gap in open-world, embodied humanoid manipulation data by introducing a large-scale, multimodal dataset (260 tasks across 7 categories) with full-body locomotion, human-humanoid interaction, and diverse real-world environments. The authors pair this dataset with a highly efficient teleoperation pipeline and a cloud-based evaluation platform to enable standardized, reproducible benchmarking of policies on real humanoid hardware. They analyze representative imitation-learning methods and demonstrate that pretraining on Humanoid Everyday provides strong priors for Vision-Language-Action models, while also highlighting the remaining challenges in high-dimensional humanoid control, especially in loco-manipulation and precision tasks. Collectively, Humanoid Everyday and the cloud evaluation platform lower barriers to entry for researchers and lay groundwork for more robust, general-purpose humanoid agents in real-world scenarios.

Abstract

From loco-motion to dextrous manipulation, humanoid robots have made remarkable strides in demonstrating complex full-body capabilities. However, the majority of current robot learning datasets and benchmarks mainly focus on stationary robot arms, and the few existing humanoid datasets are either confined to fixed environments or limited in task diversity, often lacking human-humanoid interaction and lower-body locomotion. Moreover, there are a few standardized evaluation platforms for benchmarking learning-based policies on humanoid data. In this work, we present Humanoid Everyday, a large-scale and diverse humanoid manipulation dataset characterized by extensive task variety involving dextrous object manipulation, human-humanoid interaction, locomotion-integrated actions, and more. Leveraging a highly efficient human-supervised teleoperation pipeline, Humanoid Everyday aggregates high-quality multimodal sensory data, including RGB, depth, LiDAR, and tactile inputs, together with natural language annotations, comprising 10.3k trajectories and over 3 million frames of data across 260 tasks across 7 broad categories. In addition, we conduct an analysis of representative policy learning methods on our dataset, providing insights into their strengths and limitations across different task categories. For standardized evaluation, we introduce a cloud-based evaluation platform that allows researchers to seamlessly deploy their policies in our controlled setting and receive performance feedback. By releasing Humanoid Everyday along with our policy learning analysis and a standardized cloud-based evaluation platform, we intend to advance research in general-purpose humanoid manipulation and lay the groundwork for more capable and embodied robotic agents in real-world scenarios. Our dataset, data collection code, and cloud evaluation website are made publicly available on our project website.

Figures (8)

• Figure 1: Humanoid Everyday, a large-scale and diverse humanoid manipulation dataset characterized by extensive task varieties across a wide range of environments.
• Figure 2: Humanoid Everyday Overview. Humanoid Everyday covers 7 distinct categories of humanoid manipulation tasks with rich multimodal information, and provides a cloud-based evaluation platform for standardized policy deployment.
• Figure 3: Data Collection Pipeline. (a) We separate data streaming, data writing, robot control, and IK computation into distinct processes and threads to ensure reliable and efficient data collection. (b) Our pipeline substantially reduces control delay and enhances data collection efficiency.
• Figure 4: Data Distribution. Distribution of tasks and skill categories in the Humanoid Everyday Dataset.
• Figure 5: Evaluation platform. Humanoid Everyday introduces a cloud-based evaluation platform on real-world humanoid setups