HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction

TL;DR

HOI4D presents the first large-scale $4D$ egocentric dataset for category-level human–object interaction, collecting $2.4$M RGB-D frames across more than $4{,}000$ sequences from $9$ participants in $610$ indoor rooms, covering $800$ object instances across $16$ categories. It provides rich annotations including 4D panoptic segmentation, motion segmentation, 3D hand pose, category-level object pose, object meshes, and scene point clouds, enabling three benchmarks: semantic segmentation of 4D dynamic point clouds, category-level object pose tracking, and egocentric hand action segmentation. The dataset is built with a hybrid annotation pipeline combining manual labeling and optimization-based refinement (MANO-based hand pose, CAD mesh reconstruction, differentiable rendering with $SoftRas$, and HONnotate-style pose optimization) to achieve scalable, accurate 4D annotations. Cross-dataset evaluations show HOI4D is more challenging and complementary to existing datasets, driving generalization in real-world 4D HOI tasks and opening new avenues for 4D HOI research and robot learning.

Abstract

We present HOI4D, a large-scale 4D egocentric dataset with rich annotations, to catalyze the research of category-level human-object interaction. HOI4D consists of 2.4M RGB-D egocentric video frames over 4000 sequences collected by 4 participants interacting with 800 different object instances from 16 categories over 610 different indoor rooms. Frame-wise annotations for panoptic segmentation, motion segmentation, 3D hand pose, category-level object pose and hand action have also been provided, together with reconstructed object meshes and scene point clouds. With HOI4D, we establish three benchmarking tasks to promote category-level HOI from 4D visual signals including semantic segmentation of 4D dynamic point cloud sequences, category-level object pose tracking, and egocentric action segmentation with diverse interaction targets. In-depth analysis shows HOI4D poses great challenges to existing methods and produces great research opportunities.

Figures (13)

• Figure 1: Overview of HOI4D: We construct a large-scale 4D egocentric dataset with rich annotation for category-level human-object interaction. Frame-wise annotations for action segmentation(a), motion segmentation(b), panoptic segmentation(d), 3D hand pose and category-level object pose(c) are provided, together with reconstructed object meshes(e) and scene point cloud.
• Figure 2: Data capturing system.we build up a simple head-mounted data capturing suite consists of a bicycle helmet, a Kinect v2 RGB-D sensor, and an Intel RealSense D455 RGB-D sensor.
• Figure 3: Overview of annotation pipeline. Red branch: Given a dynamic RGB-D sequence, we first annotate frame-wise 2D motion segmentation. Then we mask out the moving content and reconstruct a 3D static scene. We manually annotate the reconstructed scene to obtain 3D static scene panoptic segmentation. Finally, the 2D motion segmentation and the 3D static scene panoptic segmentation are merged, resulting in the 4D dynamic scene panoptic segmentation. Blue branch: To obtain 3D hand pose labels, we first annotate a set of hand keypoints on RGB-D frames and then leverage an optimization module to recover the underlying 3D hand. For category-level object poses, we manually fit amodal oriented bounding boxes to objects or object parts in RGB-D frames and further optimize it by leveraging the object mesh. Green branch: We directly annotated fine-grained action labels on the original video.
• Figure 4: Diversity of object categories.
• Figure 5: Examples of interaction task