Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Tracking

TL;DR

HOT3D introduces a large-scale, multi-view egocentric dataset for 3D hand–object tracking, collected with real head-mounted devices and anchored by precise marker-based motion capture. It provides 3.7M images, 6DoF ground-truth poses for hands, objects, and cameras, plus rich modalities like eye gaze, SLAM point clouds, and high-fidelity PBR object models. The dataset supports both model-based and model-free tracking, includes onboarding sequences and curated clips, and sets up public ECCV 2024 challenges to standardize benchmarking and foster reproducibility. By enabling diverse, realistic hand–object interactions in AR/VR contexts, HOT3D aims to accelerate advances in perception, reconstruction, and human–machine collaboration.

Abstract

We introduce HOT3D, a publicly available dataset for egocentric hand and object tracking in 3D. The dataset offers over 833 minutes (more than 3.7M images) of multi-view RGB/monochrome image streams showing 19 subjects interacting with 33 diverse rigid objects, multi-modal signals such as eye gaze or scene point clouds, as well as comprehensive ground truth annotations including 3D poses of objects, hands, and cameras, and 3D models of hands and objects. In addition to simple pick-up/observe/put-down actions, HOT3D contains scenarios resembling typical actions in a kitchen, office, and living room environment. The dataset is recorded by two head-mounted devices from Meta: Project Aria, a research prototype of light-weight AR/AI glasses, and Quest 3, a production VR headset sold in millions of units. Ground-truth poses were obtained by a professional motion-capture system using small optical markers attached to hands and objects. Hand annotations are provided in the UmeTrack and MANO formats and objects are represented by 3D meshes with PBR materials obtained by an in-house scanner. We aim to accelerate research on egocentric hand-object interaction by making the HOT3D dataset publicly available and by co-organizing public challenges on the dataset at ECCV 2024. The dataset can be downloaded from the project website: https://facebookresearch.github.io/hot3d/.

Figures (11)

• Figure 1: HOT3D overview. The dataset includes multi-view egocentric image streams from Aria engel2023project and Quest 3 Quest3 annotated with high-quality ground-truth 3D poses and models of hands and objects. Three multi-view frames from Aria are shown on the left, with contours of 3D models of hands and objects in the ground-truth poses in white and green, respectively. Aria also provides 3D point clouds from SLAM (right) and eye gaze information. Hand annotations are provided in the UmeTrack han2022umetrack and MANO mano:romero:tog17 formats, while objects are represented by 3D mesh models from an in-house scanner.
• Figure 2: High-quality 3D mesh models. This image shows a rendering of the 33 object models, demonstrating their quality. The models were obtained by an in-house 3D scanner and include PBR materials, which enable rendering of photo-realistic training images for methods that require it. The collection includes household and office objects of diverse appearance, size, and affordances.
• Figure 3: Sample images from Aria (top) and Quest 3 (bottom). Aria recordings include one RGB and two monochrome image streams, while Quest 3 recordings include two monochrome streams -- only images from one of the multi-view streams are shown. Contours of 3D models of hands and objects in the ground-truth poses are shown in white and green respectively. In addition to simple pick-up/observe/put-down actions, the subjects perform actions that are common in a kitchen, office, and living room. To increase diversity, the lighting, furniture, and decorations in the capture lab were regularly randomized.
• Figure 4: Motion-capture lab. The HOT3D dataset was collected using a motion-capture rig equipped with a few dozens of infrared exocentric OptiTrack cameras and light diffuser panels for illumination variability.
• Figure 5: Distances traveled by HOT3D objects. In total, subjects moved the 33 objects over 13 km. While objects like the keyboard and waffles were mostly resting, the white mug is a true explorer.