IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videos

TL;DR

This work introduces IKEA Video Manuals, a dataset that features 3D models of furniture parts, instructional manuals, assembly videos from the Internet, and most importantly, annotations of dense spatio-temporal alignments between these data modalities, essential for a holistic understanding of assembly in 3D space over time.

Abstract

Shape assembly is a ubiquitous task in daily life, integral for constructing complex 3D structures like IKEA furniture. While significant progress has been made in developing autonomous agents for shape assembly, existing datasets have not yet tackled the 4D grounding of assembly instructions in videos, essential for a holistic understanding of assembly in 3D space over time. We introduce IKEA Video Manuals, a dataset that features 3D models of furniture parts, instructional manuals, assembly videos from the Internet, and most importantly, annotations of dense spatio-temporal alignments between these data modalities. To demonstrate the utility of IKEA Video Manuals, we present five applications essential for shape assembly: assembly plan generation, part-conditioned segmentation, part-conditioned pose estimation, video object segmentation, and furniture assembly based on instructional video manuals. For each application, we provide evaluation metrics and baseline methods. Through experiments on our annotated data, we highlight many challenges in grounding assembly instructions in videos to improve shape assembly, including handling occlusions, varying viewpoints, and extended assembly sequences.

Figures (24)

• Figure 1: Dataset Overview. (a) Manual images showing the assembly steps. (b) Video frames from the corresponding assembly videos. Temporal alignment between the video frames and each assembly step is also provided. (c) Segmentation masks for individual parts and sub-assemblies that are being constructed in each frame. When two parts are assembled, their masks are combined. (d) 6-DoF poses for parts and sub-assemblies in each frame. (e) Tracking of individual parts and sub-assemblies across video frames, capturing the frame-by-frame assembly process.
• Figure 2: Dataset Diversity. (a) Examples of 3D furniture models across different categories in our dataset, showing structural and functional variety. (b) Diverse assembly environments from our video collection, demonstrating real-world complexity including different lighting conditions, camera angles, and backgrounds. The diversity of both furniture types and environments presents unique challenges for grounding.
• Figure 3: Dataset Statistics. (a) Distribution of the number of assembly steps in videos. (b) Distribution of the number of sub-assembly steps (substeps) in videos. (c) Distribution of the number annotations in videos.
• Figure 4: Data Collection and Annotation Pipeline. (a) Collecting 3D furniture models, associated assembly manuals and videos. (b) Annotating coarse temporal segmentation of videos into segments showing each assembly step. (c) Tracking identities of 3D parts throughout each video keyframe. (d) Fine-grained temporal segmentation into substeps showing the construction of each sub-assembly. (e) Annotating 2D segmentation masks for parts and sub-assemblies in sampled frames using an interactive interface powered by the SAM model. (f) Estimating camera parameters and 6D part poses in each frame using 2D-3D correspondences, PnP, RANSAC, and manual refinement.
• Figure 5: Example of Hierarchical Assembly Trees. (a) Assembly tree structure derived from the high-level steps in the IKEA manual. (b) More detailed assembly tree structure extracted from the fine-grained substeps annotated in the assembly videos.