IL3D: A Large-Scale Indoor Layout Dataset for LLM-Driven 3D Scene Generation

TL;DR

IL3D tackles the need for large-scale, richly annotated indoor-scene data to train LLM-driven 3D layout generation. It introduces a two-part dataset with 27,816 layouts and 29,215 object assets, augmented by instance-level natural language descriptions and multimodal exports, built on USDZ/USDA formats and accessible via a vector retrieval system. Through objective (OOB, OOR, GSR, CLIP-Sim) and subjective (OP, PR, SC, SF, VA) metrics, the study shows supervised fine-tuning on IL3D improves generalization and often surpasses prior datasets, with larger models benefiting more from NL annotations. IL3D's flexible export formats and retrieval-driven workflow enable seamless integration with 3D perception and embodied AI tasks, establishing a benchmark and data resource that advances LLM-guided indoor scene generation and multimodal perception. The work also identifies limitations in scene-level relational descriptions and proposes future work to enrich relational annotations for deeper scene understanding.

Abstract

In this study, we present IL3D, a large-scale dataset meticulously designed for large language model (LLM)-driven 3D scene generation, addressing the pressing demand for diverse, high-quality training data in indoor layout design. Comprising 27,816 indoor layouts across 18 prevalent room types and a library of 29,215 high-fidelity 3D object assets, IL3D is enriched with instance-level natural language annotations to support robust multimodal learning for vision-language tasks. We establish rigorous benchmarks to evaluate LLM-driven scene generation. Experimental results show that supervised fine-tuning (SFT) of LLMs on IL3D significantly improves generalization and surpasses the performance of SFT on other datasets. IL3D offers flexible multimodal data export capabilities, including point clouds, 3D bounding boxes, multiview images, depth maps, normal maps, and semantic masks, enabling seamless adaptation to various visual tasks. As a versatile and robust resource, IL3D significantly advances research in 3D scene generation and embodied intelligence, by providing high-fidelity scene data to support environment perception tasks of embodied agents.

Figures (16)

• Figure 1: Statistics of room type. IL3D comprises 27,816 distinct rooms, consisting of 18 common room types. For existing datasets, we selectively supplemented some missing room types.
• Figure 2: Statistics of object numbers per room. The number of objects in a room is mostly concentrated in the range of 4 to 9. When this range is exceeded, the corresponding number of rooms shows a steady downward trend as the number of objects increases.
• Figure 3: Room Area Statistics. Navigable areas generally exhibit a certain degree of deviation compared with room areas.
• Figure 4: Statistics of object category distribution.
• Figure 5: Asset Retrieval. (a) A 3D asset vector database for object descriptions is constructed based on Qdrant. (b) When generating scenes, LLMs are used to extract object information from text descriptions and retrieve relevant assets.