MetaScenes: Towards Automated Replica Creation for Real-world 3D Scans

TL;DR

MetaScenes addresses the scalability gap in high-quality 3D scenes for Embodied AI by introducing a large-scale, simulatable dataset derived from real-world scans and a multimodal asset-replacement pipeline, Scan2Sim. It details a data-collection and annotation workflow, physics-based optimization, and two challenging benchmarks—Micro-Scene Synthesis and cross-domain VLN—to validate realism and transfer. Experiments demonstrate improved asset diversity, automatic replica creation, realistic micro-scenes, and better generalization in navigation tasks, enabling practical sim-to-real applications. The work provides scalable pathways for realistic scene generation and establishes strong baselines for automated replica creation in EAI research.

Abstract

Embodied AI (EAI) research requires high-quality, diverse 3D scenes to effectively support skill acquisition, sim-to-real transfer, and generalization. Achieving these quality standards, however, necessitates the precise replication of real-world object diversity. Existing datasets demonstrate that this process heavily relies on artist-driven designs, which demand substantial human effort and present significant scalability challenges. To scalably produce realistic and interactive 3D scenes, we first present MetaScenes, a large-scale, simulatable 3D scene dataset constructed from real-world scans, which includes 15366 objects spanning 831 fine-grained categories. Then, we introduce Scan2Sim, a robust multi-modal alignment model, which enables the automated, high-quality replacement of assets, thereby eliminating the reliance on artist-driven designs for scaling 3D scenes. We further propose two benchmarks to evaluate MetaScenes: a detailed scene synthesis task focused on small item layouts for robotic manipulation and a domain transfer task in vision-and-language navigation (VLN) to validate cross-domain transfer. Results confirm MetaScene's potential to enhance EAI by supporting more generalizable agent learning and sim-to-real applications, introducing new possibilities for EAI research. Project website: https://meta-scenes.github.io/.

Figures (18)

• Figure 1: Overview of MetaScenes, a large-scale simulatable 3D scene dataset constructed by replacing objects in real-world 3D scans with realistic and high-quality object assets retrieved or reconstructed from diverse sources.
• Figure 2: The construction of MetaScenes.MetaScenes is composed of three sequential steps: (i) Collection, where we gather diverse 3D asset candidates for each real-world object in the scan; (ii) Annotation, where annotators rank and select the best-matching 3D asset for each object based on visual similarity and geometric fit; and (iii) Optimization, where selected assets undergo post-processing and global optimization to ensure full interactivity and physical plausibility in simulation environments.
• Figure 3: Overview of our optimal asset retrieval model. We provide a multi-modal alignment model to retrieve the best asset from candidates.
• Figure 4: Automated replica creation. We visualize the optimal asset selection results in MetaScenes (left), and a digital replica automatically created via Scan2Sim on ScanNet++, before (top) and after physics-based optimization (bottom).
• Figure 5: Micro-Scene Synthesis results. We visualize the generated results in a) Object-Level with the generated small objects given the large furniture. b) Room-Level by first generating the room layout, and then generating small objects atop the large objects.