MetaFind: Scene-Aware 3D Asset Retrieval for Coherent Metaverse Scene Generation

TL;DR

MetaFind tackles the problem of incoherent 3D asset retrieval in large repositories by introducing a layout-aware, multimodal retrieval framework. It uses a dual-tower architecture based on ULIP-2 and an Equivariant Spatial-Semantic Graph Neural Network (ESSGNN) to encode current scene layout and condition asset retrieval on spatial, semantic, and stylistic context, while preserving SE(3) equivariance. The training proceeds in two stages—cross-modal alignment and layout-aware fine-tuning—with a bidirectional contrastive objective and stochastic modality masking to handle missing inputs. Empirical results on Objaverse-LVIS and ProcTHOR show improved scene coherence and realism, especially when ESSGNN is active, and ablations highlight the importance of layout conditioning, fusion strategy, and modality robustness. The work enables robust, iterative scene construction in the metaverse, facilitating coherent composition of 3D assets under complex spatial constraints and partial inputs, with practical implications for scalable, context-aware virtual environment generation.

Abstract

We present MetaFind, a scene-aware tri-modal compositional retrieval framework designed to enhance scene generation in the metaverse by retrieving 3D assets from large-scale repositories. MetaFind addresses two core challenges: (i) inconsistent asset retrieval that overlooks spatial, semantic, and stylistic constraints, and (ii) the absence of a standardized retrieval paradigm specifically tailored for 3D asset retrieval, as existing approaches mainly rely on general-purpose 3D shape representation models. Our key innovation is a flexible retrieval mechanism that supports arbitrary combinations of text, image, and 3D modalities as queries, enhancing spatial reasoning and style consistency by jointly modeling object-level features (including appearance) and scene-level layout structures. Methodologically, MetaFind introduces a plug-and-play equivariant layout encoder ESSGNN that captures spatial relationships and object appearance features, ensuring retrieved 3D assets are contextually and stylistically coherent with the existing scene, regardless of coordinate frame transformations. The framework supports iterative scene construction by continuously adapting retrieval results to current scene updates. Empirical evaluations demonstrate the improved spatial and stylistic consistency of MetaFind in various retrieval tasks compared to baseline methods.

Figures (4)

• Figure 1: Overall framework. MetaFind adopts a dual-tower design where both the user query and candidate assets are encoded using the ULIP-2 backbone. On the query side, we incorporate a plug-and-play ESSGNN module that encodes the current scene layout into a structured scene graph, which captures spatial relationships and object attributes. The user's input—text, image, point cloud, or any combination—is processed by ULIP-2 and fused with the scene context embedding from the ESSGNN to produce a layout-aware query representation. On the asset side, each 3D asset in the repository is pre-encoded independently by ULIP-2 into a fixed vector. At retrieval time, the similarity between the layout-aware query embedding and the precomputed asset embeddings is computed, and the top-matching asset is selected to be inserted into the scene.
• Figure 2: Data preparation pipeline. At the asset level (top), each 3D object from Objaverse-LVIS is rendered from multiple orthogonal views and passed through a VLM to generate structured, detailed annotations, capturing attributes such as category, dimensions, materials, and spatial placement constraints. At the scene-level (bottom), functional extraction is performed on generated rooms from the ProcTHOR, resulting in relational scene graphs encoding the spatial and semantic relationships between placed objects, enabling layout-aware retrieval capabilities in MetaFind.
• Figure 3: Visual comparison of scene generation with and without the ESSGNN encoder across two room descriptions. Room 1 — "A classical-style lounge for group leisure and conversation"; Room 2 — “An aged archive room for research and consultation”
• Figure : Room 1 Description: A classical-style lounge for group leisure and conversation