UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding

TL;DR

UniUGG is introduced, the first unified understanding and generation framework for 3D modalities that employs an LLM to comprehend and decode sentences and 3D representations and proposes a spatial decoder leveraging a latent diffusion model to generate high-quality 3D representations.

Abstract

Despite the impressive progress on understanding and generating images shown by the recent unified architectures, the integration of 3D tasks remains challenging and largely unexplored. In this paper, we introduce UniUGG, the first unified understanding and generation framework for 3D modalities. Our unified framework employs an LLM to comprehend and decode sentences and 3D representations. At its core, we propose a spatial decoder leveraging a latent diffusion model to generate high-quality 3D representations. This allows for the generation and imagination of 3D scenes based on a reference image and an arbitrary view transformation, while remaining supports for spatial visual question answering (VQA) tasks. Additionally, we propose a geometric-semantic learning strategy to pretrain the vision encoder. This design jointly captures the input's semantic and geometric cues, enhancing both spatial understanding and generation. Extensive experimental results demonstrate the superiority of our method in visual representation, spatial understanding, and 3D generation.

Figures (15)

• Figure 1: We introduce UniUGG, the first unified framework for spatial understanding and generation. (A) UniUGG supports spatial-level VQA and generates geometrically consistent 3D scenes. (B) Given a reference image, it can creatively generate 3D variations and describe them accurately. (C) UniUGG outperforms baselines in both spatial understanding and generation, with our specially tuned vision encoder excelling in downstream tasks.
• Figure 2: Pipeline overview of UniUGG. The left illustrates the three-stage training process, and the right shows the inference pipeline for spatial reasoning and 3D generation.
• Figure 3: Overview of our encoder pretraining pipeline. (a) During semantic guiding, our student encoder learns to mimic the teacher's visual representations. (b) In spatial representation learning, the spatial decoder jointly refines predictions using information from both views.
• Figure 4: Overview of UniUGG training and inferencing. (a) In the latent token learning stage (stage 2), visual representation is compressed using the Spatial-VAE, while the spatial decoder is linked for fine-tuning. (b) In the unified learning stage (stage 3), the reference image’s visual representation and view transformation are input to an LLM, which outputs conditional features for noise prediction on latent token. The LLM also performs VQA-related training to maintain its understanding capability. During inferencing, UniUGG generates the visual representation of the target view, which, together with the reference representation, is decoded into the 3D scene.
• Figure 5: Qualitative ablation on 2D projected views from 3D generation. Our UniUGG, including the geometric-semantic encoder, Spatial-VAE, and associated training paradigm, leads to noticeably better generation results, in terms of geometric accuracy and color consistency.