DreamLifting: A Plug-in Module Lifting MV Diffusion Models for 3D Asset Generation

Abstract

The labor- and experience-intensive creation of 3D assets with physically based rendering (PBR) materials demands an autonomous 3D asset creation pipeline. However, most existing 3D generation methods focus on geometry modeling, either baking textures into simple vertex colors or leaving texture synthesis to post-processing with image diffusion models. To achieve end-to-end PBR-ready 3D asset generation, we present Lightweight Gaussian Asset Adapter (LGAA), a novel framework that unifies the modeling of geometry and PBR materials by exploiting multi-view (MV) diffusion priors from a novel perspective. The LGAA features a modular design with three components. Specifically, the LGAA Wrapper reuses and adapts network layers from MV diffusion models, which encapsulate knowledge acquired from billions of images, enabling better convergence in a data-efficient manner. To incorporate multiple diffusion priors for geometry and PBR synthesis, the LGAA Switcher aligns multiple LGAA Wrapper layers encapsulating different knowledge. Then, a tamed variational autoencoder (VAE), termed LGAA Decoder, is designed to predict 2D Gaussian Splatting (2DGS) with PBR channels. Finally, we introduce a dedicated post-processing procedure to effectively extract high-quality, relightable mesh assets from the resulting 2DGS. Extensive quantitative and qualitative experiments demonstrate the superior performance of LGAA with both text- and image-conditioned MV diffusion models. Additionally, the modular design enables flexible incorporation of multiple diffusion priors, and the knowledge-preserving scheme effectively preseves the 2D priors learned on massive image dataset, which leads to data efficient finetuning to lift the MV diffuison models for 3D generation with merely 69k multi-view instances.

Figures (9)

• Figure 1: Our pipeline possesses the capability of generating diverse, PBR-ready 3D assets from either text prompts or image conditions. The synthesized assets are fully relightable with accurate PBR materials; for example, the wooden owl instance exhibits diffuse color changes under different environment maps, while the specular dog instance successfully reflects its surroundings. These results highlight the usability of the generated 3D assets.
• Figure 2: A toy experiment showing that intermediate features from the pretrained MV diffusion are geometry-aware and can be lifted to an explicit 3D representation with minimal adaptation. We construct the adapter by reusing the upsampling blocks from the Unet of MVDream v2.1, modify the output head to predict pixel-aligned 3DGS, and finetune the adapter on a subset of G-Objaverse deitke2023objaverse. This indicates that MV diffusion models contain strong geometry cues that can be lifted to an explicit 3D representation with minimal adaptation. Details about the toy experiment can be found in the Supplementary.
• Figure 3: Overall of the 3D asset generation pipeline. We propose the Lightweight Gaussian Asset Adapter (LGAA), which is composed of three components: (a) LGAA Wrapper (LGAA-W), (b) LGAA Switcher (LGAA-S), and (c) LGAA Decoder (LGAA-D), where $\mathcal{ZC}$ indicates zero-initialized convolutional layers. In (a), we adapt the priors for 3D generation by wrapping pre-trained layers with $\mathcal{ZC}$ layers, where $\bm{X}$ indicates input feature maps, $\bm{Y}$ are feature maps from MV diffusion models, and $\bm{Y}'$ are output maps from the LGAA-W. We wrap layers from RGB DM (diffusion model) to construct the geometry branch, and use those from PBR DM for the appearance branch. In our implementations, we use MVDream/ImageDream/DreamView as the RGB DMs, and use IDArb as the PBR DM. In (b), we align the geometry and appearance branches with $\mathcal{ZC}$ layers, which progressively develop bidirectional information exchange paths during training. In (c), the LGAA-D upsample the feature maps, decodes PBR channels of albedo $\bm{a} \in \mathbb{R}^3$, metallic $m \in \mathbb{R}$, and roughness $r \in \mathbb{R}$ for the appearance branch, and decodes Gaussian parameters of 3D position $\bm{\mu} \in \mathbb{R}^3$, rotation quaternion $\bm{q} \in \mathbb{R}^4$, scaling vector $\bm{s} \in \mathbb{R}^2$, opacity $o \in \mathbb{R}$ and color $\bm{c} \in \mathbb{R}^3$ for the geometry branch. During the training procedure, we tie the G-buffer maps with the RGB images via image-based deferred shading. In inference, we extract the 3D mesh with PBR material maps from the Gaussian Splat assets with carefully designed post-processing.
• Figure 4: Visual comparisons of text-conditioned 3D asset generation methods. For LGM and LaRa, we use MVDream 2.1 to generate four input views. '*' refers to the non-publicly available commercial software. 'T + TexG' refers to the two stage pipeline where geometry is generated via TRELLIS and PBR materials are produced by TexGaussian.
• Figure 5: Visual comparisons of image-conditioned 3D asset generation methods. For LGM and LaRa, we use ImageDream to generate four input views. '*' refers to the non-publicly available commercial software. 'T + TexG' refers to the two stage pipeline where geometry is generated via TRELLIS and PBR materials are produced by TexGaussian.