VIST3A: Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generator

TL;DR

VIST3A tackles text-to-3D generation by stitching a pretrained video-LDM latent space to a pretrained 3D decoder, creating a strong, reusable 3D VAE. It identifies the most transferable stitching point k* and constructs a closed-form linear stitching layer S, followed by fine-tuning with LoRA to closely reproduce the original 3D model outputs. To align generation with the stitched decoder in the latent space, it employs direct reward finetuning with a loss $L_{total} = L_{gen} - r(z_0( heta,c,z_T), c)$, where the reward combines multi-view image quality, 3D representation quality, and 3D consistency, using gradient-stable training. Across multiple video priors and 3D decoders, VIST3A achieves state-of-the-art text-to-3D performance, enabling high-quality 3D Gaussian splats and pointmaps, with strong robustness and data efficiency.

Abstract

The rapid progress of large, pretrained models for both visual content generation and 3D reconstruction opens up new possibilities for text-to-3D generation. Intuitively, one could obtain a formidable 3D scene generator if one were able to combine the power of a modern latent text-to-video model as "generator" with the geometric abilities of a recent (feedforward) 3D reconstruction system as "decoder". We introduce VIST3A, a general framework that does just that, addressing two main challenges. First, the two components must be joined in a way that preserves the rich knowledge encoded in their weights. We revisit model stitching, i.e., we identify the layer in the 3D decoder that best matches the latent representation produced by the text-to-video generator and stitch the two parts together. That operation requires only a small dataset and no labels. Second, the text-to-video generator must be aligned with the stitched 3D decoder, to ensure that the generated latents are decodable into consistent, perceptually convincing 3D scene geometry. To that end, we adapt direct reward finetuning, a popular technique for human preference alignment. We evaluate the proposed VIST3A approach with different video generators and 3D reconstruction models. All tested pairings markedly improve over prior text-to-3D models that output Gaussian splats. Moreover, by choosing a suitable 3D base model, VIST3A also enables high-quality text-to-pointmap generation.

Figures (14)

• Figure 1: Text-to-3D generation with VIST3A. Video models excel at generating latent visual content from text prompts, whereas 3D foundation models shine when it comes to decoding such a latent representation into consistent scene geometry. By stitching a video generator and a 3D reconstruction network together and aligning their latents, we obtain an end-to-end model that produces high-quality Gaussian splats (a) or point maps (b) from text prompts.
• Figure 2: Comparison with existing, LDM-based 3D generators. Instead of training a custom decoder from multi-view 2D latents to 3D outputs, we stitch and align an existing, pretrained 3D reconstruction model.
• Figure 3: VIST3A constructs a 3D VAE through model stitching (top), then aligns it with a generative model via direct reward finetuning (bottom). Stitching repurposes a part of a pretrained 3D vision model as decoder to obtain a 3D VAE. Direct reward finetuning simulates full-trajectory denoising, forcing the generative model to produce 3D-consistent, decodable latents.
• Figure 4: Qualitative results for 3DGS generation. We show samples from T3Bench (top), SceneBench (middle), and DPG-bench (bottom). VIST3A generates realistic and crisp 3D scenes and adheres to intricate details in the prompt.
• Figure 5: MSE and pointmap quality on ETH3D vs. to stitching layer. Lower MSE in the stitching layer correlates with better 3D reconstruction.