Tex4D: Zero-shot 4D Scene Texturing with Video Diffusion Models

TL;DR

Tex4D tackles the challenge of producing temporally and multi-view consistent textures for animated 3D mesh sequences driven by natural language prompts. It fuses a pre-trained video diffusion prior with UV-space latent aggregation across multiple views, and introduces a modified DDIM sampling plus a reference UV blending mechanism to maintain coherence over time and across viewpoints. The method enables zero-shot generation of 4D textures that encode dynamic appearance changes and lighting directly onto textures, demonstrated on diverse mesh sequences with user studies and quantitative metrics showing improvements over baselines. This approach advances practical content creation by allowing texture-rich, view-consistent rendering of animated scenes without additional training data or per-scene optimization, though it bears computational costs and background integration challenges.

Abstract

3D meshes are widely used in computer vision and graphics for their efficiency in animation and minimal memory use, playing a crucial role in movies, games, AR, and VR. However, creating temporally consistent and realistic textures for mesh sequences remains labor-intensive for professional artists. On the other hand, while video diffusion models excel at text-driven video generation, they often lack 3D geometry awareness and struggle with achieving multi-view consistent texturing for 3D meshes. In this work, we present Tex4D, a zero-shot approach that integrates inherent 3D geometry knowledge from mesh sequences with the expressiveness of video diffusion models to produce multi-view and temporally consistent 4D textures. Given an untextured mesh sequence and a text prompt as inputs, our method enhances multi-view consistency by synchronizing the diffusion process across different views through latent aggregation in the UV space. To ensure temporal consistency, we leverage prior knowledge from a conditional video generation model for texture synthesis. However, straightforwardly combining the video diffusion model and the UV texture aggregation leads to blurry results. We analyze the underlying causes and propose a simple yet effective modification to the DDIM sampling process to address this issue. Additionally, we introduce a reference latent texture to strengthen the correlation between frames during the denoising process. To the best of our knowledge, Tex4D is the first method specifically designed for 4D scene texturing. Extensive experiments demonstrate its superiority in producing multi-view and multi-frame consistent videos based on untextured mesh sequences.

Figures (21)

• Figure 1: Tex4D Application. Our synthesized dynamic textures can be easily integrated into graphics pipelines.
• Figure 2: Given an untextured mesh sequence and a text prompt as inputs (Top), Tex4D generates multi-view, dynamic textures. Below, we show renderings of the textured meshes from three views and four timestamps.
• Figure 3: Overview. Given a mesh sequence and a text prompt as inputs, Tex4D generates a UV-parameterized texture sequence that is both globally and temporally consistent. At each diffusion step, latent views are aggregated into UV space, followed by multi-view latent texture diffusion to ensure global consistency. To maintain temporal coherence and address self-occlusions, a Reference UV Blending module is applied at each step. Finally, the latent textures are back-projected and decoded to produce RGB textures for each frame.
• Figure 4: Qualitative Results. Our method generates multi-view consistent dynamic textures with a diverse set of styles and prompts. Zoom in to view the details. More results are provided in the supplementary material.
• Figure 5: Qualitative Comparison with Text-to-4D Methods. Our methods generates multi-view consistent compared with Text-to-4D methods as our approach fully utilizes the geometry information of the meshes.