Enhancing Texture Generation with High-Fidelity Using Advanced Texture Priors

TL;DR

The work addresses the problem of texture aliasing and noise that arise when 3D textures are generated after mesh simplification. It introduces a two-stage framework that first reconstructs geometry with multi-view consistency and depth supervision, then performs high-fidelity texture restoration using the rough input as an initial texture and a self-supervised high/low resolution strategy. Key contributions include cross-view diffusion for consistent geometry, depth-guided rendering, and a gradient-based texture optimization that separates keep and update regions to maintain fidelity while updating textures. The approach reduces noise, seams, and misalignment in high-resolution textures and demonstrates superior restoration quality and efficiency on decimated meshes, with practical implications for real-time rendering pipelines.

Abstract

The recent advancements in 2D generation technology have sparked a widespread discussion on using 2D priors for 3D shape and texture content generation. However, these methods often overlook the subsequent user operations, such as texture aliasing and blurring that occur when the user acquires the 3D model and simplifies its structure. Traditional graphics methods partially alleviate this issue, but recent texture synthesis technologies fail to ensure consistency with the original model's appearance and cannot achieve high-fidelity restoration. Moreover, background noise frequently arises in high-resolution texture synthesis, limiting the practical application of these generation technologies.In this work, we propose a high-resolution and high-fidelity texture restoration technique that uses the rough texture as the initial input to enhance the consistency between the synthetic texture and the initial texture, thereby overcoming the issues of aliasing and blurring caused by the user's structure simplification operations. Additionally, we introduce a background noise smoothing technique based on a self-supervised scheme to address the noise problem in current high-resolution texture synthesis schemes. Our approach enables high-resolution texture synthesis, paving the way for high-definition and high-detail texture synthesis technology. Experiments demonstrate that our scheme outperforms currently known schemes in high-fidelity texture recovery under high-resolution conditions.

Figures (6)

• Figure 1: The overall framework of our method comprises two stages. In the first stage of geometric restoration, we generate RGB images with multi-view consistency and corresponding normal maps for a single input image, incorporating depth supervision using the Depth anything model. This stage primarily serves to provide users with textured meshes for structural simplification, is not our focus. In the second stage, we begin by rendering the rough texture post-user operation as the initial input. We then generate an RGB image using the Depth2Img model and project it onto the texture map for gradient optimization. During this projection, we produce both low-resolution and high-resolution textures for self-supervision, effectively eliminating noise and "point gaps" in the high-resolution texture map.
• Figure 2: We show the NeRF rendering outcomes from a single-view input; the final rendering quality hinges on the efficacy of generating new views with view consistency.
• Figure 3: We demonstrate the comparison of texture synthesis effects under high-resolution conditions (4K). The textured mesh, obtained after structural simplification in the first stage, serves as the initial input for our method, whereas other methods start from a mere mesh template. Owing to this initial input, our approach exhibits fewer multi-face Janus problem and achieves a rendering effect with greater similarity to the initial texture.
• Figure 4: We present the rendering results of Wonder3D with and without depth supervision. Thanks to depth supervision, our method suppresses incorrect protrusions.
• Figure 5: We show the TEXTure synthesis results in high resolution, from left to right: a)TEXTure texture synthesis. b) Texture texture synthesis with initial input. c) our method. d)SyncMVD texture synthesis results.