TexRO: Generating Delicate Textures of 3D Models by Recursive Optimization

TL;DR

TexRO tackles the problem of generating delicate, multiview-consistent textures for a known 3D mesh by optimizing the UV texture $\mathbf{y}$ through a two-stage pipeline. It first selects a minimal, complete set of viewpoints to cover all faces and initializes the UV map from multi-view, depth-guided diffusion; then it recursively optimizes the UV texture at increasing RGB resolutions using an adaptive denoising strategy that reuses existing textures. The method introduces a Set-Cover Problem (SCP)-based optimal viewpoint selection and an interlaced, multi-resolution denoising process, enabling high visual fidelity with efficient runtime. Empirical results on Text2Tex-Data and Cap3D show TexRO surpasses state-of-the-art baselines in FID/KID metrics, qualitative texture quality, and speed, demonstrating strong applicability across diverse 3D models.

Abstract

This paper presents TexRO, a novel method for generating delicate textures of a known 3D mesh by optimizing its UV texture. The key contributions are two-fold. We propose an optimal viewpoint selection strategy, that finds the most miniature set of viewpoints covering all the faces of a mesh. Our viewpoint selection strategy guarantees the completeness of a generated result. We propose a recursive optimization pipeline that optimizes a UV texture at increasing resolutions, with an adaptive denoising method that re-uses existing textures for new texture generation. Through extensive experimentation, we demonstrate the superior performance of TexRO in terms of texture quality, detail preservation, visual consistency, and, notably runtime speed, outperforming other current methods. The broad applicability of TexRO is further confirmed through its successful use on diverse 3D models.

Figures (11)

• Figure 1: The proposed TexRO generates realistic textures for a known 3D mesh based on prompts. Our key contributions include: 1) a novel recursive optimization method that refines UV textures at increasing resolutions using the proposed interlaced denoising module, and 2) an effective viewpoints selection strategy. The proposed TexRO has achieved the fastest texture generation ($\sim 1$ min.) and the highest generation quality (as measured by FID and KID scores) compared to previous studies. Example results are showcased in the figure.
• Figure 2: The outline of the proposed TexRO is illustrated in the figure. It has two stages. In stage 1, it produces an optimal set of viewpoints and generates an initial UV texture for later optimization. In stage 2, it conducts the recursive optimization that optimizes the UV texture at increasing resolutions. The details of the proposed adaptive denoising strategy is illustrated at the top of the figure. $\widehat{\alpha_{1}}$ and $\widehat{\alpha_{1}}$ are two noise schedulers.
• Figure 3: We employ depth-controlled Zero123Plus shi2023zero123plus for our UV texture map initialization. The prompt for generating this example is “a Mercedes-Benz car”.
• Figure 4: The key difference between the proposed adaptive denoising strategy and the straightforward method used in the previous studies chen2023text2textexturepaper. In contrast to the straightforward that generates new textures from pure noise using an image inpainting diffusion model, ours injects noises to refine existing textures. We introduce how $\alpha_{t1}$ and $\alpha_{t2}$ are computed in Sec. \ref{['method:recursiveoptimization']}'s adaptive denoising.
• Figure 5: (a) showcases the outcome of each recursive step in Stage-2, where “initialization” refers to the initial UV texture produced in Stage-1. (b) compares the results from the proposed recursive optimization with those from a single-step optimization performed at the highest resolution ($1552 \times 1552$), defined as the resolution applied in Step 5 of our method.

Theorems & Definitions (2)

• lemma thmcounterlemma
• theorem thmcountertheorem