TexPro: Text-guided PBR Texturing with Procedural Material Modeling

TL;DR

TexPro tackles the challenge of turning natural language prompts into photorealistic, relightable textures for 3D meshes by leveraging a three-stage pipeline: generating multi-view reference images, initializing a procedural material graph with guidance from a GPT-4V-based material agent, and optimizing both materials and lighting through differentiable rendering. By using procedural material modeling, TexPro achieves relightable, high-fidelity textures with resolution independence and editability, addressing limitations of prior RGB-texture approaches that rely on baked lighting. A novel combination of adaptive view sampling, Matcher-based mask alignment, and part-aware material reasoning enables robust initialization and view-consistent optimization. Experimental results on multiple 3D datasets demonstrate superior quality over baselines and clear relighting capabilities, with ablations validating the contributions of the material agent and alignment strategy.

Abstract

In this paper, we present TexPro, a novel method for high-fidelity material generation for input 3D meshes given text prompts. Unlike existing text-conditioned texture generation methods that typically generate RGB textures with baked lighting, TexPro is able to produce diverse texture maps via procedural material modeling, which enables physically-based rendering, relighting, and additional benefits inherent to procedural materials. Specifically, we first generate multi-view reference images given the input textual prompt by employing the latest text-to-image model. We then derive texture maps through rendering-based optimization with recent differentiable procedural materials. To this end, we design several techniques to handle the misalignment between the generated multi-view images and 3D meshes, and introduce a novel material agent that enhances material classification and matching by exploring both part-level understanding and object-aware material reasoning. Experiments demonstrate the superiority of the proposed method over existing SOTAs, and its capability of relighting.

Figures (13)

• Figure 1: Texturing Results. We introduce TexPro, a text-driven PBR material generation method for photorealistic and relightable rendering.
• Figure 2: Overview of TexPro.
• Figure 3: Material Agent. An example of a chair backrest is shown here. Providing both the image prompt and text prompt to our defined material agent allows it to predict all possible material categories in order of likelihood.
• Figure 4: Agent Response Examples. Left: prompt images provided to the material agent. Right: responses returned by the agent.
• Figure 5: Material Matching Example. Above: the cropped image $C_{ij}$ and the cropped mask $M_{ij}'$ obtained through the bounding box. Below: the masked exemplar rectangles $M_{ij}'*E_k^s$ randomly sampled on the rendered exemplars at different scales of a fabric material.