SMPL-GPTexture: Dual-View 3D Human Texture Estimation using Text-to-Image Generation Models
Mingxiao Tu, Shuchang Ye, Hoijoon Jung, Jinman Kim
TL;DR
SMPL-GPTexture tackles the challenge of creating photorealistic UV textures for 3D human avatars without real dual-view data. It uses structured natural language prompts to synthesize paired front/back views with a text-to-image model, then applies SMPL fitting via Human Mesh Recovery, inverted rasterization to UV space, and diffusion-based inpainting to produce a unified texture. A diffusion-based fusion step completes missing regions, yielding high-resolution textures aligned with the underlying geometry. Comparative experiments across two text-to-image systems reveal a trade-off between texture fidelity and runtime, with GPT4o delivering superior geometric coherence at higher cost, while Infinity enables faster throughput. Overall, the approach offers a privacy-preserving, scalable pathway to avatar texturing with broad implications for virtual try-on, film, and gaming use cases.
Abstract
Generating high-quality, photorealistic textures for 3D human avatars remains a fundamental yet challenging task in computer vision and multimedia field. However, real paired front and back images of human subjects are rarely available with privacy, ethical and cost of acquisition, which restricts scalability of the data. Additionally, learning priors from image inputs using deep generative models, such as GANs or diffusion models, to infer unseen regions such as the human back often leads to artifacts, structural inconsistencies, or loss of fine-grained detail. To address these issues, we present SMPL-GPTexture (skinned multi-person linear model - general purpose Texture), a novel pipeline that takes natural language prompts as input and leverages a state-of-the-art text-to-image generation model to produce paired high-resolution front and back images of a human subject as the starting point for texture estimation. Using the generated paired dual-view images, we first employ a human mesh recovery model to obtain a robust 2D-to-3D SMPL alignment between image pixels and the 3D model's UV coordinates for each views. Second, we use an inverted rasterization technique that explicitly projects the observed colour from the input images into the UV space, thereby producing accurate, complete texture maps. Finally, we apply a diffusion-based inpainting module to fill in the missing regions, and the fusion mechanism then combines these results into a unified full texture map. Extensive experiments shows that our SMPL-GPTexture can generate high resolution texture aligned with user's prompts.