Geometry-Aware Texture Generation for 3D Head Modeling with Artist-driven Control

TL;DR

The paper tackles the challenge of producing realistic 3D head assets with precise artistic control. It introduces a geometry-aware texture synthesis framework that couples a GNN-based geometry generator with a texture generator that yields a skin-tone map $\mathcal{M}$ and a high-frequency detail map $\mathcal{H}$, both conditioned on geometry, plus a data-driven skin tone mapper from melanin-hemoglobin space to color space. The approach enables independent manipulation of geometry, skin tone (via a scalar $\alpha$), and fine-grained details, with edits propagating coherently across all texture maps and up to 4K resolution. Empirical results show improved realism over baselines, effective skin-tone editing, and coherent detail editing, supporting streamlined artistic workflows for diverse virtual characters and offering a path toward broader applications such as dynamic expressions and full-body avatars.

Abstract

Creating realistic 3D head assets for virtual characters that match a precise artistic vision remains labor-intensive. We present a novel framework that streamlines this process by providing artists with intuitive control over generated 3D heads. Our approach uses a geometry-aware texture synthesis pipeline that learns correlations between head geometry and skin texture maps across different demographics. The framework offers three levels of artistic control: manipulation of overall head geometry, adjustment of skin tone while preserving facial characteristics, and fine-grained editing of details such as wrinkles or facial hair. Our pipeline allows artists to make edits to a single texture map using familiar tools, with our system automatically propagating these changes coherently across the remaining texture maps needed for realistic rendering. Experiments demonstrate that our method produces diverse results with clean geometries. We showcase practical applications focusing on intuitive control for artists, including skin tone adjustments and simplified editing workflows for adding age-related details or removing unwanted features from scanned models. This integrated approach aims to streamline the artistic workflow in virtual character creation.

Figures (9)

• Figure 1: Overview of the proposed pipeline. The geometry generator $\mathcal{F}$ generates a 3D face mesh based on user input (age, gender, ethnicity). Then, the texture generator $G$ produces two skin maps: a melanin map $\mathcal{M}$ and a detail map $\mathcal{H}$. Artists can edit $\mathcal{H}$ using their preferred tools (block $E$). The melanin map $\mathcal{M}$ is converted into a color map $\mathcal{A}$ using the translation network $G\mathcal{A}$, with skin tone adjustable via the slider $\alpha$. The network $G_\mathcal{C}$ then generates the final skin reflectance map $\mathcal{C}$, incorporating any changes to $\mathcal{H}$. Finally, $G_\epsilon$ decomposes the reflectance into specular ($\mathcal{S}$) and normal ($\mathcal{N}$) maps, which are used for physically-based rendering (PBR).
• Figure 2: Geometry-aware texture generation. Given $\textbf{z}_g$ that defines a head's geometry, $G$ estimates skin tone control maps $\mathcal{M}$ and high-frequency skin details $\mathcal{H}$.
• Figure 3: Top: Samples from our model. Bottom: Samples from the baseline
• Figure 4: Top: Randomly generated samples from our model. Bottom: Closest sample in the dataset to the generated sample.
• Figure 5: A sample from the training set: mesh vertices $V$, color-map $\mathcal{C}$, specular map $\mathcal{S}$, Normal map $\mathcal{N}$, high-frequency map $\mathcal{H}$, skin-color map $\mathcal{A}$, skin tone control map $\mathcal{M}$.