UV-free Texture Generation with Denoising and Geodesic Heat Diffusions

TL;DR

A fast re-sampling of the mesh spectral properties used during the heat diffusion is introduced and a novel heat-diffusion-based self-attention mechanism is introduced to enable processing of arbitrarily sampled point-cloud textures and ensure long-distance texture consistency.

Abstract

Seams, distortions, wasted UV space, vertex-duplication, and varying resolution over the surface are the most prominent issues of the standard UV-based texturing of meshes. These issues are particularly acute when automatic UV-unwrapping techniques are used. For this reason, instead of generating textures in automatically generated UV-planes like most state-of-the-art methods, we propose to represent textures as coloured point-clouds whose colours are generated by a denoising diffusion probabilistic model constrained to operate on the surface of 3D objects. Our sampling and resolution agnostic generative model heavily relies on heat diffusion over the surface of the meshes for spatial communication between points. To enable processing of arbitrarily sampled point-cloud textures and ensure long-distance texture consistency we introduce a fast re-sampling of the mesh spectral properties used during the heat diffusion and introduce a novel heat-diffusion-based self-attention mechanism. Our code and pre-trained models are available at github.com/simofoti/UV3-TeD.

Figures (20)

• Figure 1: Random textures generated by our method, Uv3-TeD, on the surface of general objects from the Amazon Berkeley Object dataset and of chairs from ShapeNet (miniatures on the shelves).
• Figure 2: Qualitative comparison between point-cloud-textures (top-right halves) and automatically wrapped $\mathrm{UV}$-textures (bottom-left halves). All textures were generated by Point-UV Diffusion in order to showcase some of the most common issues of $\mathrm{UV}$-mapping. Although the method generated good quality textures as point-clouds, projecting them in $\mathrm{UV}$-space introduces significant artefacts.
• Figure 3: Framework of UV3-TeD. Given a mesh $\mathcal{M} = \{ \mathbf{V}, \mathbf{F}\}$ we precompute the proposed mixed Laplacian ($\mathbf{L}^R_\text{mix}$) and its eigendecomposition ($\mathbf{\Lambda}$ and $\mathbf{\Phi}$). During the online sampling we compute a coloured point-cloud $\{ \mathbf{P}, \mathbf{X}\}$ alongside its spectral quantities and other information used by our network (\ref{['fig:architecture']}). In particular, eigenvalues $\mathbf{\Lambda}$, sampled eigenvectors $\mathbf{\Phi}_p$, and approximate mass $M_p$ are used to compute the heat diffusion operations (\ref{['eq:heat_diff']}); the farthest point samples $\text{fps}(\mathbf{P})$ are used in the proposed diffused farthest-sampled attention layers (\ref{['fig:block']}), and the scale invariant heat kernel signatures $sihks$ and slope-adjusted eigenvalues $\mathbf{\Lambda}'$ are used as shape conditioning. UV3-TeD leverages these information to generate coloured point-clouds ($\mathbf{X}_0$) from noise ($\mathbf{X}_T$).
• Figure 4: Attention-enhanced Heat Diffusion block. Three consecutive Diffusion blocks (bottom) inspired by sharp2022diffusionnet and conditioned with a denoising time embedding are combined with a diffused farthest-sampled attention layer (top). The proposed attention, conditioned with local and global shape embeddings ($sihks_e$ and $\mathbf{\Lambda}'_e$), first spreads information to all the points on the surface, before computing a multi-headed self-attention on the features of the farthest samples (red points), and finally spreads them back to all the points with another heat diffusion.
• Figure 5: Heat diffusion on Ted sliced on the belly and on a topologically disconnected birdhouse. Using the mesh LBO prevents heat from spreading to disconnected regions, this is particularly visible on Ted as heat does not spread over the nose, mouth, and legs. Similarly, on the birdhouse heat spreads only on the right-hand side of the roof. Using our mixed LBO formulation heat can spread over the entire shape even in the presence of topological errors and disconnected components.