Compositional Neural Textures

TL;DR

This work introduces a fully unsupervised approach for representing textures using a compositional neural model that captures individual textons, and represents each texton as a 2D Gaussian function whose spatial support approximates its shape, and an associated feature that encodes its detailed appearance.

Abstract

Texture plays a vital role in enhancing visual richness in both real photographs and computer-generated imagery. However, the process of editing textures often involves laborious and repetitive manual adjustments of textons, which are the recurring local patterns that characterize textures. This work introduces a fully unsupervised approach for representing textures using a compositional neural model that captures individual textons. We represent each texton as a 2D Gaussian function whose spatial support approximates its shape, and an associated feature that encodes its detailed appearance. By modeling a texture as a discrete composition of Gaussian textons, the representation offers both expressiveness and ease of editing. Textures can be edited by modifying the compositional Gaussians within the latent space, and new textures can be efficiently synthesized by feeding the modified Gaussians through a generator network in a feed-forward manner. This approach enables a wide range of applications, including transferring appearance from an image texture to another image, diversifying textures,texture interpolation, revealing/modifying texture variations, edit propagation, texture animation, and direct texton manipulation. The proposed approach contributes to advancing texture analysis, modeling, and editing techniques, and opens up new possibilities for creating visually appealing images with controllable textures.

Figures (12)

• Figure 1: Encoder mapping an image into latent Gaussians. The encoder (\ref{['sec:method:encode']}) comprises a FC image encoder $\mathcal{E}_{FC}$, a segmentation network $\mathcal{E}_s$, and a Gaussian parameter estimation layer $\mathcal{E}_g$. $\mathcal{E}_{FC}$ calculates an appearance feature map $\mathbf{F}_a$ and direction map $\mathbf{V}$, which, along with segmentations $\mathbf{S}$ generated from $\mathcal{E}_s$, are used to compute latent Gaussians $\{\textbf{g}_{}\}$. Entropy $L_{E}$, compactness $L_{C}$ and consistency $L_{Csis}$ losses (\ref{['sec:method:unsupervised']}) are applied to $\mathbf{S}$ to inject desired properties (\ref{['sec:method:properties']}) into textons.
• Figure 2: Architecture of the proposed method. Our network is composed of three branches: (\ref{['sec:method:encode', 'sec:gaussian_splatting', 'sec:method:decode']}), texton and structure-appearance (\ref{['sec:method:unsupervised']}).
• Figure 3: Texture diversification. Given the input \ref{['fig:diverse_texture_synthesis_main:exemplar']}, we can generate different versions \ref{['fig:diverse_texture_synthesis_main:output1']}\ref{['fig:diverse_texture_synthesis_main:output2']}\ref{['fig:diverse_texture_synthesis_main:output3']} of the same texture by randomly reshuffling appearance features in latent Gaussians. \ref{['fig:diverse_texture_synthesis_main:exemplar']}$\copyright$Tandem Stock (stock.adobe.com).
• Figure 4: Texture transfer. Given structure-providing \ref{['fig:qualitative_texture_transfer:structure']} and appearance-providing \ref{['fig:qualitative_texture_transfer:appearance']} textures, the output \ref{['fig:qualitative_texture_transfer:output']} combines the structure from one texture and appearance from the other. We also visualize the segmentation masks \ref{['fig:qualitative_texture_transfer:structure_overlay']} of the structure-providing inputs. \ref{['fig:qualitative_texture_transfer:structure']}: (top) $\copyright$Westend61, (bottom) $\copyright$ADDICTIVE STOCK; \ref{['fig:qualitative_texture_transfer:appearance']}: (top) $\copyright$Liliya Rodnikova/Stocksy, (bottom) $\copyright$Elena Saurius&Dani Rex/Stocksy.
• Figure 5: Stylizing images by transferring textures. Our approach is adaptable to controllable image stylization. \ref{['fig:style_transfer:plain']} shows plain stylization results without controls. \ref{['fig:style_transfer:spatial']} and \ref{['fig:style_transfer:scale']} demonstrate spatial and scale controllability of our image stylization method. \ref{['fig:style_transfer:plain']} (left) $\copyright$Leonardo da Vinci; \ref{['fig:style_transfer:spatial']}(middle) $\copyright$iggii, \ref{['fig:style_transfer:scale']}(left) $\copyright$Chris Curry (unsplash.com).