A Generative Model for Texture Synthesis based on Optimal Transport between Feature Distributions

TL;DR

This work introduces GOTEX, a texture synthesis framework that enforces the distribution of local texture features to match an exemplar via optimal transport. By exploiting the semi-dual formulation of OT, GOTEX casts texture synthesis as a minimax optimization that can be solved with efficient gradient-based methods, enabling both patch-based and deep-feature representations, as well as multi-feature and barycenter extensions. The authors derive gradients for the semi-discrete OT cost, propose multi-scale feature integrations (patches and VGG-based), and demonstrate single-image synthesis, training of a fast feed-forward generator, texture inpainting, and texture interpolation. GOTEX offers a principled alternative to GANs and OT-approximation methods, delivering high-quality textures with controllable statistics and practical applications. The results indicate that multiscale patch distributions provide robust texture synthesis while mixing patch and deep features mitigates artifacts, with potential for fast on-the-fly texture generation and interpolations between textures.

Abstract

We propose GOTEX, a general framework for texture synthesis by optimization that constrains the statistical distribution of local features. While our model encompasses several existing texture models, we focus on the case where the comparison between feature distributions relies on optimal transport distances. We show that the semi-dual formulation of optimal transport allows to control the distribution of various possible features, even if these features live in a high-dimensional space. We then study the resulting minimax optimization problem, which corresponds to a Wasserstein generative model, for which the inner concave maximization problem can be solved with standard stochastic gradient methods. The alternate optimization algorithm is shown to be versatile in terms of applications, features and architecture; in particular it allows to produce high-quality synthesized textures with different sets of features. We analyze the results obtained by constraining the distribution of patches or the distribution of responses to a pre-learned VGG neural network. We show that the patch representation can retrieve the desired textural aspect in a more precise manner. We also provide a detailed comparison with state-of-the-art texture synthesis methods. The GOTEX model based on patch features is also adapted to texture inpainting and texture interpolation. Finally, we show how to use our framework to learn a feed-forward neural network that can synthesize on-the-fly new textures of arbitrary size in a very fast manner. Experimental results and comparisons with the mainstream methods from the literature illustrate the relevance of the generative models learned with GOTEX.

Figures (9)

• Figure 1: Summary of the proposed GOTEX framework. A texture formation model is encoded with a generative model $g_\theta$ and the distribution of texture images is represented through its feature distribution $\mu_\theta$. The objective is then to minimize the optimal transport cost $\mathrm{OT}_c(\mu_\theta,\nu)$ between the current feature distributions $\mu_\theta$ and the (discrete) feature distribution $\nu$ of the example target texture $u_0$. This framework also encompasses the case where the optimization is done on the image pixels by taking the latent distribution as a Dirac (see section \ref{['sec:image-optim']} for details).
• Figure 2: Results of GOTEX (Alg. \ref{['alg:MStexgenGD']}) for image optimization using various combinations of features: patches (GOTEX-patch), VGG (GOTEX-VGG), mixing patches from higher scales with VGG from lower scales (GOTEX-mix), and combining all features (GOTEX-all).
• Figure 3: Both GOTEX-patch and GOTEX-VGG are run for the same $100\times100$ sample (a) with three initial images (b). GOTEX-path produces faithful $200\times 200$ synthesis (c) for any initialization whereas the GOTEX-VGG results (d) tends to produce color inconsistencies and artifacts when the color palette of the initial guess is not close enough to the target image.
• Figure 4: Comparison of exemplar-based texture synthesis methods using patch-based image optimization and various optimal transport approximations. a) displays the exemplar image. b) shows the texture optimization results from Kwatra using coarse-to-fine optimization and nearest-neighbor patch assignment. c) uses instead Optimal Patch Assignment (OPA) Gutierrez_ssvm2017, enforcing the target patch distribution. d) is based on the proposed loss function \ref{['eq:OTpatchloss']} where OT is approximated by Sliced Wasserstein (SW) cost. e) is the proposed semi-discrete approach, solving more accurately the OT problem. See the text for more details.
• Figure 5: As in Figure \ref{['fig:patch_comp']}, we compare the results of the proposed Algorithm \ref{['alg:MStexgenGD']} (GOTEX-VGG) for image optimization based on VGG (already exposed in Fig. \ref{['fig:compfeature']}), as pioneered by gatys_texture_2015 (Gram-VGG) and with the Sliced Wasserstein approximation (SW-VGG) studied in heitz2021sliced.

Theorems & Definitions (11)

• Remark 1
• Remark 2
• Remark 1: Semi-dual formulation santambrogio2015ot
• Remark 2
• proof
• Remark 3
• proof
• Remark 4
• proof
• Remark 5