Non-Stationary Texture Synthesis by Adversarial Expansion

TL;DR

The paper tackles non-stationary texture synthesis by training a per-exemplar GAN that learns to expand a $k \times k$ texture block to a $2k \times 2k$ block, thereby capturing large-scale structures and spatially varying patterns. The generator is fully convolutional and paired with a PatchGAN discriminator, optimized with adversarial loss alongside $L_1$ and style losses, using a training regime that requires ground-truth blocks from the exemplar. The approach excels at reproducing and extending global structures in highly non-stationary textures and supports texture transfer, while enabling very fast synthesis after training; it also demonstrates diversification capabilities and robustness to extreme expansions. Limitations include training time and border artifacts, with potential improvements via multi-texture training and richer data to enhance generalization and mitigate failure modes.

Abstract

The real world exhibits an abundance of non-stationary textures. Examples include textures with large-scale structures, as well as spatially variant and inhomogeneous textures. While existing example-based texture synthesis methods can cope well with stationary textures, non-stationary textures still pose a considerable challenge, which remains unresolved. In this paper, we propose a new approach for example-based non-stationary texture synthesis. Our approach uses a generative adversarial network (GAN), trained to double the spatial extent of texture blocks extracted from a specific texture exemplar. Once trained, the fully convolutional generator is able to expand the size of the entire exemplar, as well as of any of its sub-blocks. We demonstrate that this conceptually simple approach is highly effective for capturing large-scale structures, as well as other non-stationary attributes of the input exemplar. As a result, it can cope with challenging textures, which, to our knowledge, no other existing method can handle.

Figures (16)

• Figure 1: Four non-stationary textures (middle) and our synthesis results (left and right). Again, the global structure present in the input exemplars is preserved and extended in our results. For example, in the leaf texture, additional veins appear, whose orientation and spacing is consistent with the input.
• Figure 2: Our method can also be successfully applied to stationary, regular, near-regular, or stochastic textures.
• Figure 3: Method overview. The generator learns to expand $k \times k$ texture blocks into $2k \times 2k$ ones using a combination of adversarial loss, $L_1$ loss and style loss.
• Figure 4: Architecture of our generator and discriminator. The number of feature channels and the spatial resolution of feature maps are respectively specified on the top of and under each block, while the kernel sizes are specified in the central yellow regions.
• Figure 5: Visualization of feature maps output by the middle part of our generator. Besides the intermediate results of residual blocks (from resblock_1 to resblock_6) we also visualize the final output of encoding stage (relu), and the feature map output by the first convolution layer of the decoder (conv). While the leaf (top), bricks (middle) and wood ring (bottom) textures have very different large-scale structures, it may be observed that all of the new structures emerge in the course of the residual block chain. The creation of new structures is typically complete before the end of the chain, as evidenced by the similarity between the resblock_5 and resblock_6 feature maps.