Local Padding in Patch-Based GANs for Seamless Infinite-Sized Texture Synthesis

TL;DR

This paper proposes a novel approach for generating stochastic texture images at large arbitrary sizes using GANs based on patch-by-patch generation that uses local padding in the generator that shares border features between the generated patches; providing positional context and ensuring consistency at the boundaries.

Abstract

Texture models based on Generative Adversarial Networks (GANs) use zero-padding to implicitly encode positional information of the image features. However, when extending the spatial input to generate images at large sizes, zero-padding can often lead to degradation in image quality due to the incorrect positional information at the center of the image. Moreover, zero-padding can limit the diversity within the generated large images. In this paper, we propose a novel approach for generating stochastic texture images at large arbitrary sizes using GANs based on patch-by-patch generation. Instead of zero-padding, the model uses \textit{local padding} in the generator that shares border features between the generated patches; providing positional context and ensuring consistency at the boundaries. The proposed models are trainable on a single texture image and have a constant GPU scalability with respect to the output image size, and hence can generate images of infinite sizes. We show in the experiments that our method has a significant advancement beyond existing GANs-based texture models in terms of the quality and diversity of the generated textures. Furthermore, the implementation of local padding in the state-of-the-art super-resolution models effectively eliminates tiling artifacts enabling large-scale super-resolution. Our code is available at \url{https://github.com/ai4netzero/Infinite_Texture_GANs}.

Figures (11)

• Figure 1: Overview of the training process for the proposed method. An image patch is cropped randomly from the single texture image and is passed to the discriminator. The generator takes $3\times 3$ random spatial inputs $z_i$, each of size $4 \times 4$, to generate, in parallel, $3\times3$ small-size patches $x_i = G(z_i) \, \, \forall \, i \in 1 \dots 9$ that are passed to an assembling function $F$ to form a larger image $X = F(x_1,\dots,x_9)$. The assembled image is then passed to the discriminator for evaluation. The blue arrows indicate the real patches path while the red arrows indicate the generated patches path.
• Figure 2: Comparison between local padding and zero-padding applied to a batch of 3 inputs, each of size $1 \times 3$. The output image patches from the 3 inputs are assembled together horizontally. a) In local padding, the features of neighbouring patches are used to pad each input before passing to a convolutional layer. Replicate padding is used for the border pixels in the outer patches as there is no neighbouring content. b) Zero-padding simply pads each input with zero values, leading to border inconsistency when assembling the patches together.
• Figure 3: Scaling in the horizontal direction: (Top) Two images $X_1$ and $X_2$, each composed of $1\times 3$ generated patches, where the rightmost patch in $X_1$ is regenerated to match the interior patches of $X_2$ (i.e., $x_3$ is regenerated to be consistent with $x_4$). $X_3$ is then formed by concatenating $X_2$ with $X_1$ patches after dropping $x_3$. (Bottom) Two examples of generated texture demonstrate the horizontal scaling process, where the rightmost part of the first column has been regenerated with local padding to match the parts with the images in the second column.
• Figure 4: The architecture of the generator network. The generator takes a spatial noise input $z$ and successively upsamples it to generate an image patch $x = G(z)$. Each $3\times3$ convolutional layer is preceded by local padding where the features are padded with content from neighbouring patches. Batch normalization and upsampling operations are performed on each patch independently.
• Figure 5: Examples of texture images generated by the proposed method (right column) given the source texture (left column). From top to bottom, the image names are 241, 10, 12, 34 and 73.