Conditional Image Generation and Manipulation for User-Specified Content

TL;DR

The paper tackles the challenge of producing user-specified facial images by integrating text-conditioned high-resolution image synthesis with semantic manipulation in latent space. It introduces textStyleGAN, a conditional StyleGAN variant that uses attentive guidance and cross-modal losses to achieve semantically faithful, high-quality outputs, and CelebTD-HQ to ground descriptions in facial imagery. It further presents a latent-space manipulation method that learns semantic directions to edit attributes like smile, gender, and age without retraining the generator. The results show competitive generation metrics and effective attribute editing, along with artifact reduction, supporting practical applications in content creation such as facial stock photography.

Abstract

In recent years, Generative Adversarial Networks (GANs) have improved steadily towards generating increasingly impressive real-world images. It is useful to steer the image generation process for purposes such as content creation. This can be done by conditioning the model on additional information. However, when conditioning on additional information, there still exists a large set of images that agree with a particular conditioning. This makes it unlikely that the generated image is exactly as envisioned by a user, which is problematic for practical content creation scenarios such as generating facial composites or stock photos. To solve this problem, we propose a single pipeline for text-to-image generation and manipulation. In the first part of our pipeline we introduce textStyleGAN, a model that is conditioned on text. In the second part of our pipeline we make use of the pre-trained weights of textStyleGAN to perform semantic facial image manipulation. The approach works by finding semantic directions in latent space. We show that this method can be used to manipulate facial images for a wide range of attributes. Finally, we introduce the CelebTD-HQ dataset, an extension to CelebA-HQ, consisting of faces and corresponding textual descriptions.

Figures (10)

• Figure 1: Generating a user-specified image. A first approximation is generated using a textual description. The resulting image is then further manipulated such that it is closer to the user's desire.
• Figure 2: Overview of our approach. An image is generated from a textual description (Section \ref{['sec:textStyleGAN']}). The images are generated progressively, starting from a low resolution. We only depict a single Generator block for clarity. The attention model retrieves the most relevant word vectors for generating different sub-regions. The $\mathcal{L}_\text{CMPM}$ and $\mathcal{L}_\text{CMPc}$ losses measure semantic consistency between the text and the generated image. The resulting image can be manipulated by making use of semantic directions in latent space (Section \ref{['sec:semantic_manipulation']}).
• Figure 3: textStyleGAN trained on CelebTD-HQ. Different noise vectors for all images.
• Figure 4: textStyleGAN trained on CUB. Different noise vectors for all images.
• Figure 5: textStyleGAN trained on COCO. Different noise vectors for all images.