DSE-GAN: Dynamic Semantic Evolution Generative Adversarial Network for Text-to-Image Generation

TL;DR

DSE-GAN addresses the limitation of static text semantics in multi-stage text-to-image generation by introducing Dynamic Semantic Evolution (DSE), which adaptively re-composes word features at each stage based on historical image-text feedback. The framework uses a Single Adversarial Multi-stage Architecture (SAMA) to enable extensive text-image interactions without multi-discriminator training, incorporating a dynamic element router and multi-granularity subspace routing to refine semantics over stages. Empirical results on CUB-200 and MSCOCO show notable improvements in FID and R-precision, with qualitative evidence of richer details and better layout. This approach advances T2I by aligning semantic guidance with coarse-to-fine image synthesis, potentially enabling more controllable and accurate generation in downstream applications.

Abstract

Text-to-image generation aims at generating realistic images which are semantically consistent with the given text. Previous works mainly adopt the multi-stage architecture by stacking generator-discriminator pairs to engage multiple adversarial training, where the text semantics used to provide generation guidance remain static across all stages. This work argues that text features at each stage should be adaptively re-composed conditioned on the status of the historical stage (i.e., historical stage's text and image features) to provide diversified and accurate semantic guidance during the coarse-to-fine generation process. We thereby propose a novel Dynamical Semantic Evolution GAN (DSE-GAN) to re-compose each stage's text features under a novel single adversarial multi-stage architecture. Specifically, we design (1) Dynamic Semantic Evolution (DSE) module, which first aggregates historical image features to summarize the generative feedback, and then dynamically selects words required to be re-composed at each stage as well as re-composed them by dynamically enhancing or suppressing different granularity subspace's semantics. (2) Single Adversarial Multi-stage Architecture (SAMA), which extends the previous structure by eliminating complicated multiple adversarial training requirements and therefore allows more stages of text-image interactions, and finally facilitates the DSE module. We conduct comprehensive experiments and show that DSE-GAN achieves 7.48\% and 37.8\% relative FID improvement on two widely used benchmarks, i.e., CUB-200 and MSCOCO, respectively.

Figures (6)

• Figure 1: A real case of semantic evolution at different stages of our method in a T2I generation process. Text features consist of word embeddings, and the semantics of each word is visualized by the top-3 words with the highest cosine similar in the embedding space. Previous methods use static text features at all generation stages and the word embeddings remain unchanged, while we dynamically re-compose text features conditioned on the historical stage to provide diversified and accurate semantic guidance for each stage. Take the word "bill" as an example, during the text feature evolving process, new and consistent semantics (the word in red, e.g., "pointy", "white") are automatically activated stage by stage which finally leads to more detailed and vivid generation results. For more detailed visualization refer to Fig. \ref{['interpretation']}.
• Figure 2: (a) An overview of the proposed DSE-GAN framework, which is consist of a pre-trained text encoder and a single generator-discriminator pair. The generator is consist of several Dynamic Semantic Evolution (DSE) modules and a single adversarial multi-stage structure (SAMA). Here each DSE module is responsible for re-composing the word features conditioned on the previous stage's image and word features while each sub-generator in SAMA further generates the next stage's image features under the re-composed semantic guidance. (b) The design of each sub-generator.
• Figure 3: Illustration of the proposed DSE module, which is consist of features aggregation sub-module for summarizing generative feedback, dynamic element router sub-module for selecting the words required to be re-composed and dynamic subspace router sub-module for re-composing the selected word features by dynamically enhancing or suppressing different granularity subspace’s semantics.
• Figure 4: An example to illustrate the adjacency masks.
• Figure 5: Examples of images synthesized by previous methods and our method on CUB-200 benchmark (1$^{st}$ - 4$^{th}$ columns) and MSCOCO benchmark (5$^{th}$ - 8$^{th}$ columns). The input text descriptions are given in the first row and the corresponding generated images from different methods are shown in the same column. Compared with previous methods, our DSE-GAN produces much more realistic results with rich and vivid details on both benchmarks. Best view in color and zoom in.