End-to-end Training for Text-to-Image Synthesis using Dual-Text Embeddings

TL;DR

This work tackles text-to-image synthesis by jointly learning text representations tailored to the synthesis model through end-to-end training. It introduces Dual Text Embeddings (DTE), comprising generator-ready S_G and discriminator-ready S_D, learned via separate gradient flows to optimize photo-realism and text–image alignment respectively. The architecture uses a single-stage generator with conditioning augmentation and a two-branch discriminator trained with adversarial and multi-modal contrastive losses, achieving state-of-the-art or competitive FID/R-precision metrics across Oxford-102, CUB, and MS-COCO, while enabling cross-task benefits such as text-to-image manipulation. The results demonstrate that task-specific embeddings outperform pre-trained, generic embeddings, and that dual embeddings generalize to other models (e.g., AttnGAN) and downstream tasks, highlighting the practical impact of end-to-end, dual-representation learning in language-vision systems.

Abstract

Text-to-Image (T2I) synthesis is a challenging task that requires modeling complex interactions between two modalities ( i.e., text and image). A common framework adopted in recent state-of-the-art approaches to achieving such multimodal interactions is to bootstrap the learning process with pre-trained image-aligned text embeddings trained using contrastive loss. Furthermore, these embeddings are typically trained generically and reused across various synthesis models. In contrast, we explore an approach to learning text embeddings specifically tailored to the T2I synthesis network, trained in an end-to-end fashion. Further, we combine generative and contrastive training and use two embeddings, one optimized to enhance the photo-realism of the generated images, and the other seeking to capture text-to-image alignment. A comprehensive set of experiments on three text-to-image benchmark datasets (Oxford-102, Caltech-UCSD, and MS-COCO) reveal that having two separate embeddings gives better results than using a shared one and that such an approach performs favourably in comparison with methods that use text representations from a pre-trained text encoder trained using a discriminative approach. Finally, we demonstrate that such learned embeddings can be used in other contexts as well, such as text-to-image manipulation.

Figures (8)

• Figure 1: (a) Text and image are projected into a shared embedding space to enhance mutual information capturing discriminative features (Discriminative Embeddings), (b) Word embeddings are trained by generating images capturing semantic details (Generative Embeddings) and (c) Our Dual Text Embedding approach combining both generative and discriminative embeddings.
• Figure 2: Overview of DTE-GAN architecture. DTE-GAN consists of three core components: i) a single-stage generator $G$ (Section \ref{['sec:generator']}), ii) a discriminator $D$ (Section \ref{['sec:discriminator']}), and iii) a dual text embedding setup (Section \ref{['sec:dualtextembed']}). In the Figure, $W_G$ = generator-side word embeddings, $S_G$ = generator-side sentence embedding, $W_D$ = discriminator-side word embeddings, $S_D$ = discriminator-side sentence embedding. The model is optimised using two objective functions: 1) adversarial loss, and 2) multi-modal contrastive loss.
• Figure 3: Visual comparision of the images generated by DF-GAN DF_GAN_CVPR and DTE-GAN on CUBCUB_dataset and COCOcoco_dataset Datasets.
• Figure 4: Illustration of the images generated by HDGAN hd_gan and those of DTE-GAN on Oxford-102 Flower Dataset flower_dataset.
• Figure 5: Images generated using CLIP, CLIP + Learnable Generator side embeddings (CLIP + $\mathbf{G_{emb}}$) and DTE on CUB and Oxford-102 datasets.