GMAIL: Generative Modality Alignment for generated Image Learning

TL;DR

A novel framework for discriminative use of generated images, coined GMAIL, that explicitly treats generated images as a separate modality from real images and effectively leverages the benefits of recent advances in generative models, thereby boosting the effectiveness of generated image learning across a range of vision-language tasks.

Abstract

Generative models have made it possible to synthesize highly realistic images, potentially providing an abundant data source for training machine learning models. Despite the advantages of these synthesizable data sources, the indiscriminate use of generated images as real images for training can even cause mode collapse due to modality discrepancies between real and synthetic domains. In this paper, we propose a novel framework for discriminative use of generated images, coined GMAIL, that explicitly treats generated images as a separate modality from real images. Instead of indiscriminately replacing real images with generated ones in the pixel space, our approach bridges the two distinct modalities in the same latent space through a multi-modal learning approach. To be specific, we first fine-tune a model exclusively on generated images using a cross-modality alignment loss and then employ this aligned model to further train various vision-language models with generated images. By aligning the two modalities, our approach effectively leverages the benefits of recent advances in generative models, thereby boosting the effectiveness of generated image learning across a range of vision-language tasks. Our framework can be easily incorporated with various vision-language models, and we demonstrate its efficacy throughout extensive experiments. For example, our framework significantly improves performance on image captioning, zero-shot image retrieval, zero-shot image classification, and long caption retrieval tasks. It also shows positive generated data scaling trends and notable enhancements in the captioning performance of the large multimodal model, LLaVA.

Figures (9)

• Figure 1: Illustration of the proposed framework for vision-language tuning with Gen-Real alignment from diffusion models. We propose a method that explicitly aligns a CLIP trained on real images with another CLIP model trained on generated images and then leverages the aligned CLIP to train state-of-the-art vision-language models on generated images ( i.e., Gen-CLIP Flow). For inference with real images, the original CLIP is used to process real images, thereby avoiding discrepancies between real and generated modalities.
• Figure 2: Visualizations of real (Column 1) and generated images (Columns 2-6) using the same caption. Those generated images generally capture high-level semantics in real images.
• Figure 3: Qualitative Visualizations of embeddings of real and synthetic images without (Left) and with (Right) alignment. Blue and red dots denote the embeddings for real and synthetic images, respectively. Our GMAIL with alignment significantly reduced the gap between real and synthetic images, with both modalities aligning closely in the latent space.
• Figure 4: Visualizations of real (Column 1) and generated images (Columns 2-6) using the same caption. Those generated images generally capture high-level semantics in real images.
• Figure 5: Visualizations of real (Column 1) and generated images (Columns 2-6) using the same caption. Those generated images generally capture high-level semantics in real images.