LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven Language Representation

TL;DR

LDGen addresses multilingual limitations in text-to-image diffusion by integrating large language models through a language representation strategy, a lightweight LLM alignment adapter, and a cross-modal refiner. This framework enables zero-shot multilingual image generation with reduced training demand while improving prompt adherence and image aesthetics. Key contributions include the hierarchical captioning-based language representation, the two-stage LLM alignment, and the cross-modal refiner that enriches text features with visual context. The approach achieves substantial gains over baselines and approaches larger models on semantic alignment, offering practical benefits for multilingual T2I tasks under constrained compute.

Abstract

In this paper, we introduce LDGen, a novel method for integrating large language models (LLMs) into existing text-to-image diffusion models while minimizing computational demands. Traditional text encoders, such as CLIP and T5, exhibit limitations in multilingual processing, hindering image generation across diverse languages. We address these challenges by leveraging the advanced capabilities of LLMs. Our approach employs a language representation strategy that applies hierarchical caption optimization and human instruction techniques to derive precise semantic information,. Subsequently, we incorporate a lightweight adapter and a cross-modal refiner to facilitate efficient feature alignment and interaction between LLMs and image features. LDGen reduces training time and enables zero-shot multilingual image generation. Experimental results indicate that our method surpasses baseline models in both prompt adherence and image aesthetic quality, while seamlessly supporting multiple languages. Project page: https://zrealli.github.io/LDGen.

Figures (9)

• Figure 1: Overview of LDGen. The dashed box shows our language representation strategy, with the bottom is our LLM alignment and cross-modal refiner training process. The detailed design of the cross-modal refiner is shown in the green box on the right.
• Figure 2: Distribution of text encoder and supported languages. English-based CLIP/T5 series models remain the primary text encoders.
• Figure 3: The red words in Sana's generated result highlight elements that do not align with the image. Providing incorrect instructions can change the original caption, potentially creating inaccurate descriptions.
• Figure 4: Comparison of our method with recent enhancement generative models ELLA hu2024ella, baseline Models SDXL podell2023sdxl and PixArt-$\alpha$chen2023pixart. Our method achieves the best results in terms of instruction adherence and visual appeal.
• Figure 5: Multilingual qualitative visualization results. For each panel's eight images, we generate them using eight different languages but only display the prompt in one of the languages used. Note that LDGen uses only English prompts during training but achieves zero-shot multilingual generation due to the capabilities of the LLM.