Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Support

Xiaojun Wu, Dixiang Zhang, Ruyi Gan, Junyu Lu, Ziwei Wu, Renliang Sun, Jiaxing Zhang, Pingjian Zhang, Yan Song

TL;DR

Taiyi-Diffusion-XL tackles the gap in open-source bilingual Chinese–English text-to-image generation by extending CLIP and Stable Diffusion-XL through bilingual continuous pre-training. The approach expands the CLIP vocabulary and position encoding, enriches prompts with a large vision-language model, and trains a bilingual diffusion model (Taiyi-XL) using a two-stage process that includes a bilingual text encoder and latent diffusion with a $L(\theta)$ objective. Empirical results show strong bilingual image-text retrieval and superior generation quality on both English and Chinese benchmarks, with notable gains over open-source bilingual baselines and competitive performance relative to commercial systems. The work provides an open-source Taiyi-XL release and demonstrates a practical pathway toward robust multilingual multimodal research and applications.

Abstract

Recent advancements in text-to-image models have significantly enhanced image generation capabilities, yet a notable gap of open-source models persists in bilingual or Chinese language support. To address this need, we present Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model which is developed by extending the capabilities of CLIP and Stable-Diffusion-XL through a process of bilingual continuous pre-training. This approach includes the efficient expansion of vocabulary by integrating the most frequently used Chinese characters into CLIP's tokenizer and embedding layers, coupled with an absolute position encoding expansion. Additionally, we enrich text prompts by large vision-language model, leading to better images captions and possess higher visual quality. These enhancements are subsequently applied to downstream text-to-image models. Our empirical results indicate that the developed CLIP model excels in bilingual image-text retrieval.Furthermore, the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass previous models. This research leads to the development and open-sourcing of the Taiyi-Diffusion-XL model, representing a notable advancement in the field of image generation, particularly for Chinese language applications. This contribution is a step forward in addressing the need for more diverse language support in multimodal research. The model and demonstration are made publicly available at \href{https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/}, fostering further research and collaboration in this domain.

Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with Large Vision-Language Model Support

TL;DR

Taiyi-Diffusion-XL tackles the gap in open-source bilingual Chinese–English text-to-image generation by extending CLIP and Stable Diffusion-XL through bilingual continuous pre-training. The approach expands the CLIP vocabulary and position encoding, enriches prompts with a large vision-language model, and trains a bilingual diffusion model (Taiyi-XL) using a two-stage process that includes a bilingual text encoder and latent diffusion with a objective. Empirical results show strong bilingual image-text retrieval and superior generation quality on both English and Chinese benchmarks, with notable gains over open-source bilingual baselines and competitive performance relative to commercial systems. The work provides an open-source Taiyi-XL release and demonstrates a practical pathway toward robust multilingual multimodal research and applications.

Abstract

Recent advancements in text-to-image models have significantly enhanced image generation capabilities, yet a notable gap of open-source models persists in bilingual or Chinese language support. To address this need, we present Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model which is developed by extending the capabilities of CLIP and Stable-Diffusion-XL through a process of bilingual continuous pre-training. This approach includes the efficient expansion of vocabulary by integrating the most frequently used Chinese characters into CLIP's tokenizer and embedding layers, coupled with an absolute position encoding expansion. Additionally, we enrich text prompts by large vision-language model, leading to better images captions and possess higher visual quality. These enhancements are subsequently applied to downstream text-to-image models. Our empirical results indicate that the developed CLIP model excels in bilingual image-text retrieval.Furthermore, the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass previous models. This research leads to the development and open-sourcing of the Taiyi-Diffusion-XL model, representing a notable advancement in the field of image generation, particularly for Chinese language applications. This contribution is a step forward in addressing the need for more diverse language support in multimodal research. The model and demonstration are made publicly available at \href{https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/}, fostering further research and collaboration in this domain.
Paper Structure (17 sections, 3 equations, 5 figures, 2 tables)

This paper contains 17 sections, 3 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Dataset Preparation
  4. CLIP Training
  5. Taiyi-XL Training
  6. Initialization and Training.
  7. Text-to-Image Generation.
  8. Experiment And Analysis
  9. Machine Evaluation
  10. CLIP Model Evaluation.
  11. Diffusion Model Evaluation.
  12. Human Preference Evaluation
  13. Related Work
  14. Advancements in Image Generation and Diffusion Models
  15. Text-to-Image Models in Bilingual Context
  16. ...and 2 more sections

Figures (5)

  • Figure 1: An illustration of Taiyi-XL showcasing text-to-image generation results under various styles and prompts.
  • Figure 2: Overview of the Taiyi-Diffusion-XL(Taiyi-XL) training process, encompassing data preprocessing, image-text contrastive learning and multi-resolution denoising training process.
  • Figure 3: Comparison of Different Models in Chinese Text-to-Image Generation Performance.
  • Figure 4: Comparison of Different Models in English Text-to-Image Generation Performance.
  • Figure 5: Taiyi-XL generation examples with Latent Consistency Model