TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering

TL;DR

TextDiffuser-2 tackles the persistent challenge of rendering accurate, aesthetically coherent text within diffusion-generated images by introducing two specialized language models: a layout planner that automatically derives line-level text layouts from prompts (with keyword inference and chat-based edits), and a layout encoder that embeds line-level text information into the diffusion process via a hybrid tokenization scheme. This line-level conditioning enables greater diversity in text styles while preserving accuracy, addressing prior methods’ limitations in flexibility, layout prediction, and style variety. Extensive quantitative and qualitative evaluations, augmented by human and GPT-4V user studies, demonstrate improved text-layout realism and broader stylistic diversity compared to state-of-the-art baselines. The work also explores practical extensions such as template-driven text rendering, inpainting, and text-free generation, and provides ablations to justify design choices, making TextDiffuser-2 a scalable, controllable framework for visual text rendering.

Abstract

The diffusion model has been proven a powerful generative model in recent years, yet remains a challenge in generating visual text. Several methods alleviated this issue by incorporating explicit text position and content as guidance on where and what text to render. However, these methods still suffer from several drawbacks, such as limited flexibility and automation, constrained capability of layout prediction, and restricted style diversity. In this paper, we present TextDiffuser-2, aiming to unleash the power of language models for text rendering. Firstly, we fine-tune a large language model for layout planning. The large language model is capable of automatically generating keywords for text rendering and also supports layout modification through chatting. Secondly, we utilize the language model within the diffusion model to encode the position and texts at the line level. Unlike previous methods that employed tight character-level guidance, this approach generates more diverse text images. We conduct extensive experiments and incorporate user studies involving human participants as well as GPT-4V, validating TextDiffuser-2's capacity to achieve a more rational text layout and generation with enhanced diversity. The code and model will be available at \url{https://aka.ms/textdiffuser-2}.

Figures (20)

• Figure 1: Text-to-image results generated by TextDiffuser-2. Alongside accurate text generation, TextDiffuser-2 offers reasonable text layouts and exhibits diversity in text style powered by the strong capability of language models.
• Figure 2: The architecture of TextDiffuser-2. The language model $\mathbf{M_{1}}$ and the diffusion model are trained in two stages. The language model $\mathbf{M_{1}}$ can convert the user prompt into a language-format layout and also allows users to specify keywords optionally. Further, the prompt and language-format layout is encoded with the trainable language model $\mathbf{M_{2}}$ within the diffusion model for generating images. $\mathbf{M_{1}}$ is trained via the cross-entropy loss in the first stage, while $\mathbf{M_{2}}$ and U-Net are trained using the denoising L2 loss in the second stage.
• Figure 3: Visualizations of layouts. TextDiffuser-2 generates more visually pleasing and rational layouts compared with TextDiffuser.
• Figure 4: Visualizations of text-to-image results compared with existing methods. TextDiffuser-2 can automatically extract keywords from prompts for accurate rendering. Additionally, the fonts generated by TextDiffuser-2 exhibit a wide range of diversity.
• Figure 5: Visualization of diversity in generating multiple images under the same prompt. TextDiffuser-2 is capable of generating more artistic fonts, with increased diversity in the positioning of characters and the inclination angle of text lines.