GlyphDraw2: Automatic Generation of Complex Glyph Posters with Diffusion Models and Large Language Models

TL;DR

GlyphDraw2 presents an automatic poster-generation framework that tightly integrates precise glyph-level text rendering with rich background contexts by extending diffusion models with a Triples of Cross-Attention (TCA) mechanism, an Auxiliary Align Loss, and LLM-driven layout inference. The approach fuses glyph-aware features via a Fusion Text Encoder (InternViT-based) and leverages ControlNet-conditioned blocks to balance local font detail with global background quality, all while training on high-resolution bilingual datasets. Key contributions include the TCA architecture, the semantic-alignment loss, and a two-stage training regimen (diffusion model plus LLM-based layout) supported by a 1.6B-parameter model on 64 A100 GPUs. Experiments across multiple benchmarks (AnyText-Benchmark, ICDAR13, MARIO-Eval, Complex-Benchmark, Poster-Benchmark) demonstrate superior text rendering accuracy and robust poster layouts, with ablations confirming the effectiveness of each component. The work advances automatic poster generation by enabling bilingual, high-quality, layout-aware text rendering with controllable fonts and resolutions, though it acknowledges limitations in complex bbox scenarios and background-text balance, outlining future improvements via advanced semantic-conditioned architectures.

Abstract

Posters play a crucial role in marketing and advertising by enhancing visual communication and brand visibility, making significant contributions to industrial design. With the latest advancements in controllable T2I diffusion models, increasing research has focused on rendering text within synthesized images. Despite improvements in text rendering accuracy, the field of automatic poster generation remains underexplored. In this paper, we propose an automatic poster generation framework with text rendering capabilities leveraging LLMs, utilizing a triple-cross attention mechanism based on alignment learning. This framework aims to create precise poster text within a detailed contextual background. Additionally, the framework supports controllable fonts, adjustable image resolution, and the rendering of posters with descriptions and text in both English and Chinese.Furthermore, we introduce a high-resolution font dataset and a poster dataset with resolutions exceeding 1024 pixels. Our approach leverages the SDXL architecture. Extensive experiments validate our method's capability in generating poster images with complex and contextually rich backgrounds.Codes is available at https://github.com/OPPO-Mente-Lab/GlyphDraw2.

Figures (13)

• Figure 1: GlyphDraw2 enables seamless and automated generation, eliminating the need for manual box input.
• Figure 2: The issues with conventional ControlNet in detail generation.
• Figure 3: An overview of the proposed GlyphDraw2 method.
• Figure 4: The detail generation with GlyphDraw2.
• Figure 5: Statistical indicators of English data in general datasets.