FreeText: Training-Free Text Rendering in Diffusion Transformers via Attention Localization and Spectral Glyph Injection

TL;DR

This paper tackles the persistent challenge of precise text rendering in open-domain diffusion-based image synthesis, especially for multi-line and long-tail scripts. It introduces FreeText, a training-free, plug-in framework that operates in two stages: (i) where to write by extracting stable text-region anchors from endogenous image-to-text attention (attention sinks) to localize writing regions, and (ii) what to write by injecting a spectral-modulated glyph prior (SGMI) in the latent space using a Log-Gabor-based band-pass modulation, aligned to the current noise level and injected in a mid-early denoising window. FreeText does not modify model parameters and maintains semantic alignment and aesthetics while improving text readability across multiple base models (Qwen-Image, FLUX.1-dev, SD3-M, SD3.5-L) and benchmarks (longText-Benchmark, CVTG, CLT-Bench), with only modest inference overhead. The approach introduces CLT-Bench for evaluating Chinese long-tail text rendering and demonstrates cross-region benefits via full-attention dynamics, suggesting practical impact for posters, UI, and multi-language applications where accurate glyphs matter. Overall, FreeText offers a scalable, model-agnostic solution to enhance typography fidelity in diffusion-based generation without retraining or heavy supervision, advancing the usability of open-domain text rendering in real-world scenarios.

Abstract

Large-scale text-to-image (T2I) diffusion models excel at open-domain synthesis but still struggle with precise text rendering, especially for multi-line layouts, dense typography, and long-tailed scripts such as Chinese. Prior solutions typically require costly retraining or rigid external layout constraints, which can degrade aesthetics and limit flexibility. We propose \textbf{FreeText}, a training-free, plug-and-play framework that improves text rendering by exploiting intrinsic mechanisms of \emph{Diffusion Transformer (DiT)} models. \textbf{FreeText} decomposes the problem into \emph{where to write} and \emph{what to write}. For \emph{where to write}, we localize writing regions by reading token-wise spatial attribution from endogenous image-to-text attention, using sink-like tokens as stable spatial anchors and topology-aware refinement to produce high-confidence masks. For \emph{what to write}, we introduce Spectral-Modulated Glyph Injection (SGMI), which injects a noise-aligned glyph prior with frequency-domain band-pass modulation to strengthen glyph structure and suppress semantic leakage (rendering the concept instead of the word). Extensive experiments on Qwen-Image, FLUX.1-dev, and SD3 variants across longText-Benchmark, CVTG, and our CLT-Bench show consistent gains in text readability while largely preserving semantic alignment and aesthetic quality, with modest inference overhead.

Figures (8)

• Figure 1: System overview. (a) Prior text-rendering methods typically require retraining and/or rigid layout conditions. (b) FreeText decomposes text rendering into WHERE and WHAT: it localizes text regions via endogenous attention maps, then injects a glyph-structure prior in a model-compatible way, enabling training-free enhancement while preserving the base model's aesthetics.
• Figure 2: Overview of FreeText.
• Figure 3: Typical I2T attention patterns across timesteps: early steps are coarse, mid steps concentrate on target regions, and late steps become diffuse.
• Figure 4: Baseline comparison across four text-rendering scenarios (comic, caption, poster, slide). Top: Base; bottom: Base+FreeText. Red boxes highlight the target text regions, where FreeText reduces typos/malformed glyphs and improves readability.
• Figure 5: Cross-region benefit propagation and attention evidence (example with two text lines; refining only one line can improve the other).