WordCon: Word-level Typography Control in Scene Text Rendering

TL;DR

This work tackles the challenge of precise word-level typography control in scene text rendering by introducing Text-Image Alignment (TIA), which leverages grounding-model segmentation for fine-grained text-region alignment, and WordCon, a hybrid PEFT method with latent-masked and joint-attention losses to enforce word-level disentanglement. It also provides a dedicated word-level typography dataset and demonstrates through extensive experiments that the approach achieves superior controllability while maintaining competitive image quality and OCR accuracy, with broad compatibility to image-conditioned pipelines and style LoRAs. The results are validated via quantitative metrics, qualitative visualizations, and human studies, underscoring practical impact for advertising, design, and editable text rendering tasks. The work offers a viable, plug-and-play path to scalable, accurate word-level typography control in diffusion-based scene text generation.

Abstract

Achieving precise word-level typography control within generated images remains a persistent challenge. To address it, we newly construct a word-level controlled scene text dataset and introduce the Text-Image Alignment (TIA) framework. This framework leverages cross-modal correspondence between text and local image regions provided by grounding models to enhance the Text-to-Image (T2I) model training. Furthermore, we propose WordCon, a hybrid parameter-efficient fine-tuning (PEFT) method. WordCon reparameterizes selective key parameters, improving both efficiency and portability. This allows seamless integration into diverse pipelines, including artistic text rendering, text editing, and image-conditioned text rendering. To further enhance controllability, the masked loss at the latent level is applied to guide the model to concentrate on learning the text region in the image, and the joint-attention loss provides feature-level supervision to promote disentanglement between different words. Both qualitative and quantitative results demonstrate the superiority of our method to the state of the art. The datasets and source code will be available for academic use.

Figures (11)

• Figure 1: (a) Scene text rendering results with word-level typography control from WordCon. The controlled content of each image is 'Let', 'us', 'control', 'the', 'target word', and 'Just'. The typographic attributes are 'bold', 'underline', and 'italic' from top to bottom. (b) WordCon is compatible with artistic LoRAs, Flux-fill FLUX-Fill, and image conditioned pipelines tan2024ominicontrol, which makes it suitable for various tasks, including artistic text rendering (first row), text editing (second row), and image conditioned text rendering (third row). (c) shows more visual results of diverse applications.
• Figure 2: The challenge of text rendering. The SOTA T2I models excel at general text rendering and controllability on common objects, however, they struggle with precise word-level typography control.
• Figure 3: The green regions are the attention maps of each word in the prompt. Compared to words that refer to common objects (e.g., 'Girl', 'Book', 'Dog', 'Boy', 'T-shirt', 'Cap'), attention map of words for text rendering is more likely to be misaligned. In this paper, we refer to it as word-level misalignment.
• Figure 4: Results of fine-tuning Flux with word-level control dataset across varied training steps show that total acc is mainly limited by word acc.
• Figure 5: Method overview: (a) to mitigate word-level misalignment, we employ a text-image alignment framework that leverages the cross-modal correspondence between textual query and image regions provided by grounding models. In addition, (b) to conserve computational resources and enhance flexibility, we introduce WordCon, a hybrid PEFT method that reparameterizes selective key parameters with two losses. The masked loss at the latent level is applied to guide the model to concentrate on learning the text part, and the joint-attention loss provides feature-level supervision to promote disentanglement between different words. (c) The plug-and-play inference pipeline with other modules shows the wide applicability of our method.