RepText: Rendering Visual Text via Replicating

TL;DR

This work tackles the challenge of precise multilingual visual text rendering in diffusion-based text-to-image models. It introduces RepText, a ControlNet-inspired framework that replicates glyph edges using canny and position cues, rather than requiring the model to understand the textual content semantically; training employs a text perceptual (OCR-guided) loss and a denoising objective, with a combined loss $L = L_{denoise} + \\lambda \, L_{reward}$, where $L_{reward}$ measures OCR feature alignment. At inference, RepText stabilizes rendering by initializing from a noise-free glyph latent and restricting guidance to text regions via regional masks, while allowing color control through glyph latent replication. Empirically, RepText outperforms open-source multilingual text rendering methods and rivals native multilingual closed-source models in text fidelity, with demonstrated compatibility to LoRAs, ControlNets, and IP-Adapter; limitations include handling highly complex scripts and exact color control, motivating future work toward lightweight language-aware adapters or multilingual connectors for broader, low-cost language understanding in rendering.

Abstract

Although contemporary text-to-image generation models have achieved remarkable breakthroughs in producing visually appealing images, their capacity to generate precise and flexible typographic elements, especially non-Latin alphabets, remains constrained. To address these limitations, we start from an naive assumption that text understanding is only a sufficient condition for text rendering, but not a necessary condition. Based on this, we present RepText, which aims to empower pre-trained monolingual text-to-image generation models with the ability to accurately render, or more precisely, replicate, multilingual visual text in user-specified fonts, without the need to really understand them. Specifically, we adopt the setting from ControlNet and additionally integrate language agnostic glyph and position of rendered text to enable generating harmonized visual text, allowing users to customize text content, font and position on their needs. To improve accuracy, a text perceptual loss is employed along with the diffusion loss. Furthermore, to stabilize rendering process, at the inference phase, we directly initialize with noisy glyph latent instead of random initialization, and adopt region masks to restrict the feature injection to only the text region to avoid distortion of the background. We conducted extensive experiments to verify the effectiveness of our RepText relative to existing works, our approach outperforms existing open-source methods and achieves comparable results to native multi-language closed-source models. To be more fair, we also exhaustively discuss its limitations in the end.

Figures (19)

• Figure 1: Illustrating of RepText generated samples for different text, languages and font conditions.
• Figure 2: The training pipeline of RepText, where we use both fine-grained canny edge and position mask as conditions to train text ControlNet, and further adopt a text perceptual loss.
• Figure 3: The inference framework of RepText with highlighted strategies: (1) Replicating from noise-free glyph latent, which improves text accuracy and enables color control. (2) Adopt regional mask for text regions, which avoids interference with non-text areas and ensures overall quality.
• Figure 4: RepText can render multilingual texts by replicating glyph condition.
• Figure 5: Illustrating of RepText's compatibility to community LoRAs. From top to bottom, they are FilmPortrait, wool yarn art and sketched style respectively.