GlyphMastero: A Glyph Encoder for High-Fidelity Scene Text Editing

TL;DR

This work tackles the challenge of high-fidelity scene text editing by introducing GlyphMastero, a trainable glyph encoder that provides stroke- and line-level guidance to a diffusion inpainting model. By explicitly modeling hierarchical relationships between local glyphs and global text-line structures through a dual-stream OCR feature extractor, multi-scale FPN fusion, and cross-level glyph attention, the method achieves substantial improvements in text accuracy and style preservation across multilingual scripts. Quantitative gains include an 18.02% relative increase in sentence accuracy over the prior multilingual baseline and a 53.28% reduction in text-region FID, demonstrating robust, glyph-aware editing in complex scenes. Limitations remain for long-text edits due to fixed input resolution, suggesting future work on higher-resolution training and stronger base models to extend performance further.

Abstract

Scene text editing, a subfield of image editing, requires modifying texts in images while preserving style consistency and visual coherence with the surrounding environment. While diffusion-based methods have shown promise in text generation, they still struggle to produce high-quality results. These methods often generate distorted or unrecognizable characters, particularly when dealing with complex characters like Chinese. In such systems, characters are composed of intricate stroke patterns and spatial relationships that must be precisely maintained. We present GlyphMastero, a specialized glyph encoder designed to guide the latent diffusion model for generating texts with stroke-level precision. Our key insight is that existing methods, despite using pretrained OCR models for feature extraction, fail to capture the hierarchical nature of text structures - from individual strokes to stroke-level interactions to overall character-level structure. To address this, our glyph encoder explicitly models and captures the cross-level interactions between local-level individual characters and global-level text lines through our novel glyph attention module. Meanwhile, our model implements a feature pyramid network to fuse the multi-scale OCR backbone features at the global-level. Through these cross-level and multi-scale fusions, we obtain more detailed glyph-aware guidance, enabling precise control over the scene text generation process. Our method achieves an 18.02\% improvement in sentence accuracy over the state-of-the-art multi-lingual scene text editing baseline, while simultaneously reducing the text-region Fréchet inception distance by 53.28\%.

Figures (11)

• Figure 1: Example results of our scene text editing method on random images collected from the internet. The original images (left column) show source text regions marked with red boxes, with target texts displayed at the bottom left. The edited results (right column) demonstrate our method's ability to preserve both structural accuracy and style consistency through stroke-level guidance across different visual styles and writing systems.
• Figure 2: General pipeline for conditioning latent diffusion models with additional guidance signals. An extra condition $y$ is processed through a condition encoder $\tau_\theta$ to produce a condition embedding $c$. This embedding guides the denoising UNet via cross-attention during the iterative denoising process, which transforms the noisy latent $z_T$ into a clean latent representation $z_0$ over $T$ steps. Finally, an image decoder $\mathcal{D}$ converts the latent representation $z_0$ into the final predicted conditioned image $\hat{x}$.
• Figure 3: Complete model architecture of GlyphMastero. A specialized glyph encoder that introduces stroke-level precise control to the latent diffusion model for scene text editing.
• Figure 4: Glyph Attention Module
• Figure 5: Qualitative comparison of scene text editing methods. Our GlyphMastero framework demonstrates superior text style preservation and content replacement accuracy. For Chinese cases (with English translations in brackets), our method achieves more precise text generation than DiffUTE and better style preservation than AnyText.