FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editing

TL;DR

FLUX-Text addresses multilingual scene text editing by leveraging a Diffusion Transformer backbone (DiT) augmented with lightweight Visual and Text Embedding Modules and a Regional Text Perceptual Loss. By using a frozen VAE encoder with rendered glyph priors and a single T5-based text encoder, the method achieves high text fidelity and visual quality with only 100K training examples, a 97% data reduction from prior methods. A two-stage training strategy that increases emphasis on text regions further enhances glyph details and text-background harmony, achieving SoTA results on English and Chinese benchmarks and strong performance on MARIO-Eval-edit. Overall, FLUX-Text demonstrates that DiT-based architectures, when paired with targeted visual/text priors and region-focused supervision, can deliver robust, data-efficient multilingual scene text editing suitable for real-world applications.

Abstract

Scene text editing aims to modify or add texts on images while ensuring text fidelity and overall visual quality consistent with the background. Recent methods are primarily built on UNet-based diffusion models, which have improved scene text editing results, but still struggle with complex glyph structures, especially for non-Latin ones (\eg, Chinese, Korean, Japanese). To address these issues, we present \textbf{FLUX-Text}, a simple and advanced multilingual scene text editing DiT method. Specifically, our FLUX-Text enhances glyph understanding and generation through lightweight Visual and Text Embedding Modules, while preserving the original generative capability of FLUX. We further propose a Regional Text Perceptual Loss tailored for text regions, along with a matching two-stage training strategy to better balance text editing and overall image quality. Benefiting from the DiT-based architecture and lightweight feature injection modules, FLUX-Text can be trained with only $0.1$M training examples, a \textbf{97\%} reduction compared to $2.9$M required by popular methods. Extensive experiments on multiple public datasets, including English and Chinese benchmarks, demonstrate that our method surpasses other methods in visual quality and text fidelity. All the code is available at https://github.com/AMAP-ML/FluxText.

Figures (5)

• Figure 1: Scene text editing results of FLUX-Text under various conditions (e.g., English, Chinese, Korean, Japanese, and so on).
• Figure 2: (a) The framework of our proposed FLUX-Text. (b) Loss functions used to train FLUX-Text, including the RF loss and our proposed Regional Text Perceptual (RTP) loss.
• Figure 3: (i) Our Architecture. (ii)$\thicksim$(iv) Different Visual Embedding Module. (v)$\thicksim$(vii) Different Text Embedding Module.
• Figure 4: Qualitative comparison of FLUX-Text and SoTA methods in Chinese and English scene text editing.
• Figure 5: Visual generalization of FLUX-Text on web-crawled images (Left: masked input, Right: our result).