STELLAR: Scene Text Editor for Low-Resource Languages and Real-World Data

TL;DR

STELLAR addresses the challenge of editing scene text in images across low-resource languages by introducing a language-adaptive glyph encoder, a real-world paired dataset STIPLAR, and Text Appearance Similarity (TAS) to quantify style preservation. It uses a diffusion-based generator conditioned on glyph and style representations, trained in two stages: synthetic pretraining and real-world fine-tuning. The approach yields improvements in visual fidelity and recognition accuracy across Korean, Arabic, and Japanese, with TAS correlating with human judgments and enabling evaluation without ground truth. Collectively, STELLAR and TAS offer a practical pathway for multilingual, real-world content editing in multimedia workflows.

Abstract

Scene Text Editing (STE) is the task of modifying text content in an image while preserving its visual style, such as font, color, and background. While recent diffusion-based approaches have shown improvements in visual quality, key limitations remain: lack of support for low-resource languages, domain gap between synthetic and real data, and the absence of appropriate metrics for evaluating text style preservation. To address these challenges, we propose STELLAR (Scene Text Editor for Low-resource LAnguages and Real-world data). STELLAR enables reliable multilingual editing through a language-adaptive glyph encoder and a multi-stage training strategy that first pre-trains on synthetic data and then fine-tunes on real images. We also construct a new dataset, STIPLAR(Scene Text Image Pairs of Low-resource lAnguages and Real-world data), for training and evaluation. Furthermore, we propose Text Appearance Similarity (TAS), a novel metric that assesses style preservation by independently measuring font, color, and background similarity, enabling robust evaluation even without ground truth. Experimental results demonstrate that STELLAR outperforms state-of-the-art models in visual consistency and recognition accuracy, achieving an average TAS improvement of 2.2% across languages over the baselines.

Figures (10)

• Figure 1: Example of text editing performed by STELLAR, showing visually consistent modification of Korean text within a real-world scene.
• Figure 2: Multi-task training scheme of the text style encoder $S$ in baseline.
• Figure 3: STELLAR framework and training pipeline. Language-adaptive glyph encoder $T$, pre-trained with a language-specific recognizer, extracts glyph features $C_\text{glyph}$ to guide the diffusion generator $G$ via cross-attention. Style features $C_\text{style}$ from pre-trained text style encoder $S$ are injected through skip-connections. $G$ is first pre-trained on synthetic data (Stage 1) and then fine-tuned on real-world images from the STIPLAR dataset (Stage 2).
• Figure 4: Comparison of edited results across baselines on Korean, Arabic, and Japanese text images.
• Figure A1: Examples of full images and their corresponding cropped text image pairs in Korean, Arabic, and Japanese from the STIPLAR dataset.