STEFANN: Scene Text Editor using Font Adaptive Neural Network

TL;DR

STEFANN addresses the problem of editing text directly in natural scene images without relying on optical character recognition. It introduces FANnet to generate a target glyph that preserves the source font’s structure and Colornet to transfer the source color, followed by careful placement via inpainting and seam carving to maintain inter-character spacing. The method is evaluated on COCO-Text and ICDAR, with quantitative metrics (ASSIM, nRMSE) and human judgments supporting perceptual quality, and shows advantages over single-observation font synthesis baselines. This enables error correction, text restoration, and improved reusability of edited scene images, with potential extensions toward higher resolution and broader font support.

Abstract

Textual information in a captured scene plays an important role in scene interpretation and decision making. Though there exist methods that can successfully detect and interpret complex text regions present in a scene, to the best of our knowledge, there is no significant prior work that aims to modify the textual information in an image. The ability to edit text directly on images has several advantages including error correction, text restoration and image reusability. In this paper, we propose a method to modify text in an image at character-level. We approach the problem in two stages. At first, the unobserved character (target) is generated from an observed character (source) being modified. We propose two different neural network architectures - (a) FANnet to achieve structural consistency with source font and (b) Colornet to preserve source color. Next, we replace the source character with the generated character maintaining both geometric and visual consistency with neighboring characters. Our method works as a unified platform for modifying text in images. We present the effectiveness of our method on COCO-Text and ICDAR datasets both qualitatively and quantitatively.

Figures (20)

• Figure 1: Examples of text editing using STEFANN: (a) Original images from ICDAR dataset; (b) Edited images. It can be observed that STEFANN can edit multiple characters in a word (top row) as well as an entire word (bottom row) in a text region.
• Figure 2: Architecture of FANnet and Colornet. At first, the target character ('N') is generated from the source character ('H') by FANnet keeping structural consistency. Then, the source color is transferred to the target by Colornet preserving visual consistency. Layer names in the figure are: conv = 2D convolution, FC = fully-connected, up-conv = upsampling + convolution.
• Figure 3: Generation of target characters using FANnet. In each image block, the upper row shows the ground truth and the bottom row shows the generated characters when the network has observed one particular source character ('A') in each case.
• Figure 4: Color transfer using Colornet: (a) Binary target character; (b) Color source character; (c) Ground truth; (d) Color transferred image. It can be observed that Colornet can successfully transfer solid color as well as gradient color.
• Figure 5: Images edited using STEFANN. In each image pair, the left image is the original image and the right image is the edited image. It can be observed that STEFANN can faithfully edit texts even in the presence of specular reflection, shadow, perspective distortion, etc. It is also possible to edit lower-case characters and numerals in a scene image. STEFANN can easily edit multiple characters and multiple text regions in an image. More results are included in the supplementary materials.