DiffSTR: Controlled Diffusion Models for Scene Text Removal

TL;DR

A ControlNet diffusion model is introduced, treating STR as an inpainting task, and this model improves mask prediction and utilizes rich textural information in natural scene images to provide accurate inpainting masks.

Abstract

To prevent unauthorized use of text in images, Scene Text Removal (STR) has become a crucial task. It focuses on automatically removing text and replacing it with a natural, text-less background while preserving significant details such as texture, color, and contrast. Despite its importance in privacy protection, STR faces several challenges, including boundary artifacts, inconsistent texture and color, and preserving correct shadows. Most STR approaches estimate a text region mask to train a model, solving for image translation or inpainting to generate a text-free image. Thus, the quality of the generated image depends on the accuracy of the inpainting mask and the generator's capability. In this work, we leverage the superior capabilities of diffusion models in generating high-quality, consistent images to address the STR problem. We introduce a ControlNet diffusion model, treating STR as an inpainting task. To enhance the model's robustness, we develop a mask pretraining pipeline to condition our diffusion model. This involves training a masked autoencoder (MAE) using a combination of box masks and coarse stroke masks, and fine-tuning it using masks derived from our novel segmentation-based mask refinement framework. This framework iteratively refines an initial mask and segments it using the SLIC and Hierarchical Feature Selection (HFS) algorithms to produce an accurate final text mask. This improves mask prediction and utilizes rich textural information in natural scene images to provide accurate inpainting masks. Experiments on the SCUT-EnsText and SCUT-Syn datasets demonstrate that our method significantly outperforms existing state-of-the-art techniques.

Figures (6)

• Figure 1: From left-to-right: Original scene image, Text erased image by conditional step, closer view of text background after erasure, DiffSTR result
• Figure 2: Architecture diagram of DiffSTR. The pipeline consists of two stages, the stage one is MAE pretraining for conditioning utilising masks from other methods and MRF block. The obtained results are utilised in stage two with PBE base is used to train the ControlNet with conditioning inputs resulting in $x_{g}$ output image
• Figure 3: Qualitative results and ablation of segmentation masks generated by our novel MRF module. The left three columns show the SOTA masking approaches while the blue box on the right shows ablation and final results of MRF module
• Figure 4: Qualitative comparison of our proposed scene text removal framework with state-of-the-art on SCUT-EnsText dataset.
• Figure 5: Qualitative comparison of ablation on three modules (diffusion baseline , TR-MAE and MRF)