Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TEXTS-Diff: TEXTS-Aware Diffusion Model for Real-World Text Image Super-Resolution

Haodong He, Xin Zhan, Yancheng Bai, Rui Lan, Lei Sun, Xiangxiang Chu

TL;DR

Real-world text image SR faces challenges from diverse degradations and limited text-rich data. The authors address this by building Real-Texts, a large bilingual dataset, and proposing TEXTS-Diff, a TEXTS-aware one-step diffusion model that integrates abstract text concepts and concrete text-region cues to guide SR. TEXTS-Diff leverages a CLIP-based TEXTS representation and a text-detection module to produce text-aware guidance, coupled with a FE module to preserve high-frequency details and a LoRA-tuned SD2.1 backbone trained with a multi-term loss that includes edge and ODM terms. Experiments show state-of-the-art OCR accuracy on Real-Texts and Real-CE while maintaining overall image quality, and ablations confirm the benefit of both abstract and concrete TEXTS cues for robust text restoration in real scenes. The approach offers practical improvements for multilingual text restoration in the wild and providing resources for future research and deployment.

Abstract

Real-world text image super-resolution aims to restore overall visual quality and text legibility in images suffering from diverse degradations and text distortions. However, the scarcity of text image data in existing datasets results in poor performance on text regions. In addition, datasets consisting of isolated text samples limit the quality of background reconstruction. To address these limitations, we construct Real-Texts, a large-scale, high-quality dataset collected from real-world images, which covers diverse scenarios and contains natural text instances in both Chinese and English. Additionally, we propose the TEXTS-Aware Diffusion Model (TEXTS-Diff) to achieve high-quality generation in both background and textual regions. This approach leverages abstract concepts to improve the understanding of textual elements within visual scenes and concrete text regions to enhance textual details. It mitigates distortions and hallucination artifacts commonly observed in text regions, while preserving high-quality visual scene fidelity. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple evaluation metrics, exhibiting superior generalization ability and text restoration accuracy in complex scenarios. All the code, model, and dataset will be released.

TEXTS-Diff: TEXTS-Aware Diffusion Model for Real-World Text Image Super-Resolution

TL;DR

Real-world text image SR faces challenges from diverse degradations and limited text-rich data. The authors address this by building Real-Texts, a large bilingual dataset, and proposing TEXTS-Diff, a TEXTS-aware one-step diffusion model that integrates abstract text concepts and concrete text-region cues to guide SR. TEXTS-Diff leverages a CLIP-based TEXTS representation and a text-detection module to produce text-aware guidance, coupled with a FE module to preserve high-frequency details and a LoRA-tuned SD2.1 backbone trained with a multi-term loss that includes edge and ODM terms. Experiments show state-of-the-art OCR accuracy on Real-Texts and Real-CE while maintaining overall image quality, and ablations confirm the benefit of both abstract and concrete TEXTS cues for robust text restoration in real scenes. The approach offers practical improvements for multilingual text restoration in the wild and providing resources for future research and deployment.

Abstract

Real-world text image super-resolution aims to restore overall visual quality and text legibility in images suffering from diverse degradations and text distortions. However, the scarcity of text image data in existing datasets results in poor performance on text regions. In addition, datasets consisting of isolated text samples limit the quality of background reconstruction. To address these limitations, we construct Real-Texts, a large-scale, high-quality dataset collected from real-world images, which covers diverse scenarios and contains natural text instances in both Chinese and English. Additionally, we propose the TEXTS-Aware Diffusion Model (TEXTS-Diff) to achieve high-quality generation in both background and textual regions. This approach leverages abstract concepts to improve the understanding of textual elements within visual scenes and concrete text regions to enhance textual details. It mitigates distortions and hallucination artifacts commonly observed in text regions, while preserving high-quality visual scene fidelity. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple evaluation metrics, exhibiting superior generalization ability and text restoration accuracy in complex scenarios. All the code, model, and dataset will be released.
Paper Structure (18 sections, 7 equations, 4 figures, 3 tables)

This paper contains 18 sections, 7 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Real-Texts Dataset
  3. Dataset Creation Process
  4. Dataset Analysis
  5. Methodology
  6. Overview Architecture
  7. TEXTS-Aware Guidance
  8. Diffusion Processing
  9. Loss Function
  10. Experiments
  11. Experimental Settings
  12. Dataset and Details
  13. Evaluation Metrics
  14. Experimental Results
  15. Quantitative Comparison
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Example images from other datasets and our Real-Texts dataset. Our dataset includes Chinese and English text across a wide range of scene types and font styles.
  • Figure 2: Overview of TEXTS-Diff. The framework captures text-aware visual cues to guide the diffusion process through abstract and concrete perception processes.
  • Figure 3: Visual comparison on the Real-Texts test dataset. The red box means the poor result areas.
  • Figure 4: Visual comparison on the Real-CE validation dataset. The enlarged part of the dashed red box is shown under the original results. Their corresponding OCR recognition results in the aforementioned figure, where erroneous text is highlighted in red and omitted text is marked in blue.