IPAD: Iterative, Parallel, and Diffusion-based Network for Scene Text Recognition

TL;DR

This paper regards text recognition as an image-based conditional text generation task and utilize the discrete diffusion strategy, ensuring exhaustive exploration of bidirectional contextual information.

Abstract

Nowadays, scene text recognition has attracted more and more attention due to its diverse applications. Most state-of-the-art methods adopt an encoder-decoder framework with the attention mechanism, autoregressively generating text from left to right. Despite the convincing performance, this sequential decoding strategy constrains the inference speed. Conversely, non-autoregressive models provide faster, simultaneous predictions but often sacrifice accuracy. Although utilizing an explicit language model can improve performance, it burdens the computational load. Besides, separating linguistic knowledge from vision information may harm the final prediction. In this paper, we propose an alternative solution that uses a parallel and iterative decoder that adopts an easy-first decoding strategy. Furthermore, we regard text recognition as an image-based conditional text generation task and utilize the discrete diffusion strategy, ensuring exhaustive exploration of bidirectional contextual information. Extensive experiments demonstrate that the proposed approach achieves superior results on the benchmark datasets, including both Chinese and English text images.

Figures (5)

• Figure 1: Comparison between the proposed IPAD and our conference version PIMNet qiao2021pimnet. For IPAD, the ViT-based encoder first extracts the visual feature, and then the parallel decoder adopts an iterative generation to extract the context information from the previous predictions. The discrete diffusion strategy is used during training, where the noising process adds noise by randomly sampling a timestep and then the parallel decoder works as the denoising network, which directly predicts the clean text without any iteration.
• Figure 2: An illustration of the easy-first decoding strategy. In each iteration, the characters in black represent the characters with high confidence and will be reserved in the next iteration. The characters with low confidence will be replaced with the $\langle {\rm MASK}\rangle$ token and re-predicted again based on the other reachable predictions
• Figure 3: Comparison of normalized cosine similarities of FFN outputs across different encoders. $^\dagger$ indicates models using a CNN encoder. $_\text{w/o}$ denotes models without mimicking or diffusion steps.
• Figure 4: Comparison of recognition results on the scene subset of the BCTR dataset. The characters in red are wrongly recognized. The red "_" means missing characters.
• Figure 5: Some examples to illustrate the iterative generation of IPAD with easy first.