CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Model

TL;DR

The paper addresses scene text recognition (STR) and its limitations with single-modality backbones by leveraging pre-trained vision-language models, specifically CLIP. It introduces CLIP4STR, a dual-branch architecture that combines a visual and a cross-modal decoder with permuted sequence modeling and a dual predict-and-refine decoding scheme, achieving state-of-the-art results across 13 STR benchmarks, including challenging irregular text. An extensive empirical study analyzes training strategies, freezing decisions, and PEFT options, showing that full fine-tuning of the CLIP image encoder yields the best performance and that the cross-modal branch provides additional gains. The work demonstrates that a simple, well-structured VLM-based baseline can robustly handle regular and irregular text in STR and offers a practical, reproducible approach for future research in this area.

Abstract

Pre-trained vision-language models~(VLMs) are the de-facto foundation models for various downstream tasks. However, scene text recognition methods still prefer backbones pre-trained on a single modality, namely, the visual modality, despite the potential of VLMs to serve as powerful scene text readers. For example, CLIP can robustly identify regular (horizontal) and irregular (rotated, curved, blurred, or occluded) text in images. With such merits, we transform CLIP into a scene text reader and introduce CLIP4STR, a simple yet effective STR method built upon image and text encoders of CLIP. It has two encoder-decoder branches: a visual branch and a cross-modal branch. The visual branch provides an initial prediction based on the visual feature, and the cross-modal branch refines this prediction by addressing the discrepancy between the visual feature and text semantics. To fully leverage the capabilities of both branches, we design a dual predict-and-refine decoding scheme for inference. We scale CLIP4STR in terms of the model size, pre-training data, and training data, achieving state-of-the-art performance on 13 STR benchmarks. Additionally, a comprehensive empirical study is provided to enhance the understanding of the adaptation of CLIP to STR. Our method establishes a simple yet strong baseline for future STR research with VLMs.

Figures (6)

• Figure 1: Zero-shot classification results of CLIP-ViT-B/32. CLIP can perceive and understand text in images, even for irregular text with noise, rotation, and occlusion. CLIP is potentially a powerful scene text recognition expert.
• Figure 2: Attention of CLIP-ViT-B/32 for STR images.
• Figure 3: The framework of CLIP4STR. It has a visual branch and a cross-modal branch. The cross-modal branch refines the prediction of the visual branch for the final output. The text encoder is partially frozen.
• Figure 4: The decoder of CLIP4STR. [B], [E], and [P] are the beginning, end, and padding tokens, respectively. '$[\cdots]$' in prediction represents the ignored outputs. Layer normalization 2016_ln and dropout 2014_dropout in the decoder are ignored.
• Figure 5: CLIP-Adapter (left) and LST (right).