Self-supervised Pre-training of Text Recognizers

TL;DR

The paper tackles the challenge of reducing annotation demands in document text recognition by exploring self-supervised pre-training. It categorizes approaches into masked label prediction (Feature Quantization, VQ-VAE, Post-Quantized AE) and joint-embedding (VICReg, NT-Xent), augmented with an image-shifting technique to prevent collapse. Through experiments on historical Bentham handwriting and DTA printed data, it shows that pre-training on target-domain unlabeled data is beneficial but transfer learning from related domains often yields stronger results when available, with Feature Quantization delivering the best fine-tuned performance among SSL methods. The work provides practical insights, demonstrates strong non-supervised baselines, and shares code to spur further research in document text recognition.

Abstract

In this paper, we investigate self-supervised pre-training methods for document text recognition. Nowadays, large unlabeled datasets can be collected for many research tasks, including text recognition, but it is costly to annotate them. Therefore, methods utilizing unlabeled data are researched. We study self-supervised pre-training methods based on masked label prediction using three different approaches -- Feature Quantization, VQ-VAE, and Post-Quantized AE. We also investigate joint-embedding approaches with VICReg and NT-Xent objectives, for which we propose an image shifting technique to prevent model collapse where it relies solely on positional encoding while completely ignoring the input image. We perform our experiments on historical handwritten (Bentham) and historical printed datasets mainly to investigate the benefits of the self-supervised pre-training techniques with different amounts of annotated target domain data. We use transfer learning as strong baselines. The evaluation shows that the self-supervised pre-training on data from the target domain is very effective, but it struggles to outperform transfer learning from closely related domains. This paper is one of the first researches exploring self-supervised pre-training in document text recognition, and we believe that it will become a cornerstone for future research in this area. We made our implementation of the investigated methods publicly available at https://github.com/DCGM/pero-pretraining.

Figures (6)

• Figure 1: Overview of the investigated self-supervised pre-training approaches. The first row of images depicts methods based on predicting a label of a masked input. The second row shows joint-embedding methods.
• Figure 2: Illustration of the proposed shifting technique. If shifting is applied (right image), different positional encoding is added to the corresponding parts of the two input images. Without shifting (left image), the same positional encoding is added, resulting in training collapse.
• Figure 3: Examples of text lines from datasets.
• Figure 4: Examples of reconstructions using trained VQ-VAE and AE models on the DTA dataset (left column) and the Bentham dataset (right column). The first row shows the original images, the second and the third rows show reconstructions produced by VQ-VAE and AE models respectively. More reconstructions can be found in the GitHub repository.
• Figure 5: Examples of visually similar parts of images on the DTA dataset (left column) and the Bentham dataset (right column) extracted using quantization-based approaches. In figures (a)--(d) FQ stands for Feature Quantization and the dataset name in the parentheses indicates the dataset used to train the model from which the convolutional encoder is taken. More visualizations can be found in the GitHub repository.