Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HTR-VT: Handwritten Text Recognition with Vision Transformer

Yuting Li, Dexiong Chen, Tinglong Tang, Xi Shen

TL;DR

A data-efficient ViT method that uses only the encoder of the standard transformer, incorporating a Convolutional Neural Network for feature extraction instead of the original patch embedding and employing Sharpness-Aware Minimization (SAM) optimizer to ensure that the model can converge towards flatter minima and yield notable enhancements.

Abstract

We explore the application of Vision Transformer (ViT) for handwritten text recognition. The limited availability of labeled data in this domain poses challenges for achieving high performance solely relying on ViT. Previous transformer-based models required external data or extensive pre-training on large datasets to excel. To address this limitation, we introduce a data-efficient ViT method that uses only the encoder of the standard transformer. We find that incorporating a Convolutional Neural Network (CNN) for feature extraction instead of the original patch embedding and employ Sharpness-Aware Minimization (SAM) optimizer to ensure that the model can converge towards flatter minima and yield notable enhancements. Furthermore, our introduction of the span mask technique, which masks interconnected features in the feature map, acts as an effective regularizer. Empirically, our approach competes favorably with traditional CNN-based models on small datasets like IAM and READ2016. Additionally, it establishes a new benchmark on the LAM dataset, currently the largest dataset with 19,830 training text lines. The code is publicly available at: https://github.com/YutingLi0606/HTR-VT.

HTR-VT: Handwritten Text Recognition with Vision Transformer

TL;DR

A data-efficient ViT method that uses only the encoder of the standard transformer, incorporating a Convolutional Neural Network for feature extraction instead of the original patch embedding and employing Sharpness-Aware Minimization (SAM) optimizer to ensure that the model can converge towards flatter minima and yield notable enhancements.

Abstract

We explore the application of Vision Transformer (ViT) for handwritten text recognition. The limited availability of labeled data in this domain poses challenges for achieving high performance solely relying on ViT. Previous transformer-based models required external data or extensive pre-training on large datasets to excel. To address this limitation, we introduce a data-efficient ViT method that uses only the encoder of the standard transformer. We find that incorporating a Convolutional Neural Network (CNN) for feature extraction instead of the original patch embedding and employ Sharpness-Aware Minimization (SAM) optimizer to ensure that the model can converge towards flatter minima and yield notable enhancements. Furthermore, our introduction of the span mask technique, which masks interconnected features in the feature map, acts as an effective regularizer. Empirically, our approach competes favorably with traditional CNN-based models on small datasets like IAM and READ2016. Additionally, it establishes a new benchmark on the LAM dataset, currently the largest dataset with 19,830 training text lines. The code is publicly available at: https://github.com/YutingLi0606/HTR-VT.
Paper Structure (35 sections, 4 equations, 5 figures, 10 tables)

This paper contains 35 sections, 4 equations, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Traditional approaches for Handwritten Text Recognition
  4. Transformer-based models for Handwritten Text Recognition
  5. Data-efficient Transformer for Handwritten Text Recognition
  6. Method
  7. Preliminary: Vision Transformer (ViT)
  8. ViT for handwritten text recognition
  9. CNN feature extractor
  10. Span feature masking strategy
  11. Sharpness-Aware Minimization (SAM)
  12. Implementation details
  13. Experiments
  14. Dataset and evaluation metrics
  15. LAM cascianelli2022lam
  16. ...and 20 more sections

Figures (5)

  • Figure 1: Architecture overview. Our approach encodes a text-line image into features using a CNN feature extractor. The transformer encoder takes these features as input tokens output character predictions. During the training, the span input tokens are replaced by learnable mask tokens. The entire model is optimized using CTC graves2006connectionist loss.
  • Figure 2: Visualization of attention maps with different masking strategies on IAM dataset. In the original image, we highlight the region corresponding to the token of interest with a red bounding box and average the attention across all heads. We observe that when no masking or random masking strategy is employed, each token focuses solely on its own information, as indicated by the illuminated regions in the image. However, when we apply the span masking strategy, a noticeable shift occurs, allowing the token to attend to a broader range of information.
  • Figure 3: Results on example lines from the IAM marti2002iam (First row), READ2016 sanchez2016icfhr2016 (Second row) and LAM cascianelli2022lam (Third row) of the best performing model.
  • Figure 4: Visual results on IAM marti2002iam
  • Figure 7: Visualization of attention maps