HTR-ConvText: Leveraging Convolution and Textual Information for Handwritten Text Recognition

TL;DR

This work tackles handwritten text recognition under data-scarce conditions and complex diacritics by introducing HTR-ConvText, a compact hybrid architecture that unifies local convolutional bias with global self-attention. The model combines the MVP Block for rich feature extraction, a ConvText Encoder with a hierarchical U-Net for efficient global-local modeling, and a training-time Textual Context Module to inject linguistic priors and improve CTC-based decoding. Empirical results on IAM, READ2016, LAM, and HANDS-VNOnDB demonstrate state-of-the-art CER/WER, especially in low-resource or diacritic-dense scenarios, highlighting robust generalization and practical applicability. The work also provides extensive ablations and qualitative analyses, underscoring the effectiveness of the ConvText design, MVP integration, and contextual training strategy for real-world HTR systems.

Abstract

Handwritten Text Recognition remains challenging due to the limited data, high writing style variance, and scripts with complex diacritics. Existing approaches, though partially address these issues, often struggle to generalize without massive synthetic data. To address these challenges, we propose HTR-ConvText, a model designed to capture fine-grained, stroke-level local features while preserving global contextual dependencies. In the feature extraction stage, we integrate a residual Convolutional Neural Network backbone with a MobileViT with Positional Encoding block. This enables the model to both capture structural patterns and learn subtle writing details. We then introduce the ConvText encoder, a hybrid architecture combining global context and local features within a hierarchical structure that reduces sequence length for improved efficiency. Additionally, an auxiliary module injects textual context to mitigate the weakness of Connectionist Temporal Classification. Evaluations on IAM, READ2016, LAM and HANDS-VNOnDB demonstrate that our approach achieves improved performance and better generalization compared to existing methods, especially in scenarios with limited training samples and high handwriting diversity.

Figures (6)

• Figure 1: Overview of the HTR-ConvText architecture. The model consists of a CNN-based feature extractor (MVP Block), a hierarchical hybrid encoder (ConvText Encoder), and a training-only Textual Context Module (TCM).
• Figure 2: The MVP Block architecture, integrating MobileViT blocks with a Conditional Positional Encoding (CPE) into the ResNet backbone.
• Figure 3: Structure of the Hybrid ConvText Block. It sequentially interleaves Multi-Head Self-Attention (MHSA), a Feed-Forward Network (FFN), a Depthwise Convolution, and a final FFN.
• Figure 4: The Textual Context Module (TCM). It generates left and right linguistic context vectors and fuses them with visual features via cross-attention. This module is used only during training.
• Figure 5: Visualization of attention maps for the four model configurations (A--D). Brighter colors indicate stronger attention. The red boxes on the input images mark the tokens selected for visualization.