HTR-JAND: Handwritten Text Recognition with Joint Attention Network and Knowledge Distillation

TL;DR

HTR-JAND presents an efficient, end-to-end handwritten text recognition framework tailored for historical documents. It blends a CNN backbone with FullGatedConv2d and SE blocks, a Joint Combined Attention module (Multi-Head Self-Attention plus Proxima Attention), and a knowledge distillation pipeline to train a compact Student model from a high-capacity Teacher, all within a curriculum-driven, multi-dataset training regime. The approach is augmented by synthetic data, multi-task learning, and context-aware T5 post-processing, achieving state-of-the-art Character Error Rates of $1.23\%$, $1.02\%$, and $2.02\%$ on IAM, RIMES, and Bentham, respectively, while compressing parameters by 48% (to $0.75$M) and maintaining competitive accuracy. These findings demonstrate that integrated architectural innovations and disciplined training strategies can deliver both high recognition performance and practical deployment efficiency for diverse historical corpora. The work is supported by extensive ablations, attention analyses, and a detailed post-processing pipeline, underscoring the potential for robust, scalable HTR in digital humanities contexts.

Abstract

Despite significant advances in deep learning, current Handwritten Text Recognition (HTR) systems struggle with the inherent complexity of historical documents, including diverse writing styles, degraded text quality, and computational efficiency requirements across multiple languages and time periods. This paper introduces HTR-JAND (HTR-JAND: Handwritten Text Recognition with Joint Attention Network and Knowledge Distillation), an efficient HTR framework that combines advanced feature extraction with knowledge distillation. Our architecture incorporates three key components: (1) a CNN architecture integrating FullGatedConv2d layers with Squeeze-and-Excitation blocks for adaptive feature extraction, (2) a Combined Attention mechanism fusing Multi-Head Self-Attention with Proxima Attention for robust sequence modeling, and (3) a Knowledge Distillation framework enabling efficient model compression while preserving accuracy through curriculum-based training. The HTR-JAND framework implements a multi-stage training approach combining curriculum learning, synthetic data generation, and multi-task learning for cross-dataset knowledge transfer. We enhance recognition accuracy through context-aware T5 post-processing, particularly effective for historical documents. Comprehensive evaluations demonstrate HTR-JAND's effectiveness, achieving state-of-the-art Character Error Rates (CER) of 1.23\%, 1.02\%, and 2.02\% on IAM, RIMES, and Bentham datasets respectively. Our Student model achieves a 48\% parameter reduction (0.75M versus 1.5M parameters) while maintaining competitive performance through efficient knowledge transfer. Source code and pre-trained models are available at \href{https://github.com/DocumentRecognitionModels/HTR-JAND}{Github}.

Figures (7)

• Figure 1: Sample images from different datasets, demonstrating the range of challenges including writing style variability, non-standard character shapes, and contextual dependencies.
• Figure 2: Distribution of character frequencies across the combined datasets. Note the removal of infrequent characters such as '§', 'À', and 'òe'.
• Figure 3: Proposed HTR Model Architecture: Data flow through CNN feature extraction, LSTM sequence modeling, and Combined Attention mechanisms. Additionally, CTC Matrix for "Griffiths, M P for Manchester Exchange" showing probabilities for first "Gri-" and last "-nge" ('-' represents blank symbol for CTC alignment).
• Figure 4: Overview of our proposed knowledge distillation framework for handwritten text recognition (HTR).
• Figure 5: Visualization of the model's attention heatmaps for the sample predictions. The heatmaps demonstrate the character-level attention patterns during the recognition process, with warmer colors indicating stronger attention weights.