Attention based End to end network for Offline Writer Identification on Word level data

TL;DR

This work tackles offline writer identification from word images when handwriting samples are scarce. It introduces a dual-stream CNN that processes word fragments through writer-dependent and writer-independent modules, with an attention mechanism to highlight informative regions, and an end-to-end training regime that aggregates fragment scores for writer prediction. The approach leverages transfer learning for the WI branch (pre-trained on Omniglot) and fragment-driven training to capture both local and global writer cues, showing strong performance on IAM, CVL, and CERUG-EN. The findings indicate that fragment-based, attention-guided, end-to-end models can outperform traditional word-based methods in low-data regimes and offer competitive results against recent end-to-end architectures, especially when position encoding is not used. This has practical implications for forensic and historical handwriting analysis where per-writer samples are limited and rapid identification is needed.

Abstract

Writer identification due to its widespread application in various fields has gained popularity over the years. In scenarios where optimum handwriting samples are available, whether they be in the form of a single line, a sentence, or an entire page, writer identification algorithms have demonstrated noteworthy levels of accuracy. However, in scenarios where only a limited number of handwritten samples are available, particularly in the form of word images, there is a significant scope for improvement. In this paper, we propose a writer identification system based on an attention-driven Convolutional Neural Network (CNN). The system is trained utilizing image segments, known as fragments, extracted from word images, employing a pyramid-based strategy. This methodology enables the system to capture a comprehensive representation of the data, encompassing both fine-grained details and coarse features across various levels of abstraction. These extracted fragments serve as the training data for the convolutional network, enabling it to learn a more robust representation compared to traditional convolution-based networks trained on word images. Additionally, the paper explores the integration of an attention mechanism to enhance the representational power of the learned features. The efficacy of the proposed algorithm is evaluated on three benchmark databases, demonstrating its proficiency in writer identification tasks, particularly in scenarios with limited access to handwriting data.

Figures (5)

• Figure 1: The architecture of dual-stream CNN model used for training the fragments of the words. Each convolution block and its variant (represented as Conv2D/Separable Conv2D) is followed by entries signifying the number of filters and their kernel size, respectively. The sub-figure (a) represents the overall architecture of the proposed dual stream network. Sub-figure (b) depicts the architecture of the writer-dependent module and the writer independent module. However, for pre-training of the writer dependent module in a Siamese framework, we consider augmenting the blocks of global average pooling, dropout, and Fully Connected layers after the last residual network. Sub-figure (c) represents the structure of the Residual block used in the writer dependent and independent modules, and while (d) presents the architecture of the classification block.
• Figure 2: Block diagram representing different configurations of attention module. In Figure (a) the attention module is incorporated separately to the writer dependent and independent modules. In Figure (b), the attention module is applied after the integration of the feature maps.
• Figure 3: Block diagram representing configurations of the attention module.
• Figure 4: Examples of training samples in (a) CVL database, (b) IAM database, and (c) CERUG-EN database.
• Figure 5: Visualization of heatmap activation across different network configurations. Rows 1–3 correspond to a word sample from the the IAM, CVL, and CERUG-EN datasets, respectively. Column 2 shows the heatmap produced by the WD module trained on complete word images, while columns 3–5 display heatmaps obtained from networks trained with the (i) the WD module, (ii) combined (WD+WI+Concat) module without attention and (iii) combined (WD+WI+Concat) module with the attention configuration of Fig \ref{['attention_config']}(b), respectively.