Contrastive Masked Autoencoders for Character-Level Open-Set Writer Identification

TL;DR

This work tackles open-set writer identification for online handwriting by introducing CMAE, a framework that fuses masked autoencoding with supervised contrastive learning to learn discriminative, trajectory-based representations from single characters. The CMAE encoder processes masked trajectory patches, the reconstruction decoder restores masked parts, and the contrastive module enforces writer-discriminative embeddings, all regulated by a discriminator for pairwise writer prediction. Key contributions include the single-character open-set formulation, integration of MAE and CL, and state-of-the-art results on CASIA-OLHWDB (89.7% precision with a 0.15 mask ratio), along with thorough ablations on mask effects, pretraining, and trajectory length. This approach advances forensic handwriting analysis by delivering robust, generalizable representations suitable for real-world open-set scenarios and diverse scripts.

Abstract

In the realm of digital forensics and document authentication, writer identification plays a crucial role in determining the authors of documents based on handwriting styles. The primary challenge in writer-id is the "open-set scenario", where the goal is accurately recognizing writers unseen during the model training. To overcome this challenge, representation learning is the key. This method can capture unique handwriting features, enabling it to recognize styles not previously encountered during training. Building on this concept, this paper introduces the Contrastive Masked Auto-Encoders (CMAE) for Character-level Open-Set Writer Identification. We merge Masked Auto-Encoders (MAE) with Contrastive Learning (CL) to simultaneously and respectively capture sequential information and distinguish diverse handwriting styles. Demonstrating its effectiveness, our model achieves state-of-the-art (SOTA) results on the CASIA online handwriting dataset, reaching an impressive precision rate of 89.7%. Our study advances universal writer-id with a sophisticated representation learning approach, contributing substantially to the ever-evolving landscape of digital handwriting analysis, and catering to the demands of an increasingly interconnected world.

Figures (4)

• Figure 1: This figure depicts the process for open-set scenarios, highlighting the distinction between the training and testing phases. Training is performed using data from one group, while testing is done with individuals not seen during training. Our method involves training the model by creating pairs of trajectories, $\tau_{ni}$ and $\tau_{nj}$, from the same writer $n$. This technique is intended to enable the model to learn the similarities within each writer's handwriting and the differences between writers. Testing is then carried out on data from unknown writers $m$. The middle subfigure illustrates the normalization process in our method, which scales the character trajectory to the range (0,1].
• Figure 2: Illustration of the CMAE model's training methodology. Left: Sample images from CASIA-OLHWDB. Right: Depiction of the CMAE encoder's method for extracting features from handwriting character trajectories. The CMAE encoder receives unmasked segmented patches as input, depicted as a red curve in the character "geese", which are randomly chosen and governed by the mask ratio.
• Figure 3: Schematic of Our Proposed Model. STEP1: This framework starts by processing point sequences of single characters (including English, Chinese, and symbols). For instance, the trajectory for "R" (depicted at the top in the figure) consists of coordinates $(x_i,y_i)$ at each time $t_i$, along with special tokens for pen down, pen up, and padding (not shown in this figure). The "R" is then segmented into patches and subjected to random masking. Subsequently, it passes through a transformer-based encoder, which extracts and maps the trajectory features of "R" into an embedding space. These features are then directed into two distinct modules: the CL and the MAE. The MAE decoder reconstructs the masked patches using the unmasked ones, while the CL adjusts the distance between two embeddings $(a^{\prime}, a^{\prime\prime})$ of trajectories, either from the same or different individuals. STEP2: The embedding $a^{\prime}$ and $a^{\prime\prime}$ are then processed through a transformer block followed by an MLP, culminating in the computation of CrossEntropy Loss.
• Figure 4: A heatmap illustrating the differences in distance between two handwriting trajectories of varying lengths from the same person. The heatmap's x and y axes range from 20 to 200 timepoints, representing the lengths of the handwriting trajectories. The color scale on the heatmap indicates the average Euclidean distance between two embeddings of these trajectories, derived from the same individual, using the CMAE encoder.