Image-based Freeform Handwriting Authentication with Energy-oriented Self-Supervised Learning

TL;DR

This work tackles robust freeform handwriting authentication under severe damage, high-dimensional features, and limited supervision. It introduces SherlockNet, an energy-oriented two-branch contrastive self-supervised framework with four stages: pre-processing via an energy operator, generalized pre-training with adaptive patch-weighted contrastive learning, personalized fine-tuning on few labels, and practical deployment through modular APIs. A new EN-HA dataset simulates forgery and damage to mirror real-world conditions, and extensive experiments across six benchmarks demonstrate strong robustness and efficiency, often surpassing state-of-the-art baselines without requiring annotated data. The approach enables accurate writer verification in messy, unconstrained handwriting scenarios and offers practical deployment potential for archival, security, and forensic applications.

Abstract

Freeform handwriting authentication verifies a person's identity from their writing style and habits in messy handwriting data. This technique has gained widespread attention in recent years as a valuable tool for various fields, e.g., fraud prevention and cultural heritage protection. However, it still remains a challenging task in reality due to three reasons: (i) severe damage, (ii) complex high-dimensional features, and (iii) lack of supervision. To address these issues, we propose SherlockNet, an energy-oriented two-branch contrastive self-supervised learning framework for robust and fast freeform handwriting authentication. It consists of four stages: (i) pre-processing: converting manuscripts into energy distributions using a novel plug-and-play energy-oriented operator to eliminate the influence of noise; (ii) generalized pre-training: learning general representation through two-branch momentum-based adaptive contrastive learning with the energy distributions, which handles the high-dimensional features and spatial dependencies of handwriting; (iii) personalized fine-tuning: calibrating the learned knowledge using a small amount of labeled data from downstream tasks; and (iv) practical application: identifying individual handwriting from scrambled, missing, or forged data efficiently and conveniently. Considering the practicality, we construct EN-HA, a novel dataset that simulates data forgery and severe damage in real applications. Finally, we conduct extensive experiments on six benchmark datasets including our EN-HA, and the results prove the robustness and efficiency of SherlockNet.

Figures (11)

• Figure 1: Handwriting authentication vs. freeform handwriting authentication. Compared with previous handwriting authentication, the challenging FHA requires a model: (i) not restricting data quality; (ii) not constraining handwriting content; and (iii) not relying on supervision information.
• Figure 2: Handwriting defects and pre-processing results. (a) the common defects and damages of the collected manuscripts; (b) samples of the collected handwriting data in the real world; (c) samples after pre-processing with 2 steps.
• Figure 3: The framework of the proposed SherlockNet with four stages, i.e., pre-processing stage (b), generalized pre-training stage (a), personalized fine-tuning stage (c), and practical application stage (d).
• Figure 4: Adaptive matching mechanism. The task-related patches are determined by reweighting all patches based on their contribution towards a correct classification result. In this process, the left steps, i.e., reweight important patches, aim to increase the influence (weight) of important patches, while the right steps, i.e., remove importance, take into account the differences of key patches in homologous augmented samples.
• Figure 5: Examples of the six benchmark datasets used in the experiments, including IAM marti1999full, CEDAR srihari2002individuality, CVL kleber2013cvl, QUWI al2012quwi, ICDAR2013 hassaine2013icdar, and our own constructed dataset EN-HA.