Vision-Language Model Based Handwriting Verification

TL;DR

This work investigates the use of Vision-Language Models (GPT-4o VLM and PaliGemma) for handwriting verification in forensics, aiming to improve interpretability via Visual Question Answering and 0-shot Chain-of-Thought prompts. The study compares VLMs to traditional baselines (GSC features, ResNet-18, ViT) on 1,000 sample pairs from the CEDAR AND and CEDAR Letter datasets and includes a limited supervised fine-tuning of PaliGemma with LoRA on a small curated set. Results show that CNN-based ResNet-18 achieves higher accuracy (≈84% on CEDAR AND) than the VLMs (≈70–71%) and that 0-shot VLMs offer clearer, human-readable explanations, though with lower predictive performance. The findings highlight the promise of explainable VLM-based handwriting verification and outline a path toward bridging the performance gap through domain-specific fine-tuning and collaboration with forensic experts.

Abstract

Handwriting Verification is a critical in document forensics. Deep learning based approaches often face skepticism from forensic document examiners due to their lack of explainability and reliance on extensive training data and handcrafted features. This paper explores using Vision Language Models (VLMs), such as OpenAI's GPT-4o and Google's PaliGemma, to address these challenges. By leveraging their Visual Question Answering capabilities and 0-shot Chain-of-Thought (CoT) reasoning, our goal is to provide clear, human-understandable explanations for model decisions. Our experiments on the CEDAR handwriting dataset demonstrate that VLMs offer enhanced interpretability, reduce the need for large training datasets, and adapt better to diverse handwriting styles. However, results show that the CNN-based ResNet-18 architecture outperforms the 0-shot CoT prompt engineering approach with GPT-4o (Accuracy: 70%) and supervised fine-tuned PaliGemma (Accuracy: 71%), achieving an accuracy of 84% on the CEDAR AND dataset. These findings highlight the potential of VLMs in generating human-interpretable decisions while underscoring the need for further advancements to match the performance of specialized deep learning models.

Figures (1)

• Figure 1: Overall process of evaluating Vision Language Model for Handwriting Verification against Baselines.