CM1 -- A Dataset for Evaluating Few-Shot Information Extraction with Large Vision Language Models

TL;DR

The CM1 paper tackles the challenge of extracting key-value information from handwritten forms in historic archives when labeled data are scarce. It introduces CM1, a dataset derived from postwar Care and Maintenance records, with three benchmarks (CM1-COVER, CM1-NAME, CM1-DATE) and well-defined few-shot training subsets, enabling rigorous evaluation of end-to-end information extraction. The study benchmarks a traditional full-page extraction model (DONUT) against open-weight LVLMs (PaliGemma and Qwen2.5-VL) using PEFT techniques (LoRA/QLoRA), with DONUT having ~$201\times 10^{6}$ parameters, PaliGemma ~${3\times 10^{9}}$, and Qwen2.5-VL pre-trained on $18\times 10^{12}$ tokens. Results show LVLMs provide strong few-shot performance, often outperforming the classical approach at 1–25% training data, while full fine-tuning still yields the best results with abundant data; zero-shot performance is generally inadequate for handwriting-heavy historic collections. Overall, CM1 provides a reproducible framework for evaluating few-shot information extraction on archival documents and highlights the practical potential of LVLMs in digitizing large-scale handwritten corpora, especially under limited supervision.

Abstract

The automatic extraction of key-value information from handwritten documents is a key challenge in document analysis. A reliable extraction is a prerequisite for the mass digitization efforts of many archives. Large Vision Language Models (LVLM) are a promising technology to tackle this problem especially in scenarios where little annotated training data is available. In this work, we present a novel dataset specifically designed to evaluate the few-shot capabilities of LVLMs. The CM1 documents are a historic collection of forms with handwritten entries created in Europe to administer the Care and Maintenance program after World War Two. The dataset establishes three benchmarks on extracting name and birthdate information and, furthermore, considers different training set sizes. We provide baseline results for two different LVLMs and compare performances to an established full-page extraction model. While the traditional full-page model achieves highly competitive performances, our experiments show that when only a few training samples are available the considered LVLMs benefit from their size and heavy pretraining and outperform the classical approach.

Figures (5)

• Figure 1: CM1-COVER example image with corresponding JSON annotation.
• Figure 2: Distribution over the number of people on the cover pages. The proposed dataset only includes examples with four or less individuals.
• Figure 3: Example of different document pages, containing the birthdate, first and lastname of the applicant.
• Figure 4: Left: Structure of the autoencoder that is used to generate characteristic visual embeddings. Right: Examples of image reconstruction.
• Figure 5: Example of a common name template. First- and lastname are given separately. The firstname of the main applicant occurs in a table among other family members.