Enhancing Document Key Information Localization Through Data Augmentation

TL;DR

The paper tackles key-information localization in document images under a cross-domain setting where training relies on digital documents but test data include handwritten samples. It presents a simple yet effective pipeline that augments digital documents with handwriting-like appearances using Augraphy, followed by an object-detection stage with diverse backbones (e.g., ResNet-101, DiT, LayoutLMv3). Experimental results show that the augmentation strategy improves handwritten-domain generalization for three of four backbones, with notable differences between backbones pre-trained on document versus natural images, and OCR-related challenges impacting multimodal models. The work demonstrates a practical, data-efficient approach to cross-domain document localization, with potential for further gains via unsupervised domain adaptation techniques.

Abstract

The Visually Rich Form Document Intelligence and Understanding (VRDIU) Track B focuses on the localization of key information in document images. The goal is to develop a method capable of localizing objects in both digital and handwritten documents, using only digital documents for training. This paper presents a simple yet effective approach that includes a document augmentation phase and an object detection phase. Specifically, we augment the training set of digital documents by mimicking the appearance of handwritten documents. Our experiments demonstrate that this pipeline enhances the models' generalization ability and achieves high performance in the competition.

Figures (1)

• Figure 1: An example of document augmentation from clean digital style to handwritten scan style