FS-DAG: Few Shot Domain Adapting Graph Networks for Visually Rich Document Understanding

TL;DR

The paper tackles visually rich document understanding (VRDU) under data scarcity by introducing FS-DAG, a modular few-shot graph-based framework that integrates domain-specific textual and visual backbones with a graph neural network for Key Information Extraction. It jointly fuses textual and visual features via Kronecker fusion, applies shared position embeddings and multi-head attention in the GNN, and uses carefully designed training strategies to achieve robust few-shot adaptation with under 90M parameters. Extensive experiments on two industry-focused VRDU datasets show FS-DAG achieving state-of-the-art or competitive F1 scores while reducing model size and latency compared to LayoutLMv2/v3 and other graph models, and demonstrating strong robustness to OCR errors. The work also provides ablation evidence that each architectural and training component contributes to performance, supports deployment by industry with 50+ customers and 1M+ API calls monthly, and points toward zero-shot extensions as future work.

Abstract

In this work, we propose Few Shot Domain Adapting Graph (FS-DAG), a scalable and efficient model architecture for visually rich document understanding (VRDU) in few-shot settings. FS-DAG leverages domain-specific and language/vision specific backbones within a modular framework to adapt to diverse document types with minimal data. The model is robust to practical challenges such as handling OCR errors, misspellings, and domain shifts, which are critical in real-world deployments. FS-DAG is highly performant with less than 90M parameters, making it well-suited for complex real-world applications for Information Extraction (IE) tasks where computational resources are limited. We demonstrate FS-DAG's capability through extensive experiments for information extraction task, showing significant improvements in convergence speed and performance compared to state-of-the-art methods. Additionally, this work highlights the ongoing progress in developing smaller, more efficient models that do not compromise on performance. Code : https://github.com/oracle-samples/fs-dag

Figures (3)

• Figure 1: An illustration of the model architecture for FS-DAG. Given a document image (I); its text regions $\{r_i\}$ are extracted using an OCR engine. We cluster and sort the $\{r_i\}$ to create a reading sequence $\{s\}$; textual features $\{t_i\}$ are extracted using a linear projection layer on top of a pre-trained language model processing $\{s\}$. In contrast, visual features $\{v_i\}$ are extracted using ROI-Align on top of the feature map from the Visual Model and $\{r_i\}$. The deep fusion module uses Kronecker product to fuse $\{t_i\}$ and $\{v_i\}$ to initialize the node features $\{n_i\}$. The node features are propagated and aggregated in the GNN during the message passing, which uses positional embedding $\{p_i\}$ and multi-head attention to learn the edge features dynamically. The classification head will finally classify the node features into one of the key-value classes.
• Figure 2: Sample images from each of the five document types released as part of the Category 1 dataset.
• Figure 3: Sample images from each of the seven document types released as part of the Category 2 dataset.