Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Few-shot Entity Recognition in Document Images: A Graph Neural Network Approach Robust to Image Manipulation

Prashant Krishnan, Zilong Wang, Yangkun Wang, Jingbo Shang

TL;DR

This work tackles few-shot entity recognition in document images, where coordinate-based layout cues are vulnerable to affine manipulations. It introduces LAGER, which integrates topological token relations as graphs atop a pre-trained layout-aware LM (LayoutLMv2 or LayoutLMv3) and processes them with a Graph Attention Network to produce enhanced embeddings for sequence labeling. Across FUNSD and CORD, LAGER variants consistently outperform strong baselines in few-shot settings and show superior robustness to shifting, rotation, and scaling of images. The approach offers a practical and data-efficient enhancement for document image understanding, with plans to release code and extend the framework to additional backbones and semantic relations.

Abstract

Recent advances of incorporating layout information, typically bounding box coordinates, into pre-trained language models have achieved significant performance in entity recognition from document images. Using coordinates can easily model the absolute position of each token, but they might be sensitive to manipulations in document images (e.g., shifting, rotation or scaling), especially when the training data is limited in few-shot settings. In this paper, we propose to further introduce the topological adjacency relationship among the tokens, emphasizing their relative position information. Specifically, we consider the tokens in the documents as nodes and formulate the edges based on the topological heuristics from the k-nearest bounding boxes. Such adjacency graphs are invariant to affine transformations including shifting, rotations and scaling. We incorporate these graphs into the pre-trained language model by adding graph neural network layers on top of the language model embeddings, leading to a novel model LAGER. Extensive experiments on two benchmark datasets show that LAGER significantly outperforms strong baselines under different few-shot settings and also demonstrate better robustness to manipulations.

Towards Few-shot Entity Recognition in Document Images: A Graph Neural Network Approach Robust to Image Manipulation

TL;DR

This work tackles few-shot entity recognition in document images, where coordinate-based layout cues are vulnerable to affine manipulations. It introduces LAGER, which integrates topological token relations as graphs atop a pre-trained layout-aware LM (LayoutLMv2 or LayoutLMv3) and processes them with a Graph Attention Network to produce enhanced embeddings for sequence labeling. Across FUNSD and CORD, LAGER variants consistently outperform strong baselines in few-shot settings and show superior robustness to shifting, rotation, and scaling of images. The approach offers a practical and data-efficient enhancement for document image understanding, with plans to release code and extend the framework to additional backbones and semantic relations.

Abstract

Recent advances of incorporating layout information, typically bounding box coordinates, into pre-trained language models have achieved significant performance in entity recognition from document images. Using coordinates can easily model the absolute position of each token, but they might be sensitive to manipulations in document images (e.g., shifting, rotation or scaling), especially when the training data is limited in few-shot settings. In this paper, we propose to further introduce the topological adjacency relationship among the tokens, emphasizing their relative position information. Specifically, we consider the tokens in the documents as nodes and formulate the edges based on the topological heuristics from the k-nearest bounding boxes. Such adjacency graphs are invariant to affine transformations including shifting, rotations and scaling. We incorporate these graphs into the pre-trained language model by adding graph neural network layers on top of the language model embeddings, leading to a novel model LAGER. Extensive experiments on two benchmark datasets show that LAGER significantly outperforms strong baselines under different few-shot settings and also demonstrate better robustness to manipulations.
Paper Structure (20 sections, 5 figures, 8 tables)

This paper contains 20 sections, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Task Formulation
  5. Pre-trained LM as Backbone
  6. Heuristic-based Graph Construction
  7. Graph Modeling with GAT
  8. Image manipulations
  9. Experiments
  10. Datasets
  11. Baselines
  12. Implementation Details
  13. Few-shot Experimental results
  14. Experiments with image manipulation
  15. Case studies
  16. ...and 5 more sections

Figures (5)

  • Figure 1: The Framework for LAGER. Two variants of the model are used based on the heuristic for graph construction. $M$ denotes the number of GATs used. $M=1$ for k-nearest neighbors in space approach. For the the k-nearest neighbors at angles, $M = \lfloor 360 \degree/\theta \rfloor$ and we construct graphs for $[\theta, 2\theta ... M\theta]$.
  • Figure 2: Heuristics for graph construction.
  • Figure 3: Representative examples of the image manipulations
  • Figure 4: Case studies from FUNSD. RGB]0,0,255Bl, RGB]0,255,255Bl, RGB]255,0,0Bl, RGB]255,165,0Bl, RGB]238,130,238Bl denote B-ANSWER, I-ANSWER, B-QUESTION, I-QUESTION and OTHER respectively.
  • Figure 5: Case studies from CORD. RGB]0,0,255Bl, RGB]255, 165, 0Bl, RGB]0, 255, 255Bl, denote the tags for TOTAL.TOTAL_PRICETOTAL.CASHPRICE and TOTAL.CHANGEPRICE respectively.