Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GeoContrastNet: Contrastive Key-Value Edge Learning for Language-Agnostic Document Understanding

Nil Biescas, Carlos Boned, Josep Lladós, Sanket Biswas

TL;DR

GeoContrastNet addresses language-agnostic document understanding by leveraging geometry-driven relationships between text entities in visually rich documents. It introduces a two-stage framework: Stage I learns robust geometric edge features via contrastive learning in a GNN, and Stage II grounds these features with a Graph Attention Network that fuses UNet-derived visual cues to predict semantic labels and key-value links. Evaluations on FUNSD and RVL-CDIP Invoices show competitive semantic labeling and strong edge-based link predictions, with ablations confirming the value of geometric edges and multimodal grounding. The approach offers a scalable, privacy-conscious path toward multilingual DU and points to future work in attention-based fusion and multilingual generalization.

Abstract

This paper presents GeoContrastNet, a language-agnostic framework to structured document understanding (DU) by integrating a contrastive learning objective with graph attention networks (GATs), emphasizing the significant role of geometric features. We propose a novel methodology that combines geometric edge features with visual features within an overall two-staged GAT-based framework, demonstrating promising results in both link prediction and semantic entity recognition performance. Our findings reveal that combining both geometric and visual features could match the capabilities of large DU models that rely heavily on Optical Character Recognition (OCR) features in terms of performance accuracy and efficiency. This approach underscores the critical importance of relational layout information between the named text entities in a semi-structured layout of a page. Specifically, our results highlight the model's proficiency in identifying key-value relationships within the FUNSD dataset for forms and also discovering the spatial relationships in table-structured layouts for RVLCDIP business invoices. Our code and pretrained models will be accessible on our official GitHub.

GeoContrastNet: Contrastive Key-Value Edge Learning for Language-Agnostic Document Understanding

TL;DR

GeoContrastNet addresses language-agnostic document understanding by leveraging geometry-driven relationships between text entities in visually rich documents. It introduces a two-stage framework: Stage I learns robust geometric edge features via contrastive learning in a GNN, and Stage II grounds these features with a Graph Attention Network that fuses UNet-derived visual cues to predict semantic labels and key-value links. Evaluations on FUNSD and RVL-CDIP Invoices show competitive semantic labeling and strong edge-based link predictions, with ablations confirming the value of geometric edges and multimodal grounding. The approach offers a scalable, privacy-conscious path toward multilingual DU and points to future work in attention-based fusion and multilingual generalization.

Abstract

This paper presents GeoContrastNet, a language-agnostic framework to structured document understanding (DU) by integrating a contrastive learning objective with graph attention networks (GATs), emphasizing the significant role of geometric features. We propose a novel methodology that combines geometric edge features with visual features within an overall two-staged GAT-based framework, demonstrating promising results in both link prediction and semantic entity recognition performance. Our findings reveal that combining both geometric and visual features could match the capabilities of large DU models that rely heavily on Optical Character Recognition (OCR) features in terms of performance accuracy and efficiency. This approach underscores the critical importance of relational layout information between the named text entities in a semi-structured layout of a page. Specifically, our results highlight the model's proficiency in identifying key-value relationships within the FUNSD dataset for forms and also discovering the spatial relationships in table-structured layouts for RVLCDIP business invoices. Our code and pretrained models will be accessible on our official GitHub.
Paper Structure (23 sections, 8 equations, 4 figures, 5 tables)

This paper contains 23 sections, 8 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Graph-based Document Representation
  5. Method Overview
  6. Stage -- I: Geometric Edge Feature Learning
  7. Stage -- II: Prediction Phase
  8. Semantic Entity Labeling.
  9. Entity Linking.
  10. Learning Objectives
  11. Experimental Validation
  12. Dataset Description
  13. Evaluation Metrics
  14. Implementation details
  15. Stage-I
  16. ...and 8 more sections

Figures (4)

  • Figure 1: Overall Architecture of the Proposed GeoContrastNet Framework.In Stage I, a GNN processes a document image represented as an attributed graph, learning key-value edge features in a contrastive setting using the triplet margin loss. Then, in Stage II, a GAT traverses these features to predict entity labels (nodes) and their relations (edges) as output.
  • Figure 2: Visual Illustration of Proposed Node and Edge Representation. Node features are defined by the bounding box coordinates, the area and regional encoding providing local and global structure position. Edge features provide information to geolocate each node relative to its neighbors.
  • Figure 3: Prediction on the link prediction task on FUNSD dataset. From (L to R) GT and predicted images respectively.
  • Figure 4: Prediction on the link prediction task on RVL-CDIP Invoices. From (L to R) GT and predicted images respectively.