Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A LayoutLMv3-Based Model for Enhanced Relation Extraction in Visually-Rich Documents

Wiam Adnan, Joel Tang, Yassine Bel Khayat Zouggari, Seif Edinne Laatiri, Laurent Lam, Fabien Caspani

TL;DR

The paper tackles relation extraction in Visually‑Rich Documents by extending a LayoutLMv3 backbone with a matrix‑based, bilinear relation head that operates over all entity pairs. It achieves competitive or state‑of‑the‑art RE results on FUNSD and CORD without additional geometric pre‑training and with fewer parameters, aided by an extensive ablation study. Key contributions include two strategies for incorporating entity types (joint EE/RE fine‑tuning and Entity Marker) and three geometry‑focused techniques (Layout Concatenation, Bounding Boxes Ordering, and Bounding Boxes Shuffling), plus a post‑processing RSF step to resolve hierarchical relations. The findings show that bounding box ordering and explicit entity semantics materially improve performance, highlighting the importance of spatial layout and semantic cues for VRD relation extraction and offering practical guidance for designing multimodal RE systems with limited pre‑training. The approach has practical impact for structuring information in documents where visual and spatial cues carry essential relational information, enabling more accurate extraction of complex data hierarchies.

Abstract

Document Understanding is an evolving field in Natural Language Processing (NLP). In particular, visual and spatial features are essential in addition to the raw text itself and hence, several multimodal models were developed in the field of Visual Document Understanding (VDU). However, while research is mainly focused on Key Information Extraction (KIE), Relation Extraction (RE) between identified entities is still under-studied. For instance, RE is crucial to regroup entities or obtain a comprehensive hierarchy of data in a document. In this paper, we present a model that, initialized from LayoutLMv3, can match or outperform the current state-of-the-art results in RE applied to Visually-Rich Documents (VRD) on FUNSD and CORD datasets, without any specific pre-training and with fewer parameters. We also report an extensive ablation study performed on FUNSD, highlighting the great impact of certain features and modelization choices on the performances.

A LayoutLMv3-Based Model for Enhanced Relation Extraction in Visually-Rich Documents

TL;DR

The paper tackles relation extraction in Visually‑Rich Documents by extending a LayoutLMv3 backbone with a matrix‑based, bilinear relation head that operates over all entity pairs. It achieves competitive or state‑of‑the‑art RE results on FUNSD and CORD without additional geometric pre‑training and with fewer parameters, aided by an extensive ablation study. Key contributions include two strategies for incorporating entity types (joint EE/RE fine‑tuning and Entity Marker) and three geometry‑focused techniques (Layout Concatenation, Bounding Boxes Ordering, and Bounding Boxes Shuffling), plus a post‑processing RSF step to resolve hierarchical relations. The findings show that bounding box ordering and explicit entity semantics materially improve performance, highlighting the importance of spatial layout and semantic cues for VRD relation extraction and offering practical guidance for designing multimodal RE systems with limited pre‑training. The approach has practical impact for structuring information in documents where visual and spatial cues carry essential relational information, enabling more accurate extraction of complex data hierarchies.

Abstract

Document Understanding is an evolving field in Natural Language Processing (NLP). In particular, visual and spatial features are essential in addition to the raw text itself and hence, several multimodal models were developed in the field of Visual Document Understanding (VDU). However, while research is mainly focused on Key Information Extraction (KIE), Relation Extraction (RE) between identified entities is still under-studied. For instance, RE is crucial to regroup entities or obtain a comprehensive hierarchy of data in a document. In this paper, we present a model that, initialized from LayoutLMv3, can match or outperform the current state-of-the-art results in RE applied to Visually-Rich Documents (VRD) on FUNSD and CORD datasets, without any specific pre-training and with fewer parameters. We also report an extensive ablation study performed on FUNSD, highlighting the great impact of certain features and modelization choices on the performances.
Paper Structure (21 sections, 3 equations, 2 figures, 3 tables)

This paper contains 21 sections, 3 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Approach
  4. Task formalization
  5. Architecture
  6. Including Entity Types
  7. Joint EE and RE fine-tuning (EEF):
  8. Entity Marker (EM):
  9. Geometry and Spatial Understanding
  10. Layout Concatenation (LC):
  11. Bounding Boxes Ordering (BBO):
  12. Bounding Boxes Shuffling (BBS):
  13. Datasets
  14. Experiments
  15. Experimental setup
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Illustration of the modelized Relation Extraction (RE) task from a Visually-Rich Documents (VRD). Text is split into bounding boxes and follows an order given by the OCR. The model is end-to-end trained to predict a relationship matrix via an asymmetric bilinear layer.
  • Figure 2: Example of relations modeling in CORD dataset