Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hypergraph based Understanding for Document Semantic Entity Recognition

Qiwei Li, Zuchao Li, Ping Wang, Haojun Ai, Hai Zhao

TL;DR

The paper tackles semantic entity recognition in visually-rich documents and introduces Hypergraph Attention (HGA) to jointly model entity boundaries and categories. HGALayoutLM extends GraphLayoutLM by incorporating a span-aware hypergraph head and a balanced hyperedge loss, enabling more precise extraction of semantic entities. Experiments on FUNSD, CORD, XFUND, and SROIE demonstrate state-of-the-art performance on FUNSD and XFUND and competitive results on others, while maintaining reasonable efficiency. The work highlights a scalable path to improved downstream SER in VRDU, with clear limitations around label-type sparsity and future directions for generalization and efficiency.

Abstract

Semantic entity recognition is an important task in the field of visually-rich document understanding. It distinguishes the semantic types of text by analyzing the position relationship between text nodes and the relation between text content. The existing document understanding models mainly focus on entity categories while ignoring the extraction of entity boundaries. We build a novel hypergraph attention document semantic entity recognition framework, HGA, which uses hypergraph attention to focus on entity boundaries and entity categories at the same time. It can conduct a more detailed analysis of the document text representation analyzed by the upstream model and achieves a better performance of semantic information. We apply this method on the basis of GraphLayoutLM to construct a new semantic entity recognition model HGALayoutLM. Our experiment results on FUNSD, CORD, XFUND and SROIE show that our method can effectively improve the performance of semantic entity recognition tasks based on the original model. The results of HGALayoutLM on FUNSD and XFUND reach the new state-of-the-art results.

Hypergraph based Understanding for Document Semantic Entity Recognition

TL;DR

The paper tackles semantic entity recognition in visually-rich documents and introduces Hypergraph Attention (HGA) to jointly model entity boundaries and categories. HGALayoutLM extends GraphLayoutLM by incorporating a span-aware hypergraph head and a balanced hyperedge loss, enabling more precise extraction of semantic entities. Experiments on FUNSD, CORD, XFUND, and SROIE demonstrate state-of-the-art performance on FUNSD and XFUND and competitive results on others, while maintaining reasonable efficiency. The work highlights a scalable path to improved downstream SER in VRDU, with clear limitations around label-type sparsity and future directions for generalization and efficiency.

Abstract

Semantic entity recognition is an important task in the field of visually-rich document understanding. It distinguishes the semantic types of text by analyzing the position relationship between text nodes and the relation between text content. The existing document understanding models mainly focus on entity categories while ignoring the extraction of entity boundaries. We build a novel hypergraph attention document semantic entity recognition framework, HGA, which uses hypergraph attention to focus on entity boundaries and entity categories at the same time. It can conduct a more detailed analysis of the document text representation analyzed by the upstream model and achieves a better performance of semantic information. We apply this method on the basis of GraphLayoutLM to construct a new semantic entity recognition model HGALayoutLM. Our experiment results on FUNSD, CORD, XFUND and SROIE show that our method can effectively improve the performance of semantic entity recognition tasks based on the original model. The results of HGALayoutLM on FUNSD and XFUND reach the new state-of-the-art results.
Paper Structure (21 sections, 10 equations, 5 figures, 7 tables)

This paper contains 21 sections, 10 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Traditional Semantic Entity Recognition.
  3. Hypergraph Semantic Entity Recognition.
  4. Related Work
  5. Methodology
  6. Overview
  7. Hypergraph Attention Head
  8. Span Position Encoding
  9. Balanced Hyperedge Loss
  10. HGALayoutLM
  11. Experiment
  12. Experimental Setup
  13. Model Settings.
  14. Datasets.
  15. Baselines.
  16. ...and 6 more sections

Figures (5)

  • Figure 1: Difference in Document Task.
  • Figure 2: Traditional Semantic Entity Recognition and Hypergraph Semantic Entity Recognition.The document is from FUNSD dataset. Only the text sequence is shown in the figure. The rectangles with different colors in the figure are text nodes. The colors on the document nodes represent the different class labels. The orange color represents the label "HEADER". Blue is the label "QUESTION". Green is the label "ANSWER". Pink is the nonmeaning label, which is "OTHER".
  • Figure 3: Semantic Entity Recognition Process Based on Hypergraph Attention. Only the text processing part of the model is shown in the figure. In the span position generation stage, the span position of the token feature sequence needs to be created by using the text node range span. The token features will be linearly transformed and encode the span position into a query vector Q and a key vector V. The multi-head hypergraph attention score is calculated from Q, V and added with the lower triangle mask. We regard each attention head as a sub-hypergraph corresponding to each hyperedge type.
  • Figure 4: Position Encoding Comparison Line Chart. In order to highlight the contrast effect, we omit the results for the first 300 steps when the model has not converged.
  • Figure 5: Further Study of Balance Factor.