Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segments

Aniket Bhattacharyya, Anurag Tripathi, Ujjal Das, Archan Karmakar, Amit Pathak, Maneesh Gupta

TL;DR

BLOCKIE tackles information extraction from visually rich documents by organizing text into semantic blocks and applying block-level LLM reasoning, improving generalization across heterogeneous formats. The approach creates, parses, and combines self-contained semantic blocks to assemble a complete document schema, achieving state-of-the-art $F1$ on CORD, FUNSD, and SROIE and enabling smaller LLMs to outperform larger ones through structured, localized reasoning. Key contributions include the formalization of semantic blocks and semantic atoms, a block-centric pipeline with strong cross-document sharing, and demonstrated value-absent inference capabilities. This method enhances robustness to layout variability and formats not seen during training, offering practical significance for scalable VRD information extraction in real-world, diverse document collections.

Abstract

Information extraction (IE) from Visually Rich Documents (VRDs) containing layout features along with text is a critical and well-studied task. Specialized non-LLM NLP-based solutions typically involve training models using both textual and geometric information to label sequences/tokens as named entities or answers to specific questions. However, these approaches lack reasoning, are not able to infer values not explicitly present in documents, and do not generalize well to new formats. Generative LLM-based approaches proposed recently are capable of reasoning, but struggle to comprehend clues from document layout especially in previously unseen document formats, and do not show competitive performance in heterogeneous VRD benchmark datasets. In this paper, we propose BLOCKIE, a novel LLM-based approach that organizes VRDs into localized, reusable semantic textual segments called $\textit{semantic blocks}$, which are processed independently. Through focused and more generalizable reasoning,our approach outperforms the state-of-the-art on public VRD benchmarks by 1-3% in F1 scores, is resilient to document formats previously not encountered and shows abilities to correctly extract information not explicitly present in documents.

Information Extraction from Visually Rich Documents using LLM-based Organization of Documents into Independent Textual Segments

TL;DR

BLOCKIE tackles information extraction from visually rich documents by organizing text into semantic blocks and applying block-level LLM reasoning, improving generalization across heterogeneous formats. The approach creates, parses, and combines self-contained semantic blocks to assemble a complete document schema, achieving state-of-the-art on CORD, FUNSD, and SROIE and enabling smaller LLMs to outperform larger ones through structured, localized reasoning. Key contributions include the formalization of semantic blocks and semantic atoms, a block-centric pipeline with strong cross-document sharing, and demonstrated value-absent inference capabilities. This method enhances robustness to layout variability and formats not seen during training, offering practical significance for scalable VRD information extraction in real-world, diverse document collections.

Abstract

Information extraction (IE) from Visually Rich Documents (VRDs) containing layout features along with text is a critical and well-studied task. Specialized non-LLM NLP-based solutions typically involve training models using both textual and geometric information to label sequences/tokens as named entities or answers to specific questions. However, these approaches lack reasoning, are not able to infer values not explicitly present in documents, and do not generalize well to new formats. Generative LLM-based approaches proposed recently are capable of reasoning, but struggle to comprehend clues from document layout especially in previously unseen document formats, and do not show competitive performance in heterogeneous VRD benchmark datasets. In this paper, we propose BLOCKIE, a novel LLM-based approach that organizes VRDs into localized, reusable semantic textual segments called , which are processed independently. Through focused and more generalizable reasoning,our approach outperforms the state-of-the-art on public VRD benchmarks by 1-3% in F1 scores, is resilient to document formats previously not encountered and shows abilities to correctly extract information not explicitly present in documents.
Paper Structure (20 sections, 4 equations, 8 figures, 9 tables)

This paper contains 20 sections, 4 equations, 8 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semantic Blocks in VRDs
  4. Proposed Methodology
  5. Block Creation
  6. Block Parsing
  7. Combining Blocks
  8. Experimental Setup and Results
  9. Experimental Setup
  10. Results
  11. Performance Analysis
  12. BLOCKIE helps smaller LLMs outperform large LLMs
  13. BLOCKIE is resistant to heterogeneity and to unseen document formats.
  14. Block creation is crucial for BLOCKIE performance
  15. BLOCKIE is able to perform value-absent inference
  16. ...and 5 more sections

Figures (8)

  • Figure 1: The Information Extraction Task, illustrated using sample images from funsd and Huang_2019
  • Figure 2: Sample image with document schema and value
  • Figure 3: Sample image with document schema and value
  • Figure 4: Illustrative flow with a simulated receipt and schema resembling CORD output requirement. The schema is passed along with the output of the block creator along with parses of similar blocks to block parser. Parsed blocks with target schema are then passed to get final output. Reasons are output at each stage.
  • Figure 5: Motivating example for the conceptualization of VRD IE as the parsing of related semantic entities organized in blocks. The entities within a block are related which allows a human to understand that the address in the company details block belongs to the invoicing company instead of say the customer.
  • ...and 3 more figures