Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An agentic system with reinforcement-learned subsystem improvements for parsing form-like documents

Ayesha Amjad, Saurav Sthapit, Tahir Qasim Syed

TL;DR

The paper tackles the challenge of reliably extracting structured data from diverse form-like documents by moving beyond monolithic LLM pipelines to a modular, agentic framework. It combines multiple LLM agents (for classification, schema building, and data extraction) with reinforcement learning and meta-prompting to continuously improve prompts based on past performance, aiming to operate across varying document types, layouts, and LLMs. The approach yields substantial improvements over baseline one-shot prompting on benchmarks like SOIRE and CORD, particularly in handling multi-page documents and complete extraction of line items in financial documents. This agentic system offers a scalable, self-correcting solution for automated document data extraction, while also addressing practical concerns such as processing efficiency, evaluation metrics, and ethical considerations around data privacy and security.

Abstract

Extracting alphanumeric data from form-like documents such as invoices, purchase orders, bills, and financial documents is often performed via vision (OCR) and learning algorithms or monolithic pipelines with limited potential for systemic improvements. We propose an agentic AI system that leverages Large Language Model (LLM) agents and a reinforcement learning (RL) driver agent to automate consistent, self-improving extraction under LLM inference uncertainty. Our work highlights the limitations of monolithic LLM-based extraction and introduces a modular, multi-agent framework with task-specific prompts and an RL policy of rewards and penalties to guide a meta-prompting agent to learn from past errors and improve prompt-based actor agents. This self-corrective adaptive system handles diverse documents, file formats, layouts, and LLMs, aiming to automate accurate information extraction without the need for human intervention. Results as reported on two benchmark datasets of SOIRE, and CORD, are promising for the agentic AI framework.

An agentic system with reinforcement-learned subsystem improvements for parsing form-like documents

TL;DR

The paper tackles the challenge of reliably extracting structured data from diverse form-like documents by moving beyond monolithic LLM pipelines to a modular, agentic framework. It combines multiple LLM agents (for classification, schema building, and data extraction) with reinforcement learning and meta-prompting to continuously improve prompts based on past performance, aiming to operate across varying document types, layouts, and LLMs. The approach yields substantial improvements over baseline one-shot prompting on benchmarks like SOIRE and CORD, particularly in handling multi-page documents and complete extraction of line items in financial documents. This agentic system offers a scalable, self-correcting solution for automated document data extraction, while also addressing practical concerns such as processing efficiency, evaluation metrics, and ethical considerations around data privacy and security.

Abstract

Extracting alphanumeric data from form-like documents such as invoices, purchase orders, bills, and financial documents is often performed via vision (OCR) and learning algorithms or monolithic pipelines with limited potential for systemic improvements. We propose an agentic AI system that leverages Large Language Model (LLM) agents and a reinforcement learning (RL) driver agent to automate consistent, self-improving extraction under LLM inference uncertainty. Our work highlights the limitations of monolithic LLM-based extraction and introduces a modular, multi-agent framework with task-specific prompts and an RL policy of rewards and penalties to guide a meta-prompting agent to learn from past errors and improve prompt-based actor agents. This self-corrective adaptive system handles diverse documents, file formats, layouts, and LLMs, aiming to automate accurate information extraction without the need for human intervention. Results as reported on two benchmark datasets of SOIRE, and CORD, are promising for the agentic AI framework.
Paper Structure (28 sections, 22 equations, 7 figures, 5 tables)

This paper contains 28 sections, 22 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Understanding Form-Like Documents
  3. Related Work
  4. Methodology
  5. Document Text Reader
  6. Evaluation Metrics
  7. Baseline Data Extraction Framework
  8. Agentic Data Extraction Framework
  9. Document Classification Agent
  10. Automated Schema Building
  11. State Representation:
  12. Reward Function:
  13. Action Space:
  14. Termination Criteria:
  15. Iterative Data Extraction Environment
  16. ...and 13 more sections

Figures (7)

  • Figure 1: Architecture of the agentic form-like document data extraction framework comprising seven agents (blue), two Gymnasium environments (dark pink), and five evaluation metrics (cyan). Agents include a document classifier, schema builder, data extractor, two Gymnasium agents, a meta-prompting agent, and an evaluator. Evaluation metrics guide iterative optimization of the environments.
  • Figure 2: a) Extracted JSON of a receipt (96 dpi) scored 90% on exact matches, 100% on semantic matches, and 96% on cosine similarity. b) Extracted JSON of a utility bill (66 dpi) scored lower: 30% exact matches, 65% semantic matches, 71% similarity.
  • Figure 3: Incorrect data extracted for a 2-page scanned document. “3093” is extracted as “goog” and “4155” is extracted as “4is5”
  • Figure 4: Top: 'Invoice Total' is misidentified as 'total' due to semantic closeness. Bottom: explicitly specifying 'CODE' as a column header enables correct extraction, even though there is no 'CODE' column header in the document table.
  • Figure 5: This salary slip shows four tables, but only two are unique, as earnings and tax deductions each appear twice. Schema builder prompt (left) retains only unique tables.
  • ...and 2 more figures