TASER: Table Agents for Schema-guided Extraction and Recommendation

TL;DR

TASER tackles the challenge of extracting structured holdings data from highly heterogeneous, multi-page financial tables by employing a three-agent, schema-guided extraction framework. A Detector, Extractor, and Recommender operate in a recursive loop over an initial Portfolio schema, with the Recommender refining the schema and triggering re-extraction to improve fidelity. Empirical results show TASER outperforms the Table Transformer by 10.1% in detection and achieves superior dollar-value fidelity, aided by adaptive batch-size strategies that balance schema coverage, diversity, and precision. The work introduces TASERTab, a large real-world financial table dataset, and demonstrates the practical viability of continuous, schema-guided, agent-based extraction for complex regulatory documents.

Abstract

Real-world financial documents report essential information about an entity's financial holdings that can span millions of different financial instrument types. Yet, these details are often buried in messy, multi-page, fragmented tables - for example, 99.4% of the tables in our dataset have no bounding boxes with the maximum number of rows amounting to 426 per table across 44 pages. To tackle these unique challenges from real-world tables, we present a continuously learning, agentic table extraction system, TASER (Table Agents for Schema-guided Extraction and Recommendation) that extracts highly unstructured, multi-page, heterogeneous tables into normalized, schema-conforming outputs. Our table agents execute on table detection, classification, extraction, and recommendations by leveraging an initial schema. Then, our Recommender Agent reviews the outputs, recommends schema revisions, and decides on the final recommendations, enabling TASER to outperform existing table detection models such as Table Transformer by 10.1%. Within this continuous learning process, we highlight that larger batch sizes result in a 104.3% increase in schema recommendations that are actionable and utilized, resulting in a 9.8% increase in extracted holdings - highlighting the importance of a continuous learning process. To train TASER, we have manually labeled 22,584 pages (28,150,449 tokens), 3,213 tables for $731,685,511,687 of holdings culminating in one of the first real financial table datasets. We release our dataset TASERTab to enable the research community to access real-world financial tables and outputs. Our results highlight the promise of agentic, schema-guided extraction systems for robust understanding of real-world financial tables.

Figures (37)

• Figure 1: Complexity of Holdings Table in Regulatory Filings. In the original format, multiple data attributes are displayed in a single line, with no bounding boxes, rendering the generation of structured outputs highly challenging. TASER enables the generation of structured outputs from highly variable, multi-page financial tables for complex instrument holdings. Negative quantities or market values denote short positions. See Appendix \ref{['app:exampleholdings']} for additional outputs.
• Figure 2: Variety and complexity of financial tables. From leftmost column - for a single financial filing type, such as annual reports, there are a variety of formats by which these reports are filed. Within each filing, there are numerous table types - spanning from Performance, Financial Holdings, or Cash Flow Tables. Each table taxonomy houses very different types of information and has different objectives. For example, the Cash Flow Table houses information relating to operational cash movements for an entity. In contrast, the Financial Holdings Table (which is our table of interest in this paper) displays all financial instruments that an entity holds. For the Financial Holdings Table, there are numerous layout structures and formats by which tables appear, as seen in the rightmost column. Due to the extreme heterogeneity of formatting, document layout, and table structure, traditional table extraction methods fail to perform for financial filings.
• Figure 3: Agentic feedback loop for schema-guided table extraction and continuous schema recommendations. The pipeline begins with raw document ingestion, where candidate pages are evaluated by a Table Detector Agent. Only pages identified as Financial Holdings Tables proceed to the extraction stage, where our Extractor Agent parses each table into a structured, schema-conformant response (e.g., extracting option instrument fields as shown). In the Refinement stage, outputs are partitioned into declared (matched) and unmatched holdings. Unmatched holdings are grouped into batches, which trigger a schema recommendation and feedback loop from the Recommender Agent (Figure \ref{['fig:schema-refinement-flow']}). The Recommender Agent proposes schema refinements, reconciles its recommendations, and finally, updates the schema, enabling TASER to adapt and improve extraction fidelity over successive iterations.
• Figure 4: Schema-Guided Agentic Extraction and Refinement Loop. The extraction pipeline begins with an Initial Schema Definition (v1), which guides the LLM Extractor Agent as it processes the raw Holdings Table to produce Declared Holdings. Holdings that do not match the schema are routed as Unmatched Holdings, triggering the generation of Schema Update Suggestions. These suggestions are reviewed, clustered, and aggregated by our Validation Agent before updating the schema (v2), replacing the prior definition and closing the agentic feedback loop. This process enables continuous schema refinement and robust extraction from heterogeneous, visually complex financial tables.
• Figure 5: Left: Cumulative unique schemas per iteration; larger batches discover schemas rapidly but plateau quickly. Right: Cumulative unique schemas per unmatched holding seen; smaller batches ultimately yield more unique schemas but require more suggestions and generate more redundancy.