A General Information Extraction Framework Based on Formal Languages
Markus L. Schmid
TL;DR
This work presents a unifying, language-theoretic framework for information extraction that generalizes document spanners by mapping input strings to tables of position-sets across attributes. It formalizes extractors as total functions from $Σ^*$ to Γ-tables and encodes outputs via marker languages, enabling a rich set of operations (set operations, joins, concatenation, Kleene star, projections, renaming, etc.) that correspond to natural language- and automata-theory operations. The paper analyzes two central language classes, regular and context-free extractors, establishing closure properties, multiple representation schemes (algebraic expressions and MSO formulas), and the complexity of natural problems such as table and membership problems, including conditional results under SETH. The results bridge formal language theory and information extraction, highlighting tractable cases for regular extractors and outlining theoretical and practical avenues such as subregular classes and new table-problem formulations for broader applicability and deeper understanding.
Abstract
For a terminal alphabet $Σ$ and an attribute alphabet $Γ$, a $(Σ, Γ)$-extractor is a function that maps every string over $Σ$ to a table with a column per attribute and with sets of positions of $w$ as cell entries. This rather general information extraction framework extends the well-known document spanner framework, which has intensively been investigated in the database theory community over the last decade. Moreover, our framework is based on formal language theory in a particularly clean and simple way. In addition to this conceptual contribution, we investigate closure properties, different representation formalisms and the complexity of natural decision problems for extractors.