A framework for extraction and transformation of documents
Cristian Riveros, Markus L. Schmid, Nicole Schweikardt
TL;DR
The paper addresses how to extract information from text and transform it into new documents by a two-phase ET framework that combines regex multispanners for extraction with polyregular (notably linear) string-to-string transformations. It establishes that linear ET programs are expressively equivalent to nondeterministic streaming string transducers under bag semantics, and proves that linear ET programs are closed under composition with an efficient enumeration algorithm that achieves linear preprocessing and output-linear delay. By introducing multispanners and a unique multiref-word encoding, the authors provide a principled approach to model, manipulate, and enumerate transformed outputs, including duplicates, which are meaningful in the ET pipeline. The work contributes a solid theoretical foundation for practical IE workflows, enabling scalable, compositional, and enumerative extraction-transform processes with strong formal guarantees.
Abstract
We present a theoretical framework for the extraction and transformation of text documents. We propose to use a two-phase process where the first phase extracts span-tuples from a document, and the second phase maps the content of the span-tuples into new documents. We base the extraction phase on the framework of document spanners and the transformation phase on the theory of polyregular functions, the class of regular string-to-string functions with polynomial growth. For supporting practical extract-transform scenarios, we propose an extension of document spanners described by regex formulas from span-tuples to so-called multispan-tuples, where variables are mapped to sets of spans. We prove that this extension, called regex multispanners, has the same desirable properties as standard spanners described by regex formulas. In our framework, an Extract-Transform (ET) program is given by a regex multispanner followed by a polyregular function. In this paper, we study the expressibility and evaluation problem of ET programs when the transformation function is linear, called linear ET programs. We show that linear ET programs are equally expressive as non-deterministic streaming string transducers under bag semantics. Moreover, we show that linear ET programs are closed under composition. Finally, we present an enumeration algorithm for evaluating every linear ET program over a document with linear time preprocessing and constant delay.