TFLOP: Table Structure Recognition Framework with Layout Pointer Mechanism

TL;DR

This paper addresses the challenge of Table Structure Recognition (TSR) from images by mitigating text-region misalignment issues that plague traditional bounding-box mapping approaches. It introduces TFLOP, a four-module framework that uses a Layout Pointer to directly connect text regions to HTML structure tokens, and augments it with span-aware contrastive supervision to handle complex row/column spans. The method achieves state-of-the-art performance on PubTabNet, FinTabNet, and SynthTabNet, while also showing versatility in watermark-filtered and cross-lingual (Korean) TSR scenarios. The findings highlight TFLOP’s practical impact for industrial document parsing, reducing post-processing needs and improving robustness across languages and noisy inputs.

Abstract

Table Structure Recognition (TSR) is a task aimed at converting table images into a machine-readable format (e.g. HTML), to facilitate other applications such as information retrieval. Recent works tackle this problem by identifying the HTML tags and text regions, where the latter is used for text extraction from the table document. These works however, suffer from misalignment issues when mapping text into the identified text regions. In this paper, we introduce a new TSR framework, called TFLOP (TSR Framework with LayOut Pointer mechanism), which reformulates the conventional text region prediction and matching into a direct text region pointing problem. Specifically, TFLOP utilizes text region information to identify both the table's structure tags and its aligned text regions, simultaneously. Without the need for region prediction and alignment, TFLOP circumvents the additional text region matching stage, which requires finely-calibrated post-processing. TFLOP also employs span-aware contrastive supervision to enhance the pointing mechanism in tables with complex structure. As a result, TFLOP achieves the state-of-the-art performance across multiple benchmarks such as PubTabNet, FinTabNet, and SynthTabNet. In our extensive experiments, TFLOP not only exhibits competitive performance but also shows promising results on industrial document TSR scenarios such as documents with watermarks or in non-English domain.

Figures (14)

• Figure 1: Overview of two TSR frameworks. The dual decoder identifies table cell regions and their HTML structure, requiring further cell and text region mapping for the final output. In contrast, TFLOP utilizes text region information and directly identifies the HTML structure with its corresponding text region relations.
• Figure 2: Overview illustration of TFLOP. Given a tabular image and its text region bounding boxes, visual features and layout embedding are output by the Image and Layout Encoders. Logical Structure Decoder then receives these features to auto-regressively generate table structure tokens (tags) while also predicting the associations between text region bounding boxes and table data tags through the Layout Pointer. These associations and table tags are aggregated to generate the full table structure.
• Figure 3: Sample visualisation of span-aware constrastive supervision involving multi-span structures. In the column-wise contrastive supervision example above, for a given bounding box ($i$, pink), positive samples ($P(i)$) are those with either full overlap (green) or partial overlap (orange), while the rest (red) are negative samples.
• Figure 4: Visualisations of tables constructed from generated HTML sequences with corresponding tabular images (recreated for improved legibility) for reference. TFLOP successfully constructs tables with complex structures such as hierarchical row-spans (top) or hierarchical column-spans (bottom).
• Figure 5: Row-wise and column-wise t-SNE visualisation of bounding box embeddings. PubTabNet table image (top, table recreated for improved legibility) and 25 bounding boxes are sampled for visualisation. Filled-colours represent different row-span groups while border-colours represent different column-span groups sampled for visualisation. Colours in t-SNE plots match that of table above and boxes spanning multi-rows/columns are marked with a red star.