TableFormer: Table Structure Understanding with Transformers

TL;DR

TableFormer presents an end-to-end transformer-based approach for table structure understanding that predicts both the table’s structural tokens and per-cell bounding boxes, enabling direct coupling to PDF cell content. By introducing a dedicated Cell BBox Decoder and a CNN+Transformer architecture, the method achieves state-of-the-art accuracy on structure (TEDS) across PubTabNet, FinTabNet, and TableBank, while maintaining language-agnostic capabilities and efficient inference. The work also introduces SynthTabNet, a large synthetic dataset to diversify structure, style, and language, and provides a detailed post-processing pipeline to align predictions with PDF documents. Overall, TableFormer advances robust, cross-domain table parsing suitable for search, knowledge graphs, and downstream information extraction tasks.

Abstract

Tables organize valuable content in a concise and compact representation. This content is extremely valuable for systems such as search engines, Knowledge Graph's, etc, since they enhance their predictive capabilities. Unfortunately, tables come in a large variety of shapes and sizes. Furthermore, they can have complex column/row-header configurations, multiline rows, different variety of separation lines, missing entries, etc. As such, the correct identification of the table-structure from an image is a non-trivial task. In this paper, we present a new table-structure identification model. The latter improves the latest end-to-end deep learning model (i.e. encoder-dual-decoder from PubTabNet) in two significant ways. First, we introduce a new object detection decoder for table-cells. In this way, we can obtain the content of the table-cells from programmatic PDF's directly from the PDF source and avoid the training of the custom OCR decoders. This architectural change leads to more accurate table-content extraction and allows us to tackle non-english tables. Second, we replace the LSTM decoders with transformer based decoders. This upgrade improves significantly the previous state-of-the-art tree-editing-distance-score (TEDS) from 91% to 98.5% on simple tables and from 88.7% to 95% on complex tables.

Figures (17)

• Figure 1: Picture of a table with subtle, complex features such as (1) multi-column headers, (2) cell with multi-row text and (3) cells with no content. Image from PubTabNet evaluation set, filename: 'PMC2944238_004_02'.
• Figure 2: Distribution of the tables across different table dimensions in PubTabNet + FinTabNet datasets
• Figure 3: TableFormer takes in an image of the PDF and creates bounding box and HTML structure predictions that are synchronized. The bounding boxes grabs the content from the PDF and inserts it in the structure.
• Figure 4: Given an input image of a table, the Encoder produces fixed-length features that represent the input image. The features are then passed to both the Structure Decoder and Cell BBox Decoder. During training, the Structure Decoder receives 'tokenized tags' of the HTML code that represent the table structure. Afterwards, a transformer encoder and decoder architecture is employed to produce features that are received by a linear layer, and the Cell BBox Decoder. The linear layer is applied to the features to predict the tags. Simultaneously, theCell BBox Decoder selects features referring to the data cells ('', '') and passes them through an attention network, an MLP, and a linear layer to predict the bounding boxes.
• Figure 5: One of the benefits of TableFormer is that it is language agnostic, as an example, the left part of the illustration demonstrates TableFormer predictions on previously unseen language (Japanese). Additionally, we see that TableFormer is robust to variability in style and content, right side of the illustration shows the example of the TableFormer prediction from the FinTabNet dataset.