UniTable: Towards a Unified Framework for Table Recognition via Self-Supervised Pretraining

TL;DR

UniTable tackles the challenge of table recognition by unifying the training paradigm and objective into a single language‑modeling framework that ingests raw tabular images and outputs a token sequence representing table structure, cell bbox, and content as HTML. It combines self‑supervised pretraining of the visual encoder on unannotated tabular images with a unified finetuning stage, enabling the model to perform all TSR tasks without task‑specific heads or pdf post‑processing. The approach achieves state‑of‑the-art performance on four large TSR datasets, surpassing prior TSR methods and general vision‑language models, and generalizes to pdf inputs converted to images. The authors release open‑source code and a Jupyter notebook to reproduce results and facilitate practical use via a HuggingFace API.

Abstract

Tables convey factual and quantitative data with implicit conventions created by humans that are often challenging for machines to parse. Prior work on table recognition (TR) has mainly centered around complex task-specific combinations of available inputs and tools. We present UniTable, a training framework that unifies both the training paradigm and training objective of TR. Its training paradigm combines the simplicity of purely pixel-level inputs with the effectiveness and scalability empowered by self-supervised pretraining from diverse unannotated tabular images. Our framework unifies the training objectives of all three TR tasks - extracting table structure, cell content, and cell bounding box - into a unified task-agnostic training objective: language modeling. Extensive quantitative and qualitative analyses highlight UniTable's state-of-the-art (SOTA) performance on four of the largest TR datasets. UniTable's table parsing capability has surpassed both existing TR methods and general large vision-language models, e.g., GPT-4o, GPT-4-turbo with vision, and LLaVA. Our code is publicly available at https://github.com/poloclub/unitable, featuring a Jupyter Notebook that includes the complete inference pipeline, fine-tuned across multiple TR datasets, supporting all three TR tasks.

Figures (8)

• Figure 1: UniTable, a training framework that unifies both training paradigm and training objective of tsr. In UniTable, the visual encoder is self-supervised pretrained and then finetuned along with the task decoder on supervised datasets. UniTable unifies the training objectives of all three tsr tasks --- extracting table structure, cell bbox, and cell content --- into a unified task-agnostic training objective: language modeling. With UniTable, the user inputs a tabular image and obtains the corresponding digitalized table in HTML.
• Figure 2: The table parsing capability of UniTable has surpassed that of general large vlm, e.g., GPT-4o, GPT-4-turbo with vision, and LLaVA. For complex tables that include multiple spanning cells, UniTable can successfully reconstruct the table in HTML, whereas general large vlm fail in various aspects.
• Figure 3: The key reason that ssp works is because each tokens have visual semantics and the codebook shows a fine-grained categorization to represent the implicit conventions in the table. The codebook used in ssp has learned to represent abstract concepts by using different groups of tokens to represent different concepts: (a) empty background and (b) separations within a table. Red highlights the token indices under investigation. Appendix \ref{['appx: codebook']} provides a zoomed-in version of these images labeled with token indices. We lay the color patches over the original tabular image to present the selected token indices from the 2M vqvae.
• Figure 4: Table cell bbox detection results of UniTable on unannotated tables in practice.
• Figure 5: A zoomed-in version of the token indices from the 2M vqvae. The codebook used in ssp has learned to represent abstract concepts by using different groups of tokens to represent empty white backgrounds within a table. Red highlights the token indices representing the concept of "empty white backgrounds".