UniTabNet: Bridging Vision and Language Models for Enhanced Table Structure Recognition

TL;DR

This paper introduces UniTabNet, a novel framework for table structure parsing based on the image-to-text model, utilizing an image-to-text model to decouple table cells and integrating both physical and logical decoders to reconstruct the complete table structure.

Abstract

In the digital era, table structure recognition technology is a critical tool for processing and analyzing large volumes of tabular data. Previous methods primarily focus on visual aspects of table structure recovery but often fail to effectively comprehend the textual semantics within tables, particularly for descriptive textual cells. In this paper, we introduce UniTabNet, a novel framework for table structure parsing based on the image-to-text model. UniTabNet employs a ``divide-and-conquer'' strategy, utilizing an image-to-text model to decouple table cells and integrating both physical and logical decoders to reconstruct the complete table structure. We further enhance our framework with the Vision Guider, which directs the model's focus towards pertinent areas, thereby boosting prediction accuracy. Additionally, we introduce the Language Guider to refine the model's capability to understand textual semantics in table images. Evaluated on prominent table structure datasets such as PubTabNet, PubTables1M, WTW, and iFLYTAB, UniTabNet achieves a new state-of-the-art performance, demonstrating the efficacy of our approach. The code will also be made publicly available.

Figures (9)

• Figure 1: The illustration of the rich textual features in tabular images. (a) displays the original tabular image. (b) and (c) provide zoomed-in views of the area outlined by the red dashed box in (a). (b) shows the prediction result of the recent state-of-the-art table structure recognition method SEMv2SEMv2. (c) presents the ground truth label for table structure. The red dashed box highlights the discrepancy between the prediction and the ground truth label.
• Figure 2: The illustration of the table structure recognition task.
• Figure 3: The overall architecture of UniTabNet. It mainly consists of a vision encoder and a text decoder. Using the text decoder's output, the Cell Decoder decodes the physical and logical attributes of table cells. The Vision Guider directs the model's focus on row and column information, while the Language Guider aids in understanding textual semantics.
• Figure 4: The illustration of the task design.
• Figure 5: The illustration of the Vision Guider and Language Guider. Panels (a) and (b) compare the attention distributions within the decoding cells (regions indicated by red dashed boxes) for systems T2 and T3, respectively. Panels (c) and (d) display the comparative structural prediction results on iFLYTAB-DP for systems T3 and T4. The red dashed boxes highlight the regions where the predictions differ between the two systems, with system T4 accurately predicting in these areas.