TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy

TL;DR

TabPedia introduces a unified large vision-language framework for Visual Table Understanding by treating diverse VTU tasks as interchangeable concepts and enabling their synergy via meditative tokens. Using dual vision encoders to capture both global and fine-grained cues, it integrates TD, TSR, TQ, and TQA within a single pipeline through an LLM (Vicuna-7B). The approach is validated on a suite of benchmarks, notably introducing ComTQA, a challenging real-world VTU dataset with ~9k QA pairs, and achieving strong results across perception and comprehension tasks. This work demonstrates a practical pathway toward end-to-end visual table understanding in real-world documents, with open-sourced data and code to foster further research.

Abstract

Tables contain factual and quantitative data accompanied by various structures and contents that pose challenges for machine comprehension. Previous methods generally design task-specific architectures and objectives for individual tasks, resulting in modal isolation and intricate workflows. In this paper, we present a novel large vision-language model, TabPedia, equipped with a concept synergy mechanism. In this mechanism, all the involved diverse visual table understanding (VTU) tasks and multi-source visual embeddings are abstracted as concepts. This unified framework allows TabPedia to seamlessly integrate VTU tasks, such as table detection, table structure recognition, table querying, and table question answering, by leveraging the capabilities of large language models (LLMs). Moreover, the concept synergy mechanism enables table perception-related and comprehension-related tasks to work in harmony, as they can effectively leverage the needed clues from the corresponding source perception embeddings. Furthermore, to better evaluate the VTU task in real-world scenarios, we establish a new and comprehensive table VQA benchmark, ComTQA, featuring approximately 9,000 QA pairs. Extensive quantitative and qualitative experiments on both table perception and comprehension tasks, conducted across various public benchmarks, validate the effectiveness of our TabPedia. The superior performance further confirms the feasibility of using LLMs for understanding visual tables when all concepts work in synergy. The benchmark ComTQA has been open-sourced at https://huggingface.co/datasets/ByteDance/ComTQA. The source code and model also have been released athttps://github.com/zhaowc-ustc/TabPedia.

Figures (9)

• Figure 1: Comparison with previous task-specific pipelines for visual table understanding. In contrast to design different architectures for various table tasks, our TabPedia effectively performs these tasks in a unified framework through delicately leveraging the understanding capability of LLMs.
• Figure 2: The illustration of our proposed TabPedia. Given the input image, TabPedia feeds it into both vision encoders attached projections to extract different granular features. Then, the visual tokens are combined with instruction-derived tokens, and fed into the LLM. The LLM leverages its powerful understanding ability to generate a plausible response.
• Figure 3: Qualitative results of TabPedia on diverse tasks. The first row shows its perception capability on both TD and TSR tasks. The second row further exhibits TabPedia's powerful ability by employing multiple instructions of different tasks. The bottom row showcases TabPedia's accurate responses based on intricate contents in visual tables. Zoom in for best view.
• Figure A1: The illustration of an example for generating QA pairs with the powerful LVLM, Gemini Pro reid2024gemini. The prompt includes several key rules to ensure the response quality as much as possible.
• Figure A2: Statistics of ComTQA benchmark.