Beyond Embeddings: The Promise of Visual Table in Visual Reasoning

TL;DR

Visual reasoning benefits from world knowledge beyond what standard visual embeddings capture. The authors introduce Visual Table, a hierarchical, text-based visual representation comprising scene-level and object-level descriptions (categories, attributes, and instance-level knowledge). They build a 61K-annotation COCO-derived dataset and train a generator with three-stage training to produce JSON-formatted tables, enabling both standalone and MLLM-friendly inputs. Empirical results on 11 benchmarks show Visual Tables outperform traditional text-based representations and consistently boost state-of-the-art multimodal LLMs, highlighting interpretability and editable advantages. The work offers a practical pathway to inject structured world knowledge into visual reasoning systems, with open-source code available.

Abstract

Visual representation learning has been a cornerstone in computer vision, involving typical forms such as visual embeddings, structural symbols, and text-based representations. Despite the success of CLIP-type visual embeddings, they often lack access to world knowledge critical for visual reasoning. In this work, we propose Visual Table, a novel form of visual representation tailored for visual reasoning. Visual tables are constructed as hierarchical descriptions of visual scenes, featuring a scene description and multiple object-centric descriptions covering categories, attributes, and knowledge. Thanks to the structural and textual formats, visual tables offer unique advantages over mere visual embeddings, such as interpretability and controllable editing. Furthermore, they deliver instance-level world knowledge and detailed attributes that are essential for visual reasoning. To create visual tables, we develop a generator trained on the dataset with collected, small-scale annotations. Extensive results on 11 visual reasoning benchmarks demonstrate that the generated visual tables significantly outperform previous structural and text-based representations. Moreover, they consistently enhance state-of-the-art multimodal large language models across diverse benchmarks, showcasing their potential for advancing visual reasoning tasks. Our code is available at https://github.com/LaVi-Lab/Visual-Table.

Figures (5)

• Figure 1: Left Top: An example that requires world knowledge to answer the questions about the image. Left Bottom: Our proposed visual tables represent thorough visual content in structured text. Middle: Visual tables provide instance-level knowledge in considering specific visual instances. Right: The generated visual tables can consistently enhance the state-of-the-art multimodal large language models across diverse benchmarks.
• Figure 2: An overview of learning a visual table generator and its application on MLLMs. Left: We design a prompt to collect visual table annotations on a small scale of images. These annotations are used to train our visual table generator which consists of a frozen visual encoder, a vision-language connector, and a pre-trained LLM. Right: Our generator is employed to generate visual tables for the images in downstream tasks. With the generated visual table either as standalone or as additional visual representations, LLM performs reasoning to output text response.
• Figure 3: Visualization of visual reasoning examples. For simplicity, we visualize partial visual tables that relate to the question, with attribute and knowledge separated by ";". Best viewed in color.
• Figure 4: An overview of our designed prompt and the collected visual tables.
• Figure 5: Visualization of visual reasoning examples. For simplicity, we visualize partial visual tables that relate to the questions, with attribute and knowledge separated by ";".