MSG-Chart: Multimodal Scene Graph for ChartQA

TL;DR

This work addresses ChartQA by introducing a joint multimodal scene graph for charts, comprising a visual graph to model spatial relations and a textual graph to encode semantic relations from labels and OCR. The graphs are processed with graph convolutional networks and injected as an inductive bias between chart understanding backbones (UniChart and VL-T5), enabling object-level structural and semantic information to guide question answering. Empirical results on ChartQA and OpenCQA show consistent improvements across backbones, confirming the effectiveness of explicit graph-based reasoning for chart understanding. The proposed framework offers a flexible plug-in that enhances diverse chartQA architectures and lays groundwork for more robust, relational reasoning in chart analytics.

Abstract

Automatic Chart Question Answering (ChartQA) is challenging due to the complex distribution of chart elements with patterns of the underlying data not explicitly displayed in charts. To address this challenge, we design a joint multimodal scene graph for charts to explicitly represent the relationships between chart elements and their patterns. Our proposed multimodal scene graph includes a visual graph and a textual graph to jointly capture the structural and semantical knowledge from the chart. This graph module can be easily integrated with different vision transformers as inductive bias. Our experiments demonstrate that incorporating the proposed graph module enhances the understanding of charts' elements' structure and semantics, thereby improving performance on publicly available benchmarks, ChartQA and OpenCQA.

Figures (2)

• Figure 1: Cutting-Edge LLMs and Our MSG-Chart
• Figure 2: Two Graph Integration Architectures (VG: Visual Graph, TG: Textual Graph)