AskChart: Universal Chart Understanding through Textual Enhancement

TL;DR

AskChart targets universal chart understanding by explicitly integrating embedded textual information with visual chart cues through a sparse Mixture of Experts. It introduces ChartBank, a large-scale dataset (~$7.5$M samples) to align visual-textual cues and support multimodal learning across chart tasks, paired with a three-stage training strategy (visual-textual alignment, multi-task instruction tuning, MoE fine-tuning). The approach achieves state-of-the-art results on several benchmarks, notably surpassing a $13$B-parameter baseline by $68.3\%$ in Open-ended ChartQA and $49.2\%$ in Chart-to-Text, while remaining competitive on ChartQA and Chart-to-Table with a much smaller model ($4.6$B parameters). This work demonstrates the practical value of explicit text enhancement for charts, enabling a compact, generalist model with strong cross-task performance.

Abstract

Chart understanding tasks such as ChartQA and Chart-to-Text involve automatically extracting and interpreting key information from charts, enabling users to query or convert visual data into structured formats. State-of-the-art approaches primarily focus on visual cues from chart images, failing to explicitly incorporate rich textual information (e.g., data labels and axis labels) embedded within the charts. This textual information is vital for intuitive human comprehension and interpretation of charts. Moreover, existing models are often large and computationally intensive, limiting their practical applicability. In this paper, we introduce AskChart, a universal model that explicitly integrates both textual and visual cues from charts using a Mixture of Experts (MoE) architecture. AskChart facilitates the learning of enhanced visual-textual representations of charts for effectively handling multiple chart understanding tasks, while maintaining a smaller model size. To capture the synergy between visual and textual modalities, we curate a large-scale dataset named ChartBank with about 7.5M data samples, which helps align textual and visual information and facilitates the extraction of visual entities and text. To effectively train AskChart, we design a three-stage training strategy to align visual and textual modalities for learning robust visual-textual representations and optimizing the learning of the MoE layer. Extensive experiments across five datasets demonstrate the significant performance gains of AskChart in four chart understanding tasks. Remarkably, AskChart with 4.6B parameters outperforms state-of-the-art models with 13B parameters by 68.3% in Open-ended ChartQA and 49.2% in Chart-to-Text tasks, while achieving comparable performance in ChartQA and Chart-to-Table tasks.

Figures (12)

• Figure 1: Comparison between the conventional approach (specialized MLLMs) and our proposed method (AskChart) for chart understanding tasks. Our approach explicitly integrates both visual and textual information from charts, resulting in better performance in chart understanding tasks.
• Figure 2: The framework of AskChart. The upper part shows the processing pipeline and AskChart structure while the lower part shows examples in ChartBank for pretraining. We newly curate three datasets: (a) Visual Prompt Dataset, (b) OCR-aware Data Prompt Dataset, and (c) Chart-to-Table Instruction Following Dataset. For ChartBank examples in lower part, blocks in green indicate tasks (a1, b1, c1); blocks with purple borders indicate input charts (a3, b2, c2); block in blue is the OCR result (b3); blocks in yellow indicate answers (a4, b4, c3).
• Figure 3: Evaluation results on chart-related benchmarks. The size of the markers represents the size of the corresponding models, with larger markers indicating larger model sizes. Our AskChart is represented as a red star, demonstrating its good performance across various tasks.
• Figure 4: Modalities across different experts.
• Figure 5: Results on the ChartQA Human test set by chart type.