ChartAdapter: Large Vision-Language Model for Chart Summarization
Peixin Xu, Yujuan Ding, Wenqi Fan
TL;DR
ChartAdapter introduces a cross-modal bridge between chart encoders and large language models by leveraging learnable queries and a cross-modal projector to extract implicit chart semantics. The architecture couples four components—cross-modal projector, latent textual embeddings, cross-modal interaction, and implicit semantic decoder—within a transformer-based framework and translates visual chart features into textual summaries via an end-to-end LLM pipeline. A multi-stage training strategy, together with a new ChartSumm dataset of $190{,}618$ samples, enables robust vision-to-language alignment and high-quality chart summaries, achieving state-of-the-art results on the Chart-to-Text Pew benchmark (e.g., BLEU-4 $=35.55$, Rouge-L $=25.79$). Ablation studies confirm the importance of each component and training stage, underscoring the value of chart-specific alignment in LVLM-based chart understanding and its potential for scalable, accurate data communication.
Abstract
Chart summarization, which focuses on extracting key information from charts and interpreting it in natural language, is crucial for generating and delivering insights through effective and accessible data analysis. Traditional methods for chart understanding and summarization often rely on multi-stage pipelines, which may produce suboptimal semantic alignment between visual and textual information. In comparison, recently developed LLM-based methods are more dependent on the capability of foundation images or languages, while ignoring the characteristics of chart data and its relevant challenges. To address these limitations, we propose ChartAdapter, a novel lightweight transformer module designed to bridge the gap between charts and textual summaries. ChartAdapter employs learnable query vectors to extract implicit semantics from chart data and incorporates a cross-modal alignment projector to enhance vision-to-language generative learning. By integrating ChartAdapter with an LLM, we enable end-to-end training and efficient chart summarization. To further enhance the training, we introduce a three-stage hierarchical training procedure and develop a large-scale dataset specifically curated for chart summarization, comprising 190,618 samples. Experimental results on the standard Chart-to-Text testing set demonstrate that our approach significantly outperforms existing methods, including state-of-the-art models, in generating high-quality chart summaries. Ablation studies further validate the effectiveness of key components in ChartAdapter. This work highlights the potential of tailored LLM-based approaches to advance chart understanding and sets a strong foundation for future research in this area.