End-to-End Chart Summarization via Visual Chain-of-Thought in Vision-Language Models

TL;DR

End-to-End Visual Chain-of-Thought (V-CoT) for Chart Summarization introduces an LVLM-based, end-to-end pipeline that reads chart images and generates textual summaries while implicitly performing visual reasoning. The method factorizes the generation into a visual reasoning stage and a summary generation stage, optimizing the joint likelihood $P(S,V|I) = P(V|I)\;P(S|I,V)$ with $F_I = E_{image}(I)$ and $S$ produced by a decoder $D_{text}$, where the reasoning steps $V$ are embedded in hidden states. Training employs instruction fine-tuning on Chart-Sum-QA with loss $\mathcal{L}^{(k)} = - \log P(S^{(k)}|I^{(k)}) = - \sum_{i=1}^{m_k} \log P(s_i^{(k)}|I^{(k)}, s_{<i}^{(k)})$, augmented by data augmentation, curriculum learning, and LoRA-based optimization. On Chart-Sum-QA, V-CoT achieves state-of-the-art automatic metrics (BLEU, BLEURT, CIDEr, CS) and superior human judgments in matching data and reasoning, establishing a new benchmark for end-to-end chart summarization with LVLMs.

Abstract

Automated chart summarization is crucial for enhancing data accessibility and enabling efficient information extraction from visual data. While recent advances in visual-language models (VLMs) have demonstrated promise, existing methods often suffer from limitations in matching the generated summary to the chart data and in reasoning about complex chart patterns. This paper introduces End-to-End Visual Chain-of-Thought (V-CoT) for chart summarization, a novel approach optimized for Large Vision-Language Models (LVLMs). Our method directly trains an LVLM to process chart images and generate textual summaries in an end-to-end fashion, eliminating the need for explicit chart parsing modules. We incorporate a visual Chain-of-Thought mechanism through instruction fine-tuning, implicitly guiding the LVLM to perform visual reasoning steps during summary generation. Evaluated on the large-scale Chart-Sum-QA dataset, our V-CoT method significantly outperforms state-of-the-art baselines across a range of automatic metrics, including BLEU, BLEURT, CIDEr, and CS, and demonstrates superior matching degree and reasoning correctness in human evaluations. Ablation studies and detailed analyses further validate the effectiveness and robustness of our proposed approach, establishing a new benchmark for end-to-end chart summarization.