Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMs

TL;DR

This work assesses whether large vision-language models can robustly comprehend and reason about charts, a multimodal challenge where data tables may be absent. It presents the first comprehensive LVLM evaluation across five chart-oriented tasks using seven benchmarks, comparing GPT-4V, Gemini, Claude-3, and Phi-3 under zero-shot CoT and PAL prompting, plus a detailed qualitative analysis of outputs. Key findings show LVLMs generate fluent, high-level insights but suffer from hallucinations, factual errors, and biases; performance varies by task, with GPT-4V excelling in discriminative ChartQA and Phi-3 performing well on ChartQA* when labels are removed, while Gemini often leads OpenCQA and Chart-to-Text. The study highlights critical limitations and proposes directions for future work, including instruction-tuning for open-source LVLMs and improved evaluation to mitigate hallucinations and bias in chart-centric reasoning. Overall, the results illuminate the current capabilities and gaps of LVLMs in data-visualization reasoning and guide future research toward more reliable chart understanding systems.

Abstract

Natural language is a powerful complementary modality of communication for data visualizations, such as bar and line charts. To facilitate chart-based reasoning using natural language, various downstream tasks have been introduced recently such as chart question answering, chart summarization, and fact-checking with charts. These tasks pose a unique challenge, demanding both vision-language reasoning and a nuanced understanding of chart data tables, visual encodings, and natural language prompts. Despite the recent success of Large Language Models (LLMs) across diverse NLP tasks, their abilities and limitations in the realm of data visualization remain under-explored, possibly due to their lack of multi-modal capabilities. To bridge the gap, this paper presents the first comprehensive evaluation of the recently developed large vision language models (LVLMs) for chart understanding and reasoning tasks. Our evaluation includes a comprehensive assessment of LVLMs, including GPT-4V and Gemini, across four major chart reasoning tasks. Furthermore, we perform a qualitative evaluation of LVLMs' performance on a diverse range of charts, aiming to provide a thorough analysis of their strengths and weaknesses. Our findings reveal that LVLMs demonstrate impressive abilities in generating fluent texts covering high-level data insights while also encountering common problems like hallucinations, factual errors, and data bias. We highlight the key strengths and limitations of chart comprehension tasks, offering insights for future research.

Figures (9)

• Figure 1: Chart comprehension and reasoning tasks.
• Figure 2: Figure (a) is where the Gemini is successful in 0-shot CoT, but the GPT-4V fails. Figure (b) shows the GPT-4V's success in PAL setup, while the Gemini fails. Here, Red text denotes incorrect, and Green text is correct.
• Figure 3: Confusion matrices for different LVLMs on the ChartFC dataset.
• Figure 4: Both Gemini and GPT-4V fail to identify trends (a). Out-of-context but relevant information generated by Gemini (b). Here, Red text indicates incorrect facts, Pink text denotes out-of-context, and Blue text represents domain-specific details. '…' indicates abbreviated text for brevity.
• Figure 5: The figure is where the GPT-4V model shows some potential bias in the response. Here, Orange text denotes potentially biased output that is out-of-context of the chart data.'…' indicates abbreviated text for brevity.