Chart-HQA: A Benchmark for Hypothetical Question Answering in Charts

TL;DR

Chart-HQA addresses inherent output biases in multimodal LLMs by introducing a hypothetical question answering task over charts and a human-machine interactive data synthesis framework (HAI) to generate diverse, high-quality HQA data. HAI combines a Counterfactual Proposal Generator and a Human-feedback Discriminator to automatically create HQA instances from publicly available chart data with expert validation, enabling scalable dataset construction. The resulting Chart-HQA benchmark reveals significant generalization gaps and imbalanced reasoning across 18 MLLMs, even among strong specialists and generalists, underscoring the need for improved counterfactual chart understanding and symbolic reasoning. The work provides a practical methodology for generating challenging chart-reasoning data and highlights important directions for future research in counterfactual reasoning and chart-based visual question answering.

Abstract

Multimodal Large Language Models (MLLMs) have garnered significant attention for their strong visual-semantic understanding. Most existing chart benchmarks evaluate MLLMs' ability to parse information from charts to answer questions. However, they overlook the inherent output biases of MLLMs, where models rely on their parametric memory to answer questions rather than genuinely understanding the chart content. To address this limitation, we introduce a novel Chart Hypothetical Question Answering (HQA) task, which imposes assumptions on the same question to compel models to engage in counterfactual reasoning based on the chart content. Furthermore, we introduce HAI, a human-AI interactive data synthesis approach that leverages the efficient text-editing capabilities of LLMs alongside human expert knowledge to generate diverse and high-quality HQA data at a low cost. Using HAI, we construct Chart-HQA, a challenging benchmark synthesized from publicly available data sources. Evaluation results on 18 MLLMs of varying model sizes reveal that current models face significant generalization challenges and exhibit imbalanced reasoning performance on the HQA task.

Figures (7)

• Figure 1: An example of biased output on charts from MLLMS and proposed hypothetical QA task. (a) Factoid QA results based on the original chart. (b) The response after counterfactual editing of the chart, where the land areas of "China" and "USA" are swapped. (c) The model's answers without the chart image input. (d) Illustration of hypothetical question and the corresponding counterfactual context to be imagined.
• Figure 2: The illustration of our approach for synthesizing hypothetical questions, including two stages that synthesize new instruction proposals, and human verification.
• Figure 3: Counterfactual operations in generated instruction proposals. The inner circle denotes noun objects in charts, the outer circle represents the action against the noun object.
• Figure 4: Human evaluation performance of three data synthesis methods, including human, machine, and our human-machine interaction approach. From left to right, the comparison includes question rationality, complexity, diversity, and synthesis cost (unit: CNY).
• Figure 5: The visualization of examples in ChartQA and Chart-HQA(ours). We use black bold to highlight key reasoning steps of the model and red to mark incorrect reasoning steps.