Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What Lies Beneath: A Call for Distribution-based Visual Question & Answer Datasets

Jill P. Naiman, Daniel J. Evans, JooYoung Seo

TL;DR

The paper argues that current VQA benchmarks inadequately assess reasoning about data distributions in scientific charts, particularly when chart marks do not map directly to underlying data. It introduces a synthetic histogram dataset with ground-truth distributions, bounding boxes, and JSON-encoded figures and questions to enable distribution-aware reasoning. The study compares human annotators and a small prompting-enabled LMM under a structured prompt regime, showing that LMMs can match statistics-background humans on certain tasks and emphasizing the impact of prompt design. This work provides an open-source resource to foster distribution-aware VQA research and advocates benchmarks that test reasoning over data transformations beyond direct chart data.

Abstract

Visual Question Answering (VQA) has become an important benchmark for assessing how large multimodal models (LMMs) interpret images. However, most VQA datasets focus on real-world images or simple diagrammatic analysis, with few focused on interpreting complex scientific charts. Indeed, many VQA datasets that analyze charts do not contain the underlying data behind those charts or assume a 1-to-1 correspondence between chart marks and underlying data. In reality, charts are transformations (i.e. analysis, simplification, modification) of data. This distinction introduces a reasoning challenge in VQA that the current datasets do not capture. In this paper, we argue for a dedicated VQA benchmark for scientific charts where there is no 1-to-1 correspondence between chart marks and underlying data. To do so, we survey existing VQA datasets and highlight limitations of the current field. We then generate synthetic histogram charts based on ground truth data, and ask both humans and a large reasoning model questions where precise answers depend on access to the underlying data. We release the open-source dataset, including figures, underlying data, distribution parameters used to generate the data, and bounding boxes for all figure marks and text for future research.

What Lies Beneath: A Call for Distribution-based Visual Question & Answer Datasets

TL;DR

The paper argues that current VQA benchmarks inadequately assess reasoning about data distributions in scientific charts, particularly when chart marks do not map directly to underlying data. It introduces a synthetic histogram dataset with ground-truth distributions, bounding boxes, and JSON-encoded figures and questions to enable distribution-aware reasoning. The study compares human annotators and a small prompting-enabled LMM under a structured prompt regime, showing that LMMs can match statistics-background humans on certain tasks and emphasizing the impact of prompt design. This work provides an open-source resource to foster distribution-aware VQA research and advocates benchmarks that test reasoning over data transformations beyond direct chart data.

Abstract

Visual Question Answering (VQA) has become an important benchmark for assessing how large multimodal models (LMMs) interpret images. However, most VQA datasets focus on real-world images or simple diagrammatic analysis, with few focused on interpreting complex scientific charts. Indeed, many VQA datasets that analyze charts do not contain the underlying data behind those charts or assume a 1-to-1 correspondence between chart marks and underlying data. In reality, charts are transformations (i.e. analysis, simplification, modification) of data. This distinction introduces a reasoning challenge in VQA that the current datasets do not capture. In this paper, we argue for a dedicated VQA benchmark for scientific charts where there is no 1-to-1 correspondence between chart marks and underlying data. To do so, we survey existing VQA datasets and highlight limitations of the current field. We then generate synthetic histogram charts based on ground truth data, and ask both humans and a large reasoning model questions where precise answers depend on access to the underlying data. We release the open-source dataset, including figures, underlying data, distribution parameters used to generate the data, and bounding boxes for all figure marks and text for future research.
Paper Structure (5 sections, 2 figures, 1 table)

This paper contains 5 sections, 2 figures, 1 table.

Table of Contents

  1. Related Work in Scientific Chart VQA
  2. Figure & Question Dataset Generation
  3. Human & LMM Prompting
  4. Results and Discussion
  5. Caveats, Conclusions, and Future Work

Figures (2)

  • Figure 1: Example histogram (left) along with results of human annotators (second from left, second from right) vs LMM (right) estimations of median of underlying histogram data across 80 histograms. Residuals from the ground truth answers are shown in bottom panels. The green "X"'s in the LMM plot denotes a incorrectly formatted response from the LMM at the specified ground truth median (LMM refuses to answer, or median is approximately an order of magnitude outside plotted range).
  • Figure 2: Human annotators and LMM estimate differences from ground truth (GT) for number of gaussians (#G) in the underlying histogram distributions.