Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ChartAnchor: Chart Grounding with Structural-Semantic Fidelity

Xinhang Li, Jingbo Zhou, Pengfei Luo, Yixiong Xiao, Tong Xu

TL;DR

ChartAnchor introduces a comprehensive benchmark for chart grounding, unifying chart-to-code generation and controlled chart-to-table reconstruction across 30 chart types and diverse libraries. It provides a four-dimensional evaluation framework—functional validity, visual structure, semantic data fidelity, and perceptual similarity—plus a dedicated semantic data fidelity metric. Across 14 MLLMs, including GPT-4o and Claude-3-7-Sonnet, the study reveals strong structural parsing but persistent challenges in precise data recovery and color accuracy, underscoring the need for integrated symbolic and visual reasoning. The dataset and evaluations lay a foundation for robust, interpretable chart understanding in scientific, financial, and policy domains.

Abstract

Recent advances in multimodal large language models (MLLMs) highlight the need for benchmarks that rigorously evaluate structured chart comprehension. Chart grounding refers to the bidirectional alignment between a chart's visual appearance and the structured semantics. This task requires models to produce a symbolic specification that faithfully captures the chart's visual and structural intent, while also recovering the underlying tabular data with precise values and relationships. Chart grounding directly reflects a model's capabilities in numerical reasoning, multimodal alignment, and structural reconstruction, and has several important applications in real-world scenarios. Existing benchmarks, constrained by narrow chart diversity, isolated tasks, and incomplete evaluation frameworks, fail to holistically assess grounding. To address this, we propose ChartAnchor, a comprehensive benchmark of 8k+ chart-table-code triples spanning 30 chart types drawn from diverse real-world and augmented sources. ChartAnchor introduces two complementary tasks: chart-to-code generation (synthesizing executable code to replicate charts) and controlled chart-to-table reconstruction (extracting exact data with predefined headers), enabling cross-validation of visual and numerical fidelity. A multi-level evaluation framework integrates semantic validation, stylistic analysis, and perceptual metrics to assess both structural and content-level correctness. Extensive experiments on MLLMs reveal critical limitations in numerical precision and code synthesis, emphasizing the need for structured reasoning beyond surface-level perception. By unifying symbolic and data-driven grounding, ChartAnchor establishes a rigorous foundation for chart grounding, offering meaningful insights for advancing MLLMs in scientific, financial, and industrial domains.

ChartAnchor: Chart Grounding with Structural-Semantic Fidelity

TL;DR

ChartAnchor introduces a comprehensive benchmark for chart grounding, unifying chart-to-code generation and controlled chart-to-table reconstruction across 30 chart types and diverse libraries. It provides a four-dimensional evaluation framework—functional validity, visual structure, semantic data fidelity, and perceptual similarity—plus a dedicated semantic data fidelity metric. Across 14 MLLMs, including GPT-4o and Claude-3-7-Sonnet, the study reveals strong structural parsing but persistent challenges in precise data recovery and color accuracy, underscoring the need for integrated symbolic and visual reasoning. The dataset and evaluations lay a foundation for robust, interpretable chart understanding in scientific, financial, and policy domains.

Abstract

Recent advances in multimodal large language models (MLLMs) highlight the need for benchmarks that rigorously evaluate structured chart comprehension. Chart grounding refers to the bidirectional alignment between a chart's visual appearance and the structured semantics. This task requires models to produce a symbolic specification that faithfully captures the chart's visual and structural intent, while also recovering the underlying tabular data with precise values and relationships. Chart grounding directly reflects a model's capabilities in numerical reasoning, multimodal alignment, and structural reconstruction, and has several important applications in real-world scenarios. Existing benchmarks, constrained by narrow chart diversity, isolated tasks, and incomplete evaluation frameworks, fail to holistically assess grounding. To address this, we propose ChartAnchor, a comprehensive benchmark of 8k+ chart-table-code triples spanning 30 chart types drawn from diverse real-world and augmented sources. ChartAnchor introduces two complementary tasks: chart-to-code generation (synthesizing executable code to replicate charts) and controlled chart-to-table reconstruction (extracting exact data with predefined headers), enabling cross-validation of visual and numerical fidelity. A multi-level evaluation framework integrates semantic validation, stylistic analysis, and perceptual metrics to assess both structural and content-level correctness. Extensive experiments on MLLMs reveal critical limitations in numerical precision and code synthesis, emphasizing the need for structured reasoning beyond surface-level perception. By unifying symbolic and data-driven grounding, ChartAnchor establishes a rigorous foundation for chart grounding, offering meaningful insights for advancing MLLMs in scientific, financial, and industrial domains.

Paper Structure

This paper contains 36 sections, 2 equations, 23 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The ChartAnchor Benchmark
  4. Task Definitions
  5. Data Collection
  6. Data Source
  7. Code Generation and Augmentation for Existing Data- sets.
  8. Filtering.
  9. Dataset Analysis
  10. Comparition with other benchmarks
  11. Evaluation Metrics
  12. Semantic Data Fidelity.
  13. Evaluation
  14. Benchmarking Models
  15. Experimental Setting and Resource
  16. ...and 21 more sections

Figures (23)

  • Figure 1: Chart Type Distribution in ChartAnchor.
  • Figure 2: Illustration of our chart-to-code generation task and controlled chart-to-table reconstruction task.
  • Figure 3: Illustration of the Data Collection Pipeline.
  • Figure 4: Chart-level visual statistics.
  • Figure 5: Comparison of Data Attribute Prediction Across Chart Types for Different Models
  • ...and 18 more figures