Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VegaChat: A Robust Framework for LLM-Based Chart Generation and Assessment

Marko Hostnik, Rauf Kurbanov, Yaroslav Sokolov, Artem Trofimov

TL;DR

VegaChat presents a robust NL2VIS framework that converts natural language into declarative Vega-Lite specifications with automated validation and ambiguity handling. It introduces two complementary metrics, Spec Score (deterministic) and Vision Score (multimodal), to enable cross-library evaluation and alignment with human judgments. Evaluations on the NLv Corpus and ChartLLM demonstrate high chart validity and strong correlations with human judgments, while an error-correction feedback loop substantially reduces empty or invalid outputs. The work provides practical tools for producing reliable visualizations and lays groundwork for grammar-constrained generation and data-driven metric weighting in NL2VIS research.

Abstract

Natural-language-to-visualization (NL2VIS) systems based on large language models (LLMs) have substantially improved the accessibility of data visualization. However, their further adoption is hindered by two coupled challenges: (i) the absence of standardized evaluation metrics makes it difficult to assess progress in the field and compare different approaches; and (ii) natural language descriptions are inherently underspecified, so multiple visualizations may be valid for the same query. To address these issues, we introduce VegaChat, a framework for generating, validating, and assessing declarative visualizations from natural language. We propose two complementary metrics: Spec Score, a deterministic metric that measures specification-level similarity without invoking an LLM, and Vision Score, a library-agnostic, image-based metric that leverages a multimodal LLM to assess chart similarity and prompt compliance. We evaluate VegaChat on the NLV Corpus and on the annotated subset of ChartLLM. VegaChat achieves near-zero rates of invalid or empty visualizations, while Spec Score and Vision Score exhibit strong correlation with human judgments (Pearson 0.65 and 0.71, respectively), indicating that the proposed metrics support consistent, cross-library comparison. The code and evaluation artifacts are available at https://zenodo.org/records/17062309.

VegaChat: A Robust Framework for LLM-Based Chart Generation and Assessment

TL;DR

VegaChat presents a robust NL2VIS framework that converts natural language into declarative Vega-Lite specifications with automated validation and ambiguity handling. It introduces two complementary metrics, Spec Score (deterministic) and Vision Score (multimodal), to enable cross-library evaluation and alignment with human judgments. Evaluations on the NLv Corpus and ChartLLM demonstrate high chart validity and strong correlations with human judgments, while an error-correction feedback loop substantially reduces empty or invalid outputs. The work provides practical tools for producing reliable visualizations and lays groundwork for grammar-constrained generation and data-driven metric weighting in NL2VIS research.

Abstract

Natural-language-to-visualization (NL2VIS) systems based on large language models (LLMs) have substantially improved the accessibility of data visualization. However, their further adoption is hindered by two coupled challenges: (i) the absence of standardized evaluation metrics makes it difficult to assess progress in the field and compare different approaches; and (ii) natural language descriptions are inherently underspecified, so multiple visualizations may be valid for the same query. To address these issues, we introduce VegaChat, a framework for generating, validating, and assessing declarative visualizations from natural language. We propose two complementary metrics: Spec Score, a deterministic metric that measures specification-level similarity without invoking an LLM, and Vision Score, a library-agnostic, image-based metric that leverages a multimodal LLM to assess chart similarity and prompt compliance. We evaluate VegaChat on the NLV Corpus and on the annotated subset of ChartLLM. VegaChat achieves near-zero rates of invalid or empty visualizations, while Spec Score and Vision Score exhibit strong correlation with human judgments (Pearson 0.65 and 0.71, respectively), indicating that the proposed metrics support consistent, cross-library comparison. The code and evaluation artifacts are available at https://zenodo.org/records/17062309.
Paper Structure (25 sections, 9 figures, 4 tables)

This paper contains 25 sections, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. NL2VIS Metrics
  4. Declarative Chart Generation
  5. Chart Generator
  6. Auxiliary Components
  7. Evaluation Metrics
  8. Specification Correctness Score
  9. Vision Score
  10. Spec and Vision Score Limitations
  11. ECR, MPB and SEVQ
  12. Experimental Results
  13. Datasets
  14. Chart Generation Results
  15. Chart Generation Results
  16. ...and 10 more sections

Figures (9)

  • Figure 1: Schema of our chart generation system
  • Figure 2: Example of the Spec Score metric giving a score of $50\%$
  • Figure 3: Example of the Vision Score metric giving a score of $58\%$
  • Figure 4: Example where Spec Score assigns a low score of $0.6\%$, while Vision Score assigns $78\%$
  • Figure 5: Distribution of VegaChat scores on the NLV and ChartLLM datasets
  • ...and 4 more figures