Visualization Generation with Large Language Models: An Evaluation

TL;DR

The paper addresses NL2VIS by systematically evaluating six open-source LLMs across eight prompt strategies to generate Vega-Lite specifications from nvBench data. It introduces a baseline evaluation framework, reports large performance disparities tied to prompt design, chart type, and model, and uncovers counterintuitive findings such as the limited value of deeper reasoning and the benefit of explicit exemplars. It also identifies nvBench limitations (query errors, mapping issues, and data/label inaccuracies) that can skew results and suggests benchmark improvements and more grounded prompting. Overall, the work provides a foundation for reliable, interpretable NL2VIS systems and guides future research on prompt design, evaluation methodologies, and benchmark quality.

Abstract

The frequent need for analysts to create visualizations to derive insights from data has driven extensive research into the generation of natural Language to Visualization (NL2VIS). While recent progress in large language models (LLMs) suggests their potential to effectively support NL2VIS tasks, existing studies lack a systematic investigation into the performance of different LLMs under various prompt strategies. This paper addresses this gap and contributes a crucial baseline evaluation of LLMs' capabilities in generating visualization specifications of NL2VIS tasks. Our evaluation utilizes the nvBench dataset, employing six representative LLMs and eight distinct prompt strategies to evaluate their performance in generating six target chart types using the Vega-Lite visualization specification. We assess model performance with multiple metrics, including vis accuracy, validity and legality. Our results reveal substantial performance disparities across prompt strategies, chart types, and LLMs. Furthermore, based on the evaluation results, we uncover several counterintuitive behaviors across these dimensions, and propose directions for enhancing the NL2VIS benchmark to better support future NL2VIS research.

Figures (6)

• Figure 1: A summary of prompt strategies:Each column corresponds to a study that utilizes or evaluates prompt strategies. Each row corresponds to a prompt strategy category. The strategies can be grouped into four categories: In-Context Learning (gray), Chain-of-Thought and its variants (yellow), Rationale Engineering (blue) and Problem Decomposition (green).
• Figure 2: Legality heat map of prompts The figure presents the legality performance heat map of different prompt strategies.
• Figure 3: Legality heat map of chart types The figure presents the legality performance heat map of different chart types in the nvBench dataset.
• Figure 4: Legality heat map of LLMs. The figure presents the legality performance heat map of different LLMs.
• Figure 5: Counterintuitive findings observed in results. The figure presents illustrative examples corresponding to the findings discussed in Section \ref{['subsection:deep-findings']}. First, regarding prompting strategies: (a) illustrates ambiguous reasoning trajectories that emerge under the Zero-shot-CoT and Least-to-Most prompting strategies. (b) highlights issues in the plans generated by LLMs. Second, regarding chart types: (c) shows comparisons of queries from pie charts and bar charts, where bar-chart queries contain richer and more fine-grained information than pie-chart queries. (d) depicts queries for scatter charts and grouped scatter charts, showing that scatter-chart queries exhibit a high proportion of task-specific terminology, whereas grouped-scatter queries contain a higher proportion of general-purpose words. Third, regarding LLM behavior: (e) presents the first three feedback examples used by Llama-3 within the Self-Refine framework.