Automated Data Visualization from Natural Language via Large Language Models: An Exploratory Study

TL;DR

This study investigates whether Large Language Models (LLMs) can automate data visualization from natural-language descriptions grounded in tabular data. By formulating NL2Vis around a Visualization Query Language (VQL) and testing several prompt designs, in-context learning regimes, and iterative updating strategies, the authors compare finetuned and inference-only LLMs against traditional NL2Vis baselines on the nvBench dataset. They find that LLMs generally outperform baselines, with inference-only models improving with more demonstrations and multi-turn prompting, and that iterative approaches like chain-of-thought, role-playing, self-repair, and code-interpreter can substantially mitigate failures, especially in data-filtering and axis specification. The work highlights practical implications for deploying NL2Vis via LLMs, including the importance of encoding tables as code, leveraging diverse cross-domain demonstrations, and adopting multi-turn optimization to handle complex visual analyses in real-world settings.

Abstract

The Natural Language to Visualization (NL2Vis) task aims to transform natural-language descriptions into visual representations for a grounded table, enabling users to gain insights from vast amounts of data. Recently, many deep learning-based approaches have been developed for NL2Vis. Despite the considerable efforts made by these approaches, challenges persist in visualizing data sourced from unseen databases or spanning multiple tables. Taking inspiration from the remarkable generation capabilities of Large Language Models (LLMs), this paper conducts an empirical study to evaluate their potential in generating visualizations, and explore the effectiveness of in-context learning prompts for enhancing this task. In particular, we first explore the ways of transforming structured tabular data into sequential text prompts, as to feed them into LLMs and analyze which table content contributes most to the NL2Vis. Our findings suggest that transforming structured tabular data into programs is effective, and it is essential to consider the table schema when formulating prompts. Furthermore, we evaluate two types of LLMs: finetuned models (e.g., T5-Small) and inference-only models (e.g., GPT-3.5), against state-of-the-art methods, using the NL2Vis benchmarks (i.e., nvBench). The experimental results reveal that LLMs outperform baselines, with inference-only models consistently exhibiting performance improvements, at times even surpassing fine-tuned models when provided with certain few-shot demonstrations through in-context learning. Finally, we analyze when the LLMs fail in NL2Vis, and propose to iteratively update the results using strategies such as chain-of-thought, role-playing, and code-interpreter. The experimental results confirm the efficacy of iterative updates and hold great potential for future study.

Figures (13)

• Figure 1: An illustration of NL2Vis. The framework presents our investigation into prompt engineering (RQ1), overall performance (RQ2), and iterative updating (RQ3).
• Figure 2: An example to illustrate the usage of prompt in in-context learning of LLMs, the $k$ demonstration examples, the test item, and the CoT prompt.
• Figure 3: The pipeline of NL2Vis. The task process flows from (a) feeding data into the LLMs through prompts, to (b) generating visualization queries, and finally to (c) rendering language-aware visual charts.
• Figure 4: A summary of approaches explored to transforming the table into textual prompts. (A) Table Serialization flattens database tables into linear schemas, (B) Table Summarization distills table content into concise descriptions, (C) Table Markup Formatting converts data into XML, Markdown, and CSV formats, and (D) Table Programming translates table structures into code representations.
• Figure 5: An illustration of ASTs of VQL queries. Even though they may not be exactly matched, their execution results are identical.