Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code

TL;DR

The paper introduces PandasPlotBench, a human-curated, leakage-free benchmark with 175 synthetic tasks to evaluate LLMs as plotting-code assistants for Pandas DataFrames. It analyzes multiple models (e.g., GPT-4o, Claude, Gemini, Llama) across three plotting libraries (Matplotlib, Seaborn, Plotly) and investigates how task length and prompt structure affect performance. Key findings include high success for Matplotlib and Seaborn, notable gaps with Plotly (approximately 22% incorrect code), and only minor performance loss when task descriptions are shortened; task-based scoring correlates strongly with human judgments. The benchmark, data, and code are openly available, and the study provides a scalable framework for extending evaluation to additional libraries and languages, ultimately aiding the design of better AI-assisted visualization tools. The work emphasizes the practical impact of user-interface considerations and urges broader validation across models and environments.

Abstract

This paper introduces the human-curated PandasPlotBench dataset, designed to evaluate language models' effectiveness as assistants in visual data exploration. Our benchmark focuses on generating code for visualizing tabular data - such as a Pandas DataFrame - based on natural language instructions, complementing current evaluation tools and expanding their scope. The dataset includes 175 unique tasks. Our experiments assess several leading Large Language Models (LLMs) across three visualization libraries: Matplotlib, Seaborn, and Plotly. We show that the shortening of tasks has a minimal effect on plotting capabilities, allowing for the user interface that accommodates concise user input without sacrificing functionality or accuracy. Another of our findings reveals that while LLMs perform well with popular libraries like Matplotlib and Seaborn, challenges persist with Plotly, highlighting areas for improvement. We hope that the modular design of our benchmark will broaden the current studies on generating visualizations. Our dataset and benchmark code are available online: https://huggingface.co/datasets/JetBrains-Research/PandasPlotBench; https://github.com/JetBrains-Research/PandasPlotBench.