AstroVisBench: A Code Benchmark for Scientific Computing and Visualization in Astronomy

TL;DR

AstroVisBench introduces the first end-to-end benchmark for scientific computing and visualization in astronomy, evaluating how LLMs support complete research workflows from data processing to visualization. It combines an execution-based processing evaluation with expert- and LLM-driven visualization assessment, using 864 tasks drawn from 110 publicly available notebooks to reflect real-world astronomy data (catalogs, spectra, images) and domain-specific tools. The framework reveals a substantial gap in current LLMs’ ability to produce correct, domain-aligned processing and visualization outputs, even when code executes, underscoring the need for domain knowledge and specialized APIs. By providing a rich dataset, a novel evaluation pipeline, and cross-model results across eight LLMs, AstroVisBench offers a path forward for developing visualization-based workflows and broader AI-assisted scientific discovery across disciplines.

Abstract

Large Language Models (LLMs) are being explored for applications in scientific research, including their capabilities to synthesize literature, answer research questions, generate research ideas, and even conduct computational experiments. Ultimately, our goal is for these to help scientists derive novel scientific insights. In many areas of science, such insights often arise from processing and visualizing data to understand its patterns. However, evaluating whether an LLM-mediated scientific workflow produces outputs conveying the correct scientific insights is challenging to evaluate and has not been addressed in past work. We introduce AstroVisBench, the first benchmark for both scientific computing and visualization in the astronomy domain. AstroVisBench judges a language model's ability to both (1) create astronomy-specific workflows to process and analyze data and (2) visualize the results of these workflows through complex plots. Our evaluation of visualizations uses a novel LLM-as-a-judge workflow, which is validated against annotation by five professional astronomers. Using AstroVisBench we present an evaluation of state-of-the-art language models, showing a significant gap in their ability to engage in astronomy research as useful assistants. This evaluation provides a strong end-to-end evaluation for AI scientists that offers a path forward for the development of visualization-based workflows, which are central to a broad range of domains from physics to biology.

Figures (6)

• Figure 1: An overview of AstroVisBench, evaluating astronomy research workflow implementation that leads to a visualization. In a Jupyter notebook environment, given a task query $Q$ and the task context (i.e., code prior to the cell-under-test), a subject LLM generates code $\hat{c}$ that is validated as to whether it correctly leads to the right visualization (Section \ref{['sec:task_setup']}). There are two types of tasks: processing tasks$t_\text{process}$ involve scientific computing necessary to prepare for the visualization, and visualization tasks$t_\text{visualize}$ involve code that creates a visualization (Section \ref{['sec:sourcing_construction']}). Processing tasks are evaluated by executing the ground truth and generated code, and comparing the values of key products necessary for the visualization (Section \ref{['sec:processeval']}). Visualization tasks use a VLM-as-judge we show to correlate highly with expert judgments from professional astronomers (Section \ref{['sec:visualeval']}).
• Figure 2: To construct benchmark tasks, we trace dependencies from visualization cells within a notebook, and split these dependencies into three stages, jointly merging these original cells and generating queries for each stage.
• Figure 3: Examples of visualizations in AstroVisBench showing (a) a color-magnitude diagram before and after point spread function correction, (b) a spatially integrated spectral energy distribution (left) and associated spatially resolved intensity map (right), (c) a wide-field all-sky projection of galaxy source counts within a survey footprint, (d) a Kepler mission source pixel map, (e) a pixel-level flux map, (f) galaxy spectra featuring bright emission lines, (g) corrected and uncorrected time-series light curves, and (h) a wide-field image.
• Figure 4: Breakdown on the types and counts of execution errors resulting from the LLM evaluation on both processing and visualization tasks.
• Figure 5: Overview of astronomy-specific Python libraries and functions grouped by technical or topical field.