Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents

Kuangshi Ai, Haichao Miao, Kaiyuan Tang, Nathaniel Gorski, Jianxin Sun, Guoxi Liu, Helgi I. Ingolfsson, David Lenz, Hanqi Guo, Hongfeng Yu, Teja Leburu, Michael Molash, Bei Wang, Tom Peterka, Chaoli Wang, Shusen Liu

Abstract

Recent advances in large language models (LLMs) have enabled agentic systems that translate natural language intent into executable scientific visualization (SciVis) tasks. Despite rapid progress, the community lacks a principled and reproducible benchmark for evaluating these emerging SciVis agents in realistic, multi-step analysis settings. We present SciVisAgentBench, a comprehensive and extensible benchmark for evaluating scientific data analysis and visualization agents. Our benchmark is grounded in a structured taxonomy spanning four dimensions: application domain, data type, complexity level, and visualization operation. It currently comprises 108 expert-crafted cases covering diverse SciVis scenarios. To enable reliable assessment, we introduce a multimodal outcome-centric evaluation pipeline that combines LLM-based judging with deterministic evaluators, including image-based metrics, code checkers, rule-based verifiers, and case-specific evaluators. We also conduct a validity study with 12 SciVis experts to examine the agreement between human and LLM judges. Using this framework, we evaluate representative SciVis agents and general-purpose coding agents to establish initial baselines and reveal capability gaps. SciVisAgentBench is designed as a living benchmark to support systematic comparison, diagnose failure modes, and drive progress in agentic SciVis. The benchmark is available at https://scivisagentbench.github.io/.

SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agents

Abstract

Recent advances in large language models (LLMs) have enabled agentic systems that translate natural language intent into executable scientific visualization (SciVis) tasks. Despite rapid progress, the community lacks a principled and reproducible benchmark for evaluating these emerging SciVis agents in realistic, multi-step analysis settings. We present SciVisAgentBench, a comprehensive and extensible benchmark for evaluating scientific data analysis and visualization agents. Our benchmark is grounded in a structured taxonomy spanning four dimensions: application domain, data type, complexity level, and visualization operation. It currently comprises 108 expert-crafted cases covering diverse SciVis scenarios. To enable reliable assessment, we introduce a multimodal outcome-centric evaluation pipeline that combines LLM-based judging with deterministic evaluators, including image-based metrics, code checkers, rule-based verifiers, and case-specific evaluators. We also conduct a validity study with 12 SciVis experts to examine the agreement between human and LLM judges. Using this framework, we evaluate representative SciVis agents and general-purpose coding agents to establish initial baselines and reveal capability gaps. SciVisAgentBench is designed as a living benchmark to support systematic comparison, diagnose failure modes, and drive progress in agentic SciVis. The benchmark is available at https://scivisagentbench.github.io/.

Paper Structure

This paper contains 45 sections, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. General Agent Evaluation Frameworks
  4. Agentic Workflows and Visualization Systems
  5. Visualization Benchmarks and Evaluation Methods
  6. Benchmark Goals and Design Requirements
  7. Overview of SciVisAgentBench
  8. Evaluation Framework
  9. Evaluation Taxonomy
  10. Task Definition and Outcome-Centric Structuring
  11. Multimodal Outcome Evaluation with LLM Judges
  12. Deterministic Evaluators
  13. Evaluation Pipeline
  14. Validity Study
  15. Task Validity
  16. ...and 30 more sections

Figures (2)

  • Figure 1: This gallery displays 24 representative benchmark cases from SciVisAgentBench, organized into four task suites: ParaView visualization, molecular visualization, bioimage visualization, and topology visualization. Each case shows the ground-truth visualization and its associated visualization operations.
  • Figure 2: Overview of the SciVisAgentBench execution and evaluation pipeline. LLM-based SciVis agents interact with SciVis tools and datasets to perform benchmark tasks and produce recorded outcomes. These outcomes are evaluated against ground-truth references using a combination of MLLM judges and deterministic evaluators, yielding visualization quality, efficiency, and final benchmark scores for recorded outcomes.