SpatialBench: Can Agents Analyze Real-World Spatial Biology Data?

TL;DR

SpatialBench tackles the challenge of extracting biologically meaningful insights from messy real-world spatial data by introducing a benchmark of 146 verifiable problems derived from actual spatial workflows. It systematically evaluates frontier AI agents under multiple harnesses, platforms, and task categories, revealing pervasive model–task–platform interactions and that harness design substantially influences performance. The study shows current base accuracies are modest (roughly 20–38%), with domain knowledge and efficient instruction-following being critical differentiators, and highlights the necessity of platform-aware, instrumented evaluation for progress. By serving as both a measurement tool and a diagnostic framework, SpatialBench enables test-driven development of spatial data agents that operate faithfully, transparently, and reproducibly on real datasets.

Abstract

Spatial transcriptomics assays are rapidly increasing in scale and complexity, making computational analysis a major bottleneck in biological discovery. Although frontier AI agents have improved dramatically at software engineering and general data analysis, it remains unclear whether they can extract biological insight from messy, real-world spatial datasets. We introduce SpatialBench, a benchmark of 146 verifiable problems derived from practical spatial analysis workflows spanning five spatial technologies and seven task categories. Each problem provides a snapshot of experimental data immediately prior to an analysis step and a deterministic grader that evaluates recovery of a key biological result. Benchmark data on frontier models shows that base model accuracy remains low (20-38% across model families), with strong model-task and model-platform interactions. Harness design has a large empirical effect on performance, indicating that tools, prompts, control flow, and execution environment should be evaluated and improved as first-class objects. SpatialBench serves both as a measurement tool and a diagnostic lens for developing agents that can interact with real spatial datasets faithfully, transparently, and reproducibly.

Figures (9)

• Figure 1: Overview of SpatialBench.
• Figure 2: SpatialBench benchmark construction.
• Figure 3: Aggregate model performance across SpatialBench.
• Figure 4: Model accuracy stratified by task category.
• Figure 5: Model accuracy stratified by experimental platform.