Benchmarking Autonomy in Scientific Experiments: A Hierarchical Taxonomy for Autonomous Large-Scale Facilities

TL;DR

The Benchmarking Autonomy in Scientific Experiments (BASE) Scale is proposed, a 6-level taxonomy (Levels 0-5) specifically adapted for these unique constraints of Large-Scale User Facilities.

Abstract

The transition from automated data collection to fully autonomous discovery requires a shared vocabulary to benchmark progress. While the automotive industry relies on the SAE J3016 standard, current taxonomies for autonomous science presuppose an owner-operator model that is incompatible with the operational rigidities of Large-Scale User Facilities. Here, we propose the Benchmarking Autonomy in Scientific Experiments (BASE) Scale, a 6-level taxonomy (Levels 0-5) specifically adapted for these unique constraints. Unlike owner-operator models, User Facilities require zero-shot deployment where agents must operate immediately without extensive training periods. We define the specific technical requirements for each tier, identifying the Inference Barrier (Level 3) as the critical latency threshold where decisions shift from scalar feedback to semantic digital twins. Fundamentally, this level extends the decision manifold from spatial exploration to temporal gating, enabling the agent to synchronise acquisition with the onset of transient physical events. By establishing these operational definitions, the BASE Scale provides facility directors, funding bodies, and beamline scientists with a standardised metric to assess risk, define liability, and quantify the intelligence of experimental workflows.

Figures (2)

• Figure 1: The Benchmarking Autonomy in Scientific Experiments (BASE) Scale. A hierarchical taxonomy of experimental agency adapted for Large-Scale Facilities. The scale progresses from Level 0 (Manual) to Level 5 (Autonomous). The critical operational discontinuity occurs at the Inference Barrier (Orange), where the system must invert raw data within the latency budget ($\tau_{lat} < t/10$) to enable Level 3 (Heuristic) control. At this level, agents use physics-based priors (Entropy-Driven Search) to target information-rich features. The Liability Threshold (Purple) marks the transition from human-validated safety to algorithmic safety, requiring a rigorous Safety Case before Level 4 (Supervisory) deployment.
• Figure 2: The Information Efficiency Gap. (a) Level 1 (Scripted Automation): A traditional dense raster scan applies equal incident flux to all spatial voxels. In deterministic failure modes, this wastes $>90\%$ of beamtime measuring the stable, elastic bulk. (b) Level 3 (Semantic Autonomy): An active heuristic agent uses a physics-based prior (e.g., surface kurtosis) to drive an adaptive scan. The agent autonomously clusters measurement points around the information-rich stress concentrators (notches), maximising Entropy-Scaled Measurement Efficiency ($E_{\text{SME}}$).