Automating High Energy Physics Data Analysis with LLM-Powered Agents

TL;DR

This study demonstrates that LLM-based supervisor-coder agents can be integrated into a Snakemake-driven, reproducible workflow to automate end-to-end high-energy physics data analysis, exemplified by a Higgs to diphoton cross-section study using ATLAS Open Data. By benchmarking diverse LLM architectures and measuring success rates, errors, and costs, the work reveals both feasibility and model-dependent reliability, while highlighting the importance of deterministic workflow management. The framework enables systematic assessment of model capabilities, stability, and limitations in real-world scientific computing environments, and provides baseline code for further development. The results offer practical insights into how agent-based automation can be deployed in collider analyses, with implications for reproducibility, resource planning, and future enhancements in prompting and retrieval-augmented strategies.

Abstract

We present a proof-of-principle study demonstrating the use of large language model (LLM) agents to automate a representative high energy physics (HEP) analysis. Using the Higgs boson diphoton cross-section measurement as a case study with ATLAS Open Data, we design a hybrid system that combines an LLM-based supervisor-coder agent with the Snakemake workflow manager. In this architecture, the workflow manager enforces reproducibility and determinism, while the agent autonomously generates, executes, and iteratively corrects analysis code in response to user instructions. We define quantitative evaluation metrics including success rate, error distribution, costs per specific task, and average number of API calls, to assess agent performance across multi-stage workflows. To characterize variability across architectures, we benchmark a representative selection of state-of-the-art LLMs spanning the Gemini and GPT-5 series, the Claude family, and leading open-weight models. While the workflow manager ensures deterministic execution of all analysis steps, the final outputs still show stochastic variation. Although we set the temperature to zero, other sampling parameters (e.g., top-p, top-k) remained at their defaults, and some reasoning-oriented models internally adjust these settings. Consequently, the models do not produce fully deterministic results. This study establishes the first LLM-agent-driven automated data-analysis framework in HEP, enabling systematic benchmarking of model capabilities, stability, and limitations in real-world scientific computing environments. The baseline code used in this work is available at https://huggingface.co/HWresearch/LLM4HEP. This work was accepted as a poster at the Machine Learning and the Physical Sciences (ML4PS) workshop at NeurIPS 2025. The initial submission was made on August 30, 2025.

Figures (6)

• Figure 1: Illustration of the internal workflow for the supervisor–coder agent. The blue box denotes user-provided prompts, orange boxes denote pre-written standing instructions (system prompts), and green boxes denote dynamically generated prompts created during the workflow. The supervisor decomposes the user’s task into subtasks, maintains an evolving supervisor record, and issues structured prompts to the coder agent. The coder returns generated code, intermediate files, and error messages, which the supervisor evaluates to determine whether the task has been completed or whether further refinement is required. This iterative supervisor–coder loop continues until the coder produces a valid solution or the maximum number of allowed attempts is reached.
• Figure 2: Cross-model comparison of LLM-agent performance. (a) Success fraction for each model-step pair. (b) Error distribution across all failed trials for each model.
• Figure 3: Agent work progression across workflow steps. Average agent work (with standard-deviation error bars) shown for the nine models that successfully completed all stages of the workflow.
• Figure 4: API-call progression across workflow steps. Average API-call counts (with standard-deviation error bars) for the nine models that completed all workflow stages.
• Figure 5: Cost per analysis step across models. The estimated dollar cost per step is computed from the observed token usage and the publicly listed pricing for each model configuration, shown in Appedix \ref{['appendix:model_api_cost']}. This metric captures the effective economic cost of completing the workflow. Higher-capacity models tend to combine lower agent work with higher dollar cost, whereas smaller or open-weight models offer lower absolute cost but substantially reduced reliability.