EnvTrace: Simulation-Based Semantic Evaluation of LLM Code via Execution Trace Alignment -- Demonstrated at Synchrotron Beamlines

TL;DR

EnvTrace tackles the challenge of evaluating LLM-generated instrument-control code by measuring semantic, execution-based correctness through trace alignment in a beamline digital twin. It combines a Bluesky-driven simulator with a multi-faceted scoring system that emphasizes state accuracy, temporal dynamics, and continuous process fidelity, going beyond brittle syntactic metrics. The study demonstrates strong performance of top models on simple tasks and meaningful, though more variable, performance on complex control flows, while highlighting the limitations of CodeBLEU and Levenshtein distance as proxies for real-world correctness. The work argues for a symbiotic future where LLMs and digital twins enable safer, autonomous embodied AI in scientific facilities, with practical implications for pre-flight validation, debugging, and cross-beamline benchmarking. Overall, EnvTrace provides a rigorous, interpretable framework for evaluating and debugging instrument-control code, with broad potential to improve reliability and safety in physical AI workflows.

Abstract

Evaluating large language models (LLMs) for instrument control requires methods that go beyond standard, stateless algorithmic benchmarks, since the behavior of physical systems cannot be fully captured by unit tests alone. Here we introduce EnvTrace, a simulation-based method that evaluates execution traces to assess semantic code equivalence. EnvTrace is demonstrated with a beamline control-logic digital twin to facilitate the evaluation of instrument control code, with the digital twin itself also enabling the pre-execution validation of live experiments. Over 30 LLMs were evaluated using trace alignment to generate a multi-faceted score for functional correctness across key behavioral dimensions, showing that many top-tier models can approach human-level performance in rapid control-code generation. This is a first step toward a broader vision where LLMs and digital twins work symbiotically: LLMs providing intuitive control and agentic orchestration, and digital twins offering safe and high-fidelity environments, paving the way towards autonomous embodied AI.

Figures (12)

• Figure 1: Illustration of the EnvTrace alignment process and score calculation. The ground-truth and LLM code (predicted code) execution traces are aligned based on their Process Variable (PV) changes. Matched events are shown by connected solid lines, value mismatches by dashed lines, and extra predicted events are shown in dashed box. This alignment is then used for computing scores to evaluate the LLMs.
• Figure 2: An overview of the EnvTrace architecture. LLM-generated code and ground-truth code are executed within an interactive IPython session. This session communicates with a simulated beamline environment, which runs EPICS IOCs in Docker containers. An EPICS monitor captures all state changes and generates an execution trace. The ground-truth and predicted code traces are then aligned to compute a multi-faceted EnvTrace full score based on state, temporal, and behavioral equivalence.
• Figure 3: Selected model performance on simple-flow tasks in Table \ref{['tab:simple_results']}. Left blue bars show the EnvTrace full score, and right stacked bars give the string-based exact match (bottom, orange) and improvement (top, green) with EnvTrace accuracy.
• Figure 4: Model performance on complex-flow tasks. Left blue bars give the EnvTrace full score, while right stacked bars provide the string-based exact match (bottom, orange) and improvement (top, green) with EnvTrace accuracy.
• Figure 5: Illustration of the VISION GUI showing an example natural language query for performing in-situ thermal measurements, along with the corresponding LLM-generated code, partial PV traces, and an AI evaluation based on these inputs (query, code, and PV traces).