TrajAD: Trajectory Anomaly Detection for Trustworthy LLM Agents

TL;DR

This paper defines Trajectory Anomaly Detection as a runtime auditing task for LLM agents, arguing that reliability requires monitoring intermediate execution rather than only final outputs. It introduces TrajBench, a large, balanced dataset created via a perturb-and-complete strategy to yield both normal and anomalous trajectories with precise error localization, enabling supervised learning for anomaly verdicts and error steps. The authors propose TrajAD, a specialized generative verifier trained on TrajBench with a decoder-only Transformer and LoRA, capable of determining if a trajectory is normal and localizing the first error step, thereby enabling rollback-and-retry during execution. Empirical results show that general-purpose LLMs struggle with anomaly detection and localization, while TrajAD significantly improves both metrics, with strong cross-domain transfer and favorable data-efficiency; larger models or data alone do not fully close the localization gap, underscoring the value of dedicated supervision for process-level safety. Overall, the work shifts agent evaluation toward runtime process auditing, offering a practical path to trustworthy autonomous LLM agents in diverse domains, including embodied AI.

Abstract

We address the problem of runtime trajectory anomaly detection, a critical capability for enabling trustworthy LLM agents. Current safety measures predominantly focus on static input/output filtering. However, we argue that ensuring LLM agents reliability requires auditing the intermediate execution process. In this work, we formulate the task of Trajectory Anomaly Detection. The goal is not merely detection, but precise error localization. This capability is essential for enabling efficient rollback-and-retry. To achieve this, we construct TrajBench, a dataset synthesized via a perturb-and-complete strategy to cover diverse procedural anomalies. Using this benchmark, we investigate the capability of models in process supervision. We observe that general-purpose LLMs, even with zero-shot prompting, struggle to identify and localize these anomalies. This reveals that generalized capabilities do not automatically translate to process reliability. To address this, we propose TrajAD, a specialized verifier trained with fine-grained process supervision. Our approach outperforms baselines, demonstrating that specialized supervision is essential for building trustworthy agents.

Figures (5)

• Figure 1: Overview of the TrajAD framework. We introduce the TrajAD framework to verify the agent's trajectories. At each step, the agent generates a thought, takes an action, and receives an observation, forming an execution unit. The execution trajectory is periodically validated to check whether it remains normal. If all previous steps are valid, execution continues. When an anomaly is detected at step t, the process is halted before step t+1. The trajectory can rollback to the step t-1 and retry instead of restarting the whole task.
• Figure 2: The data construction pipeline for TrajBench. We initialize the process by filtering seed trajectories to ensure a high-quality set of valid golden trajectories. To construct negative samples, we employ a Perturb-and-Complete strategy. We inject a perturbation into a target step $S_t$ and force conditional completion to finish the subsequent trajectory based on the altered context. This process synthesizes three distinct anomaly types: Task Failure ($\mathcal{A}_{fail}$), Process Inefficiency ($\mathcal{A}_{ineff}$), and Unwarranted Continuation ($\mathcal{A}_{unw}$). The resulting dataset features a balanced composition of positive and negative samples, where anomalous trajectories are automatically annotated. The dataset consists of 13 tasks across 5 domains.
• Figure 3: Qualitative Comparison on Redundancy Loops. The baseline model overlooks the repeated cleaning action since it does not affect the final goal state. In contrast, TrajAD detects the redundancy as a Inefficiency process.
• Figure 4: Domain-Specific Performance Analysis. (Left) Macro-F1 across five domains. TrajAD (solid red) forms the outermost envelope, demonstrating consistent robustness. (Right) Exact Match. Baselines exhibit a structural collapse near the center, highlighting their inability to localize errors, whereas TrajAD maintains a functional verification boundary.
• Figure 5: Ablation and Analysis Experiments. (a) Generalization: Comparison of zero-shot transfer (Transfer Model) vs. full supervision. The detection gap is minimal, validating the universality of the learned logic. (b) Scalability: Impact of training data size and model parameters. The 4B model with 50k stratified samples achieves optimal efficiency, outperforming both the full-data 4B model and the larger 8B baseline.