Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AgentRx: Diagnosing AI Agent Failures from Execution Trajectories

Shraddha Barke, Arnav Goyal, Alind Khare, Avaljot Singh, Suman Nath, Chetan Bansal

TL;DR

AgentRx tackles the challenge of diagnosing AI agent failures by introducing a domain-agnostic framework that synthesizes constraints from tool schemas and policies, builds an auditable violation log, and uses an LLM-based judge to localize the first unrecoverable failure and assign a cross-domain category. It provides an open-source benchmark of 115 failed trajectories across API workflows, incident management, and web/file tasks, complemented by a grounded theory-derived taxonomy of nine root-cause categories. The framework demonstrates substantial improvements in both step localization and failure attribution across three domains, underscoring the value of trajectory-level constraints as diagnostic signals. Together, the benchmark, taxonomy, and AgentRx pipeline offer a practical, scalable approach to reliability-aware debugging of complex agentic systems.

Abstract

AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 115 failed trajectories spanning structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AGENTRX, an automated domain-agnostic diagnostic framework that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. Our framework improves step localization and failure attribution over existing baselines across three domains.

AgentRx: Diagnosing AI Agent Failures from Execution Trajectories

TL;DR

AgentRx tackles the challenge of diagnosing AI agent failures by introducing a domain-agnostic framework that synthesizes constraints from tool schemas and policies, builds an auditable violation log, and uses an LLM-based judge to localize the first unrecoverable failure and assign a cross-domain category. It provides an open-source benchmark of 115 failed trajectories across API workflows, incident management, and web/file tasks, complemented by a grounded theory-derived taxonomy of nine root-cause categories. The framework demonstrates substantial improvements in both step localization and failure attribution across three domains, underscoring the value of trajectory-level constraints as diagnostic signals. Together, the benchmark, taxonomy, and AgentRx pipeline offer a practical, scalable approach to reliability-aware debugging of complex agentic systems.

Abstract

AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 115 failed trajectories spanning structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AGENTRX, an automated domain-agnostic diagnostic framework that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. Our framework improves step localization and failure attribution over existing baselines across three domains.
Paper Structure (40 sections, 4 equations, 2 figures, 9 tables)

This paper contains 40 sections, 4 equations, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. The AgentRx Benchmark
  3. Grounded Theory Annotation of Agent Failures
  4. A Cross-Domain Failure Taxonomy
  5. AgentRx Framework Formulation
  6. Global and Dynamic Constraint Synthesis
  7. Guarded Constraints and Evaluation
  8. Validation Log
  9. Judge for Root Cause Attribution
  10. Experimental Evaluation
  11. Evaluation Metrics and Experimental Settings
  12. Critical Failure Localization in Agentic Systems
  13. Overall Failure Attribution Performance
  14. Impact of Violations and Taxonomy Checklist
  15. Impact of Constraint Generation Strategies
  16. ...and 25 more sections

Figures (2)

  • Figure 1: Given domain policy, tool schema, and a failed trajectory, AgentRx outputs the critical failure step and a failure category.
  • Figure 2: Failure timelines per trajectory across domains. Each horizontal line is one trajectory. Colored ticks mark detected failures at the corresponding step number (labeled with the category); the star indicates the root-cause step. Magentic-One has 295 total failures, with 68% of trajectories containing at least two failures; Flash has 52 total failures, and $\tau$-bench has 39.