Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TRAIL: Trace Reasoning and Agentic Issue Localization

Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan, Rebecca Qian

TL;DR

TRAIL addresses the need for scalable, granular evaluation of agentic workflows by introducing a fine-grained error taxonomy and a grounded, open-source trace benchmark drawn from GAIA and SWE-Bench. The study systematically analyzes how modern long-context LLMs perform on trace debugging, revealing substantial limitations even for state-of-the-art models and highlighting the crucial role of reasoning and trace length. Through expert annotations and comprehensive analyses across reasoning, planning, and execution errors, TRAIL demonstrates both the coverage of real-world fault classes and the challenges of long-context trace interpretation. The work provides a foundation for scalable debugging and evaluation of complex agentic systems and suggests directions for improving trace observability and LLM-based evaluation methodologies.

Abstract

The increasing adoption of agentic workflows across diverse domains brings a critical need to scalably and systematically evaluate the complex traces these systems generate. Current evaluation methods depend on manual, domain-specific human analysis of lengthy workflow traces - an approach that does not scale with the growing complexity and volume of agentic outputs. Error analysis in these settings is further complicated by the interplay of external tool outputs and language model reasoning, making it more challenging than traditional software debugging. In this work, we (1) articulate the need for robust and dynamic evaluation methods for agentic workflow traces, (2) introduce a formal taxonomy of error types encountered in agentic systems, and (3) present a set of 148 large human-annotated traces (TRAIL) constructed using this taxonomy and grounded in established agentic benchmarks. To ensure ecological validity, we curate traces from both single and multi-agent systems, focusing on real-world applications such as software engineering and open-world information retrieval. Our evaluations reveal that modern long context LLMs perform poorly at trace debugging, with the best Gemini-2.5-pro model scoring a mere 11% on TRAIL. Our dataset and code are made publicly available to support and accelerate future research in scalable evaluation for agentic workflows.

TRAIL: Trace Reasoning and Agentic Issue Localization

TL;DR

TRAIL addresses the need for scalable, granular evaluation of agentic workflows by introducing a fine-grained error taxonomy and a grounded, open-source trace benchmark drawn from GAIA and SWE-Bench. The study systematically analyzes how modern long-context LLMs perform on trace debugging, revealing substantial limitations even for state-of-the-art models and highlighting the crucial role of reasoning and trace length. Through expert annotations and comprehensive analyses across reasoning, planning, and execution errors, TRAIL demonstrates both the coverage of real-world fault classes and the challenges of long-context trace interpretation. The work provides a foundation for scalable debugging and evaluation of complex agentic systems and suggests directions for improving trace observability and LLM-based evaluation methodologies.

Abstract

The increasing adoption of agentic workflows across diverse domains brings a critical need to scalably and systematically evaluate the complex traces these systems generate. Current evaluation methods depend on manual, domain-specific human analysis of lengthy workflow traces - an approach that does not scale with the growing complexity and volume of agentic outputs. Error analysis in these settings is further complicated by the interplay of external tool outputs and language model reasoning, making it more challenging than traditional software debugging. In this work, we (1) articulate the need for robust and dynamic evaluation methods for agentic workflow traces, (2) introduce a formal taxonomy of error types encountered in agentic systems, and (3) present a set of 148 large human-annotated traces (TRAIL) constructed using this taxonomy and grounded in established agentic benchmarks. To ensure ecological validity, we curate traces from both single and multi-agent systems, focusing on real-world applications such as software engineering and open-world information retrieval. Our evaluations reveal that modern long context LLMs perform poorly at trace debugging, with the best Gemini-2.5-pro model scoring a mere 11% on TRAIL. Our dataset and code are made publicly available to support and accelerate future research in scalable evaluation for agentic workflows.
Paper Structure (66 sections, 7 figures, 6 tables)

This paper contains 66 sections, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Relevant Work
  3. LLM-as-a-Judge
  4. Agentic Evaluation
  5. Agent Benchmarks
  6. Traces and Error Taxonomies
  7. Agentic Error Taxonomy
  8. Reasoning Errors
  9. Hallucinations
  10. Information Processing
  11. Decision Making
  12. Output Generation
  13. System Execution Errors
  14. Configuration Issues
  15. API and System Issues
  16. ...and 51 more sections

Figures (7)

  • Figure 1: Illustration of the TRAIL taxonomy of errors
  • Figure 2: TRAIL trace's span structure and error examples
  • Figure 3: TRAIL Dataset Statistics
  • Figure 4: Heatmap for Error Category F1 across models; categories are ordered left to right based on their support
  • Figure 5: Input Token Length Distributions (plotted in logscale) across TRAIL tasks w.r.t two different models for raw trace json inputs. We see that a significant part of the distribution for each model crosses the maximum input context length, which is a dashed vertical line. Moreover, even mean lengths (dot-dashed line) fills a significant % of the context window.
  • ...and 2 more figures