From Features to Actions: Explainability in Traditional and Agentic AI Systems

TL;DR

The paper reframes explainability from static prediction to agentic trajectory analysis, showing that traditional feature attributions fail to diagnose per-run failures in multi-step AI agents. It introduces Minimal Explanation Packets (MEP) to couple explanations with execution context and verification signals, and conducts a cross-paradigm evaluation using static text classification and agentic benchmarks TAU-bench Airline and AssistantBench. Static explanations yield stable feature rankings but cannot pinpoint why a specific trajectory fails, while trace-based rubric evaluations consistently localize execution-level breakdowns such as state tracking inconsistency and tool incorrectness, predicting success more reliably. A bridging experiment demonstrates that rubric-based features can inform global importance via SHAP but remain correlative and insufficient for per-run fault localization without trajectory-level traces. The work advocates a paradigm shift toward trajectory-level explainability, emphasizing auditability, verification, and integration with agent evaluation to enable safer, more accountable autonomous AI systems.

Abstract

Over the last decade, explainable AI has primarily focused on interpreting individual model predictions, producing post-hoc explanations that relate inputs to outputs under a fixed decision structure. Recent advances in large language models (LLMs) have enabled agentic AI systems whose behaviour unfolds over multi-step trajectories. In these settings, success and failure are determined by sequences of decisions rather than a single output. While useful, it remains unclear how explanation approaches designed for static predictions translate to agentic settings where behaviour emerges over time. In this work, we bridge the gap between static and agentic explainability by comparing attribution-based explanations with trace-based diagnostics across both settings. To make this distinction explicit, we empirically compare attribution-based explanations used in static classification tasks with trace-based diagnostics used in agentic benchmarks (TAU-bench Airline and AssistantBench). Our results show that while attribution methods achieve stable feature rankings in static settings (Spearman $ρ= 0.86$), they cannot be applied reliably to diagnose execution-level failures in agentic trajectories. In contrast, trace-grounded rubric evaluation for agentic settings consistently localizes behaviour breakdowns and reveals that state tracking inconsistency is 2.7$\times$ more prevalent in failed runs and reduces success probability by 49\%. These findings motivate a shift towards trajectory-level explainability for agentic systems when evaluating and diagnosing autonomous AI behaviour. Resources: https://github.com/VectorInstitute/unified-xai-evaluation-framework https://vectorinstitute.github.io/unified-xai-evaluation-framework

Figures (4)

• Figure 1: Comparison of Minimal Explanation Packet (MEP) structure across static and agentic paradigms. Static MEPs ground explanations in a single input--output pair with optional verification. Agentic MEPs require trajectory-level context spanning multiple steps, tool interactions, and state evolution, with verification signals that assess process-level faithfulness.
• Figure 2: Agent execution loop with explicit state--action--observation semantics. The architecture separates agent execution from interpretability and verification, showing how traces are reconstructed and analyzed to produce Minimal Explanation Packets (MEPs) for auditing and diagnosis.
• Figure 3: Comparison of local (LIME) (a), and global (SHAP) (b) interpretability.
• Figure 4: SHAP summary (beeswarm) plot for rubric-level features. Each point is a run; the x-axis shows the SHAP value (contribution to predicted success), and color indicates feature value (low vs. high). Features are ordered by mean absolute SHAP value.