Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AgentAssay: Token-Efficient Regression Testing for Non-Deterministic AI Agent Workflows

Varun Pratap Bhardwaj

TL;DR

AgentAssay is presented, the first token-efficient framework for regression testing non-deterministic AI agent workflows, achieving 78-100% cost reduction while maintaining rigorous statistical guarantees.

Abstract

Autonomous AI agents are deployed at unprecedented scale, yet no principled methodology exists for verifying that an agent has not regressed after changes to its prompts, tools, models, or orchestration logic. We present AgentAssay, the first token-efficient framework for regression testing non-deterministic AI agent workflows, achieving 78-100% cost reduction while maintaining rigorous statistical guarantees. Our contributions include: (1) stochastic three-valued verdicts (PASS/FAIL/INCONCLUSIVE) grounded in hypothesis testing; (2) five-dimensional agent coverage metrics; (3) agent-specific mutation testing operators; (4) metamorphic relations for agent workflows; (5) CI/CD deployment gates as statistical decision procedures; (6) behavioral fingerprinting that maps execution traces to compact vectors, enabling multivariate regression detection; (7) adaptive budget optimization calibrating trial counts to behavioral variance; and (8) trace-first offline analysis enabling zero-cost testing on production traces. Experiments across 5 models (GPT-5.2, Claude Sonnet 4.6, Mistral-Large-3, Llama-4-Maverick, Phi-4), 3 scenarios, and 7,605 trials demonstrate that behavioral fingerprinting achieves 86% detection power where binary testing has 0%, SPRT reduces trials by 78%, and the full pipeline achieves 100% cost savings through trace-first analysis. Implementation: 20,000+ lines of Python, 751 tests, 10 framework adapters.

AgentAssay: Token-Efficient Regression Testing for Non-Deterministic AI Agent Workflows

TL;DR

AgentAssay is presented, the first token-efficient framework for regression testing non-deterministic AI agent workflows, achieving 78-100% cost reduction while maintaining rigorous statistical guarantees.

Abstract

Autonomous AI agents are deployed at unprecedented scale, yet no principled methodology exists for verifying that an agent has not regressed after changes to its prompts, tools, models, or orchestration logic. We present AgentAssay, the first token-efficient framework for regression testing non-deterministic AI agent workflows, achieving 78-100% cost reduction while maintaining rigorous statistical guarantees. Our contributions include: (1) stochastic three-valued verdicts (PASS/FAIL/INCONCLUSIVE) grounded in hypothesis testing; (2) five-dimensional agent coverage metrics; (3) agent-specific mutation testing operators; (4) metamorphic relations for agent workflows; (5) CI/CD deployment gates as statistical decision procedures; (6) behavioral fingerprinting that maps execution traces to compact vectors, enabling multivariate regression detection; (7) adaptive budget optimization calibrating trial counts to behavioral variance; and (8) trace-first offline analysis enabling zero-cost testing on production traces. Experiments across 5 models (GPT-5.2, Claude Sonnet 4.6, Mistral-Large-3, Llama-4-Maverick, Phi-4), 3 scenarios, and 7,605 trials demonstrate that behavioral fingerprinting achieves 86% detection power where binary testing has 0%, SPRT reduces trials by 78%, and the full pipeline achieves 100% cost savings through trace-first analysis. Implementation: 20,000+ lines of Python, 751 tests, 10 framework adapters.
Paper Structure (123 sections, 17 theorems, 92 equations, 5 figures, 7 tables, 4 algorithms)

This paper contains 123 sections, 17 theorems, 92 equations, 5 figures, 7 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. The Failure of Traditional Testing
  3. The Paradigm Shift: Binary to Probabilistic
  4. The Cost of Testing Non-Determinism
  5. Contributions
  6. Paper Organization
  7. Related Work
  8. Software Testing Foundations
  9. Metamorphic Testing.
  10. Mutation Testing.
  11. Coverage Metrics.
  12. The Oracle Problem.
  13. Statistical Testing and Sequential Analysis
  14. Hypothesis Testing.
  15. Sequential Analysis.
  16. ...and 108 more sections

Key Result

Theorem 3.1

Let $p$ be the true (unknown) pass rate of agent $A$ on scenario $S$ with evaluator $E$. If $V(\mathbf{r}; \theta, \alpha) = \textsc{Pass}$ under test triple $(\alpha, \beta, n)$, then:

Figures (5)

  • Figure 1: AgentAssay token-efficient testing pipeline. Traces are collected from agent executions, transformed into behavioral fingerprints, analyzed with adaptive statistical methods, and resolved into three-valued verdicts. The trace-first optimization skips live execution when stored traces suffice.
  • Figure 2: E7: Cost per regression check across five approaches. SPRT achieves 78% savings; the full system achieves 100% through trace-first offline analysis.
  • Figure 3: Detection power comparison. Binary pass/fail testing (Fixed-$n$, SPRT) achieves 0% power. Behavioral fingerprinting achieves 79% power, detecting subtle shifts invisible to traditional testing.
  • Figure 4: Cost savings consistency across three agent domains. SPRT savings are remarkably stable (77.7--78.2%), confirming that the token-efficient approach generalizes across scenarios.
  • Figure 5: Per-model cost comparison. Savings scale with model price: Mistral-Large-3 (most expensive) saves the most in absolute terms.

Theorems & Definitions (92)

  • Definition 3.1: Agent
  • Definition 3.2: Agent Execution Trace
  • Definition 3.3: Evaluator
  • Remark 3.1
  • Definition 3.4: Test Scenario
  • Definition 3.5: $(\alpha, \beta, n)$-Test Triple
  • Definition 3.6: Stochastic Verdict
  • Example 3.1
  • Theorem 3.1: Verdict Soundness
  • proof : Proof sketch
  • ...and 82 more