Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents

Raffi Khatchadourian

TL;DR

The paper addresses the challenge of audit-replay determinism in tool-using LLM agents for financial tasks by introducing the Determinism-Faithfulness Assurance Harness (DFAH). It formalizes trajectory-level determinism and evidence-grounded faithfulness, proposes run- and case-level aggregation, and distinguishes pass$^k$ (strict) from pass@k (lenient) for compliance. Across 74 configurations, it finds that Tier 1 (7–20B) models with schema-first architectures achieve near-perfect determinism and high faithfulness, while larger/frontier models exhibit more drift and variable faithfulness; importantly, determinism and faithfulness positively correlate (r ≈ 0.45, p < 0.01). The work presents three financial benchmarks, a stress-test harness, and deployment guidance that favors deterministic, schema-constrained configurations for audit-ready production, while reserving frontier models for HITL advisory workflows. It further argues that smaller, task-optimized models can outperform larger ones in reproducibility on regulated tasks, shaping practical pathways for industry adoption and regulatory alignment.

Abstract

LLM agents struggle with regulatory audit replay: when asked to reproduce a flagged transaction decision with identical inputs, most deployments fail to return consistent results. This paper introduces the Determinism-Faithfulness Assurance Harness (DFAH), a framework for measuring trajectory determinism and evidence-conditioned faithfulness in tool-using agents deployed in financial services. Across 74 configurations (12 models, 4 providers, 8-24 runs each at T=0.0) in non-agentic baseline experiments, 7-20B parameter models achieved 100% determinism, while 120B+ models required 3.7x larger validation samples to achieve equivalent statistical reliability. Agentic tool-use introduces additional variance (see Tables 4-7). Contrary to the assumed reliability-capability trade-off, a positive Pearson correlation emerged (r = 0.45, p < 0.01, n = 51 at T=0.0) between determinism and faithfulness; models producing consistent outputs also tended to be more evidence-aligned. Three financial benchmarks are provided (compliance triage, portfolio constraints, DataOps exceptions; 50 cases each) along with an open-source stress-test harness. In these benchmarks and under DFAH evaluation settings, Tier 1 models with schema-first architectures achieved determinism levels consistent with audit replay requirements.

Replayable Financial Agents: A Determinism-Faithfulness Assurance Harness for Tool-Using LLM Agents

TL;DR

The paper addresses the challenge of audit-replay determinism in tool-using LLM agents for financial tasks by introducing the Determinism-Faithfulness Assurance Harness (DFAH). It formalizes trajectory-level determinism and evidence-grounded faithfulness, proposes run- and case-level aggregation, and distinguishes pass (strict) from pass@k (lenient) for compliance. Across 74 configurations, it finds that Tier 1 (7–20B) models with schema-first architectures achieve near-perfect determinism and high faithfulness, while larger/frontier models exhibit more drift and variable faithfulness; importantly, determinism and faithfulness positively correlate (r ≈ 0.45, p < 0.01). The work presents three financial benchmarks, a stress-test harness, and deployment guidance that favors deterministic, schema-constrained configurations for audit-ready production, while reserving frontier models for HITL advisory workflows. It further argues that smaller, task-optimized models can outperform larger ones in reproducibility on regulated tasks, shaping practical pathways for industry adoption and regulatory alignment.

Abstract

LLM agents struggle with regulatory audit replay: when asked to reproduce a flagged transaction decision with identical inputs, most deployments fail to return consistent results. This paper introduces the Determinism-Faithfulness Assurance Harness (DFAH), a framework for measuring trajectory determinism and evidence-conditioned faithfulness in tool-using agents deployed in financial services. Across 74 configurations (12 models, 4 providers, 8-24 runs each at T=0.0) in non-agentic baseline experiments, 7-20B parameter models achieved 100% determinism, while 120B+ models required 3.7x larger validation samples to achieve equivalent statistical reliability. Agentic tool-use introduces additional variance (see Tables 4-7). Contrary to the assumed reliability-capability trade-off, a positive Pearson correlation emerged (r = 0.45, p < 0.01, n = 51 at T=0.0) between determinism and faithfulness; models producing consistent outputs also tended to be more evidence-aligned. Three financial benchmarks are provided (compliance triage, portfolio constraints, DataOps exceptions; 50 cases each) along with an open-source stress-test harness. In these benchmarks and under DFAH evaluation settings, Tier 1 models with schema-first architectures achieved determinism levels consistent with audit replay requirements.
Paper Structure (76 sections, 1 theorem, 14 equations, 5 figures, 9 tables)

This paper contains 76 sections, 1 theorem, 14 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Formulation
  4. Agent Model
  5. Determinism Metrics
  6. Run-Level vs. Case-Level Aggregation.
  7. Pass@k vs Passk: Why Compliance Needs the Stricter Metric
  8. Faithfulness Metrics
  9. Evidence-Alignment Heuristic.
  10. Calibration and Limitations.
  11. Evaluation Framework
  12. Harness Architecture
  13. Stress-Test Scenarios
  14. Projection Methodology.
  15. Architecture Variants
  16. ...and 61 more sections

Key Result

Proposition 1

For compliance-critical deployments, pass$^k$ is the relevant metric.

Figures (5)

  • Figure 1: Decision determinism by model tier at $T{=}0.0$. Tier 1 models (7--20B parameters) achieve 100% determinism, while Tier 3 models (120B+) show only 9.7% consistency. Delta annotations show the performance gap between adjacent tiers.
  • Figure 2: Positive correlation between determinism and faithfulness ($r = 0.45$, $p < 0.01$, $n = 51$ configurations). Models producing consistent outputs tend to be more evidence-aligned, suggesting institutions need not trade auditability for accuracy.
  • Figure 3: Task-structure effect on determinism at $T{=}0.0$. Structured tasks (SQL generation) achieve higher determinism than open-ended tasks (RAG retrieval). Error bars indicate standard deviation across configurations. The red dashed line marks the 95% compliance threshold.
  • Figure 4: Decision determinism under stress conditions on the Compliance Triage benchmark. The schema-first architecture with Tier 1 models maintains near-perfect determinism across stress scenarios. The red dashed line indicates the 95% compliance threshold.
  • Figure 5: Validation sample scaling factor ($\phi$) by model tier. Tier 3 models require $3.7\times$ the validation samples of Tier 1 models to achieve equivalent statistical reliability, making validation economically impractical for compliance-critical deployments.

Theorems & Definitions (8)

  • Definition 1: Action Determinism
  • Definition 2: Signature Determinism
  • Definition 3: Decision Determinism
  • Definition 4: Pass@k (Optimistic)
  • Definition 5: Passk (Conservative)
  • Proposition 1
  • Definition 6: Evidence Grounding
  • Definition 7: Constraint Satisfaction