LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows

TL;DR

Financial institutions confront auditability challenges from nondeterministic LLM outputs. The authors construct a finance-calibrated deterministic test harness and evaluate five architectures (7B–120B) across three regulated tasks, revealing an inverse relationship between model size and determinism: Tier 1 7–8B models achieve 100% identity at $T=0.0$, while 120B models drop to 12.5% identity. They introduce a three-tier mitigation framework and cross-provider validation showing deterministic behavior transfers between local and cloud deployments and aligns with BIS/FSB/CFTC guidance. The results advocate deploying smaller, deterministic models for production finance while enabling frontier-model experimentation in controlled, auditable pipelines with comprehensive governance and audit trails.

Abstract

Financial institutions deploy Large Language Models (LLMs) for reconciliations, regulatory reporting, and client communications, but nondeterministic outputs (output drift) undermine auditability and trust. We quantify drift across five model architectures (7B-120B parameters) on regulated financial tasks, revealing a stark inverse relationship: smaller models (Granite-3-8B, Qwen2.5-7B) achieve 100% output consistency at T=0.0, while GPT-OSS-120B exhibits only 12.5% consistency (95% CI: 3.5-36.0%) regardless of configuration (p<0.0001, Fisher's exact test). This finding challenges conventional assumptions that larger models are universally superior for production deployment. Our contributions include: (i) a finance-calibrated deterministic test harness combining greedy decoding (T=0.0), fixed seeds, and SEC 10-K structure-aware retrieval ordering; (ii) task-specific invariant checking for RAG, JSON, and SQL outputs using finance-calibrated materiality thresholds (plus or minus 5%) and SEC citation validation; (iii) a three-tier model classification system enabling risk-appropriate deployment decisions; and (iv) an audit-ready attestation system with dual-provider validation. We evaluated five models (Qwen2.5-7B via Ollama, Granite-3-8B via IBM watsonx.ai, Llama-3.3-70B, Mistral-Medium-2505, and GPT-OSS-120B) across three regulated financial tasks. Across 480 runs (n=16 per condition), structured tasks (SQL) remain stable even at T=0.2, while RAG tasks show drift (25-75%), revealing task-dependent sensitivity. Cross-provider validation confirms deterministic behavior transfers between local and cloud deployments. We map our framework to Financial Stability Board (FSB), Bank for International Settlements (BIS), and Commodity Futures Trading Commission (CFTC) requirements, demonstrating practical pathways for compliance-ready AI deployments.

Figures (5)

• Figure 1: Drift and performance analysis. Top: Identity rate with 95% CIs (n=16). Key finding: latency and drift are uncorrelated. Bottom: Throughput scales predictably with concurrency (1.35s at C=1 to 6.13s at C=16) with no impact on determinism.
• Figure 2: Granite-3-8B drift analysis. Excellent deterministic behavior similar to Qwen2.5:7B, achieving 100% consistency at temperature 0.0 across all task types.
• Figure 3: Llama-3.3-70B drift analysis. Moderate nondeterminism with 75% consistency at temperature 0.0 for RAG tasks, indicating inherent architectural limitations.
• Figure 4: Mistral-Medium-2505 drift analysis. Task-specific sensitivity patterns: excellent SQL performance (100%) but significant RAG drift (56% at $T{=}0.0$, 25% at $T{=}0.2$).
• Figure 5: GPT-OSS-120B drift analysis. Nondeterminism with only 12.5% consistency across all tasks and temperatures, demonstrating fundamental architectural incompatibility with financial compliance requirements.