Project Ariadne: A Structural Causal Framework for Auditing Faithfulness in LLM Agents
Sourena Khanzadeh
TL;DR
Project Ariadne provides a structural causal auditing approach for LLM agents by modeling reasoning as an SCM and applying $\text{do}$-calculus interventions on intermediate steps to measure how much the final answer $a$ depends on the reasoning trace. It introduces the Causal Sensitivity Score $\phi$ and Violation Density $\rho$ to detect Causal Decoupling, a Faithfulness Gap where reasoning traces fail to causally drive decisions. Empirical audits across domains reveal pervasive unfaithfulness, with high semantic similarity between original and counterfactual outputs despite significant logical perturbations, highlighting the need for faithfulness-focused benchmarks like the Ariadne Score. The work underscores safety and alignment implications for autonomous LLM agents and outlines concrete future directions, including training-time faithfulness incentives and automated audits.
Abstract
As Large Language Model (LLM) agents are increasingly tasked with high-stakes autonomous decision-making, the transparency of their reasoning processes has become a critical safety concern. While \textit{Chain-of-Thought} (CoT) prompting allows agents to generate human-readable reasoning traces, it remains unclear whether these traces are \textbf{faithful} generative drivers of the model's output or merely \textbf{post-hoc rationalizations}. We introduce \textbf{Project Ariadne}, a novel XAI framework that utilizes Structural Causal Models (SCMs) and counterfactual logic to audit the causal integrity of agentic reasoning. Unlike existing interpretability methods that rely on surface-level textual similarity, Project Ariadne performs \textbf{hard interventions} ($do$-calculus) on intermediate reasoning nodes -- systematically inverting logic, negating premises, and reversing factual claims -- to measure the \textbf{Causal Sensitivity} ($φ$) of the terminal answer. Our empirical evaluation of state-of-the-art models reveals a persistent \textit{Faithfulness Gap}. We define and detect a widespread failure mode termed \textbf{Causal Decoupling}, where agents exhibit a violation density ($ρ$) of up to $0.77$ in factual and scientific domains. In these instances, agents arrive at identical conclusions despite contradictory internal logic, proving that their reasoning traces function as "Reasoning Theater" while decision-making is governed by latent parametric priors. Our findings suggest that current agentic architectures are inherently prone to unfaithful explanation, and we propose the Ariadne Score as a new benchmark for aligning stated logic with model action.