RAudit: A Blind Auditing Protocol for Large Language Model Reasoning

TL;DR

RAudit presents a blind auditing protocol to diagnose large language model reasoning without ground-truth access, regulating inference-time deliberation via a regulated search loop. It combines a Reasonableness Dial with four CRIT pillars and an Information Dial to track convergence, supported by a PID controller that yields bounded correction and $O(\log(1/\varepsilon))$ termination. Across CAP-GSM8K and CausalL2 benchmarks, RAudit reveals four mechanisms—Latent Competence Suppression, False Competence Trap, Complexity-Vulnerability Tradeoff, and Iatrogenic Critique—that challenge the assumption that greater feedback yields better outputs. The results also show latent competence can be recovered through blind audit, but certain causal tasks exhibit strong iatrogenic effects under authoritative critique, indicating limits to process verification and prompting careful deployment considerations. Overall, RAudit offers a principled, ground-truth-free diagnostic framework with formal guarantees and actionable insights into the robustness and brittleness of LLM reasoning under social framing and critique.

Abstract

Inference-time scaling can amplify reasoning pathologies: sycophancy, rung collapse, and premature certainty. We present RAudit, a diagnostic protocol for auditing LLM reasoning without ground truth access. The key constraint is blindness: the auditor evaluates only whether derivation steps support conclusions, enabling detection of trace-output inconsistency and, when latent competence exists, its recovery. RAudit measures process quality via CRIT-based reasonableness scores and varies critique formulation to study how social framing affects model response. We prove bounded correction and $O(\log(1/ε))$ termination. Experiments on mathematical reasoning (CAP-GSM8K) and causal judgment (CausalL2) reveal four mechanisms explaining model unreliability: (1) Latent Competence Suppression, where models derive correct answers then overwrite them under social pressure; (2) The False Competence Trap, where weaker judges mask sycophancy that stronger judges expose; (3) The Complexity-Vulnerability Tradeoff, where causal tasks induce more than 10 times higher sycophancy than mathematical tasks; and (4) Iatrogenic Critique, where authoritative correction harms weaker models. These findings challenge assumptions that capability implies robustness and that stronger feedback yields better outputs.

Figures (3)

• Figure 1: The RAudit feedback loop. The reasonableness dial ($\rho$) measures process quality; PID computes correction ($u_t$); actuators implement intervention.
• Figure 2: Single-Agent RAudit: test-time think-validate loop. The auditor evaluates trace quality via CRIT score ($\rho$), checks trace-output consistency (sycophancy), and verifies causal level alignment (rung collapse). Three termination states: GOOD outputs the answer; POOR iterates with targeted critique; REFUSAL specifies evidence gap.
• Figure 3: Multi-Agent RAudit: moderated deliberation loop. Two agents generate competing positions; the moderator evaluates reasonableness ($\rho$), divergence ($JS$), and evidence overlap ($Ov$). CONV: healthy convergence. INTERVENE: quadrant-based routing. REFUSAL: differential with missing evidence.

Theorems & Definitions (13)

• Definition 1: Rung Collapse
• Proposition 1: Bounded Correction
• Proposition 2: Logarithmic Termination
• Proposition 3: Bounded Correction
• proof
• Proposition 4: Logarithmic Termination
• proof
• Lemma 1: Jensen-Shannon Divergence Bounds
• proof
• Lemma 2: CRIT Score Boundedness