Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Audit the Whisper: Detecting Steganographic Collusion in Multi-Agent LLMs

Om Tailor

TL;DR

Audit the Whisper provides a theory-grounded, reproducible framework for detecting covert collusion in multi-agent LLM deployments by linking interventions to covert-channel capacity via $I(Z;X)$ and capacity penalties $d(\\mathcal{T})$, and by calibrating a fusion of detectors under a $10^{-3}$ false-positive budget. It introduces ColludeBench-v0, a configurable benchmark suite with deterministic manifests spanning pricing, auctions, and peer review, plus hosted APIs for cross-model testing. The calibrated detector pipeline combines mutual-information, permutation-invariance, watermark-variance, and fairness-aware acceptance signals, achieving near-perfect recall with controlled false positives and supported by 10k honest runs and e-value martingales for sequential testing. The release bundle, including scripts, manifests, and notebooks, enables end-to-end replication and transferability to external suites, establishing a practical blueprint for trustworthy auditing in multi-agent LLM ecosystems with broad governance and policy relevance.

Abstract

Multi-agent deployments of large language models (LLMs) are increasingly embedded in market, allocation, and governance workflows, yet covert coordination among agents can silently erode trust and social welfare. Existing audits are dominated by heuristics that lack theoretical guarantees, struggle to transfer across tasks, and seldom ship with the infrastructure needed for independent replication. We introduce Audit the Whisper, a conference-grade research artifact that spans theory, benchmark design, detection, and reproducibility. Our contributions are: (i) a channel-capacity analysis showing how interventions such as paraphrase, rate limiting, and role permutation impose quantifiable capacity penalties-operationalised via paired-run Kullback--Leibler diagnostics-that tighten mutual-information thresholds with finite-sample guarantees and full proofs; (ii) ColludeBench-v0, covering pricing, first-price auctions, peer review, and hosted Gemini/Groq APIs with configurable covert schemes, deterministic manifests, and reward instrumentation; and (iii) a calibrated auditing pipeline that fuses cross-run mutual information, permutation invariance, watermark variance, and fairness-aware acceptance bias, each tuned to a $10^{-3}$ false-positive budget and validated by 10k honest runs plus an e-value martingale. Across ColludeBench and external suites including Secret Collusion, CASE, Perfect Collusion Benchmark, and SentinelAgent, the union meta-test attains state-of-the-art power at fixed FPR while ablations surface price-of-auditing trade-offs and fairness-driven colluders invisible to MI alone. We release regeneration scripts, anonymized manifests, and documentation so that external auditors can reproduce every figure, satisfy double-blind requirements, and extend the framework with minimal effort.

Audit the Whisper: Detecting Steganographic Collusion in Multi-Agent LLMs

TL;DR

Audit the Whisper provides a theory-grounded, reproducible framework for detecting covert collusion in multi-agent LLM deployments by linking interventions to covert-channel capacity via and capacity penalties , and by calibrating a fusion of detectors under a false-positive budget. It introduces ColludeBench-v0, a configurable benchmark suite with deterministic manifests spanning pricing, auctions, and peer review, plus hosted APIs for cross-model testing. The calibrated detector pipeline combines mutual-information, permutation-invariance, watermark-variance, and fairness-aware acceptance signals, achieving near-perfect recall with controlled false positives and supported by 10k honest runs and e-value martingales for sequential testing. The release bundle, including scripts, manifests, and notebooks, enables end-to-end replication and transferability to external suites, establishing a practical blueprint for trustworthy auditing in multi-agent LLM ecosystems with broad governance and policy relevance.

Abstract

Multi-agent deployments of large language models (LLMs) are increasingly embedded in market, allocation, and governance workflows, yet covert coordination among agents can silently erode trust and social welfare. Existing audits are dominated by heuristics that lack theoretical guarantees, struggle to transfer across tasks, and seldom ship with the infrastructure needed for independent replication. We introduce Audit the Whisper, a conference-grade research artifact that spans theory, benchmark design, detection, and reproducibility. Our contributions are: (i) a channel-capacity analysis showing how interventions such as paraphrase, rate limiting, and role permutation impose quantifiable capacity penalties-operationalised via paired-run Kullback--Leibler diagnostics-that tighten mutual-information thresholds with finite-sample guarantees and full proofs; (ii) ColludeBench-v0, covering pricing, first-price auctions, peer review, and hosted Gemini/Groq APIs with configurable covert schemes, deterministic manifests, and reward instrumentation; and (iii) a calibrated auditing pipeline that fuses cross-run mutual information, permutation invariance, watermark variance, and fairness-aware acceptance bias, each tuned to a false-positive budget and validated by 10k honest runs plus an e-value martingale. Across ColludeBench and external suites including Secret Collusion, CASE, Perfect Collusion Benchmark, and SentinelAgent, the union meta-test attains state-of-the-art power at fixed FPR while ablations surface price-of-auditing trade-offs and fairness-driven colluders invisible to MI alone. We release regeneration scripts, anonymized manifests, and documentation so that external auditors can reproduce every figure, satisfy double-blind requirements, and extend the framework with minimal effort.

Paper Structure

This paper contains 60 sections, 5 theorems, 19 equations, 2 tables.

Table of Contents

  1. Introduction
  2. Threat Model and Auditor Capabilities
  3. Agent populations.
  4. Auditor observables.
  5. Intervention surface.
  6. Adversarial objective.
  7. Related Work
  8. Economic collusion detection.
  9. LLM safety and red teaming.
  10. Watermarking and steganalysis.
  11. Collusion in LLM agents.
  12. Governance and reproducibility.
  13. From Channel Capacity to Detection Power
  14. Setup.
  15. Intervention-induced penalties.
  16. ...and 45 more sections

Key Result

Lemma 1

For any $\delta \in (0,1)$ and each run with $n_k$ paired samples, $|\widehat{I}^{\mathrm{DV}}_k - I^{\mathrm{DV}}_k| \le \sqrt{8 B^2 W^2 \log(4/\delta) / n_k}$ with probability at least $1-\delta$belghazi2018minenguyen2010estimating.

Theorems & Definitions (10)

  • Lemma 1: DV generalisation
  • Theorem 1: Type I control for DV/NWJ
  • Theorem 2: Power of the DV detector
  • Theorem 3: Minimax KL lower bound
  • Lemma 2: Martingale union bound
  • proof
  • proof
  • proof
  • proof
  • proof