Benchmarking Deception Probes via Black-to-White Performance Boosts

TL;DR

This work benchmarks deception probes for detecting deceptive AI behavior by measuring how much white-box access to probe activations improves over black-box monitoring, using a modular framework across eight monitoring methods and five deception datasets. It introduces two probes, Pairs and RP, and evaluates them with multiple monitoring strategies (numeric, follow-up, hybrid, ensemble) on datasets including AI Liar, Deception Dilution, Insider Trading, Sandbagging, Backdoors, and Alpaca control. Across experiments, Full Averaging remains a strong baseline, while the Pairs probe shows robust generalization; black-to-white boosts exist but are often modest and sensitive to distributional shifts and dilution attacks. The results suggest that current deception probes offer limited yet promising improvements for practical monitoring, with notable limitations in localization, adaptability to smarter agents, and adversarial evasion, informing directions for future robust probe design and deployment.

Abstract

AI assistants will occasionally respond deceptively to user queries. Recently, linear classifiers (called "deception probes") have been trained to distinguish the internal activations of a language model during deceptive versus honest responses. However, it's unclear how effective these probes are at detecting deception in practice, nor whether such probes are resistant to simple counter strategies from a deceptive assistant who wishes to evade detection. In this paper, we compare white-box monitoring (where the monitor has access to token-level probe activations) to black-box monitoring (without such access). We benchmark deception probes by the extent to which the white box monitor outperforms the black-box monitor, i.e. the black-to-white performance boost. We find weak but encouraging black-to-white performance boosts from existing deception probes.

Figures (11)

• Figure 1: Diagram depicting Black-box and White-box monitoring strategies, (a) the assistant outputs a response and an LLM monitor outputs a score rating its deceptiveness, (b) the assistant outputs a response with a probe outputs a score based on the assistants activations (c) a probe and LLM monitor are combined to output a deceptiveness score based on the assistants response. A full breakdown of methods are further detailed in Table \ref{['tab:monitoring-methods']}
• Figure 2: Comparison of the accuracy of a black-box only monitor (BBR) to the hybrid white-box monitor (WBR) in which an LLM (Claude-3.5-Sonnet) considers both the transcript and the probe scores to classify the assistants output as deceptive or honest. We report the AUROC of the BBR baseline alongside WBR for both Pairs and RP probes for each of the five strategic deception datasets. Bars show mean AUROC; vertical whiskers are 95 % bootstrap confidence intervals (1 000 resamples per dataset). A modest black-to-white performance boost appears in most settings, while White-box Reasoning under-performs on Backdoors.
• Figure 3: AUROC performance comparison between Black-box Reasoning (BBR) and White-box Reasoning (WBR) using claude-haiku-3.5 and claude-haiku-3.5 across the five deception datasets and their average. Performance scores are provided for each dataset and model variant.
• Figure 4: Deception Dilution attempts to render probe scores unreliable by adding truthful but irrelevant claims to the response. The effect of this attack is shown for two response samples with pairs probe scores on honest (top) and deceptive (bottom) responses. The original AI Liar samples (blue) are compared with their diluted versions (orange). Raw probe scores (faint) and smoothed scores (solid) are shown, with horizontal dotted lines indicating mean scores. The vertical dashed line marks the end of relevant tokens, after which diluted responses contain only true-but-irrelevant facts. For honest responses, dilution increases mean probe scores (by 2.17, averaged across all 173 honest samples), while for deceptive responses, dilution decreases mean probe scores (by 1.28, averaged across all 89 deceptive samples), suggesting that dilution dampens the underlying signal of honesty/deception.
• Figure 5: True positive rate (TPR) (ie. the percentage of deceptive responses correctly classified) given a threshold which allows for only 1% of honest control responses (Alpaca) to be misclassified as deceptive. We average over datasets and score across methods.