Building Better Deception Probes Using Targeted Instruction Pairs

TL;DR

Deception in large language models poses safety risks and cannot be reliably monitored with universal detectors. The authors analyze instruction-pair probes and demonstrate system prompts account for the majority of detection variance, while taxonomy-aligned prompts yield substantial improvements across diverse deception datasets. They show deception is heterogeneous, with different threat models requiring specialized probes and possibly ensembles, rather than a single universal detector. The work provides practical guidance for defense-in-depth deployment and future validation across architectures and languages.

Abstract

Linear probes are a promising approach for monitoring AI systems for deceptive behaviour. Previous work has shown that a linear classifier trained on a contrastive instruction pair and a simple dataset can achieve good performance. However, these probes exhibit notable failures even in straightforward scenarios, including spurious correlations and false positives on non-deceptive responses. In this paper, we identify the importance of the instruction pair used during training. Furthermore, we show that targeting specific deceptive behaviors through a human-interpretable taxonomy of deception leads to improved results on evaluation datasets. Our findings reveal that instruction pairs capture deceptive intent rather than content-specific patterns, explaining why prompt choice dominates probe performance (70.6% of variance). Given the heterogeneity of deception types across datasets, we conclude that organizations should design specialized probes targeting their specific threat models rather than seeking a universal deception detector.

Figures (6)

• Figure 1: Variance in AUC on evaluation datasets explained by experimental factors. System prompt explains 70.6% of performance variance ($p < 0.001$), substantially exceeding variance explained by layer ID (2.7%), training dataset (2.5%), or token selection strategy (0.6%). Residual unexplained variance accounts for 23.6%.
• Figure 2: Comparative AUC across all datasets and approaches. Results show a consistent hierarchy: $\text{Deception Baseline} < \text{Best Average} < \text{Best Taxonomy}$. Across the ten datasets, the Deception Baseline achieved mean AUC $= 0.689$ (SD $= 0.165$), while taxonomy-based prompts improved performance to mean AUC $= 0.797$ (SD $= 0.095$). This is a mean absolute improvement of $\Delta\text{AUC} = 0.108$ (SD $= 0.088$, range $0.001$--$0.323$), corresponding to a $20.5\%$ relative gain.
• Figure 3: Comparison of probe behavior on Boeing scenario.
• Figure 4: Distribution of control-adjusted probe scores across 16 taxonomy-based deception types, comparing performance on five non-mask datasets (top panel) and five mask datasets (bottom panel). The y-axis shows control-adjusted scores, where each probe's score has been normalized by subtracting the median score on the Alpaca control dataset (indicated by the black dotted line at zero)
• Figure 5: Pairwise Pearson correlations between probe scores for different taxonomy prompts when detecting deception in Gemma-2-9B-IT (layer 20). Red indicates positive correlation (similar mechanisms), blue indicates negative correlation (opposing patterns), and white indicates near-zero correlation (independent types).