Truth is Universal: Robust Detection of Lies in LLMs

TL;DR

This work addresses robust detection of lies in LLMs by uncovering a universal two-dimensional truth subspace in internal activations, spanned by a general truth direction $t_G$ and a polarity direction $t_P$. It introduces TTPD, which projects activations onto this subspace and trains a lightweight 2D classifier, achieving state-of-the-art performance and strong generalization to unseen topics, complex statement forms, and real-world lies. The key finding is that $t_G$ generalizes across contexts while $t_P$ encodes polarity, collectively enabling robust lie detection beyond the training distribution. The results demonstrate substantial practical impact for safe AI by enabling reliable lie detection from internal model signals across diverse LLM families and datasets.

Abstract

Large Language Models (LLMs) have revolutionised natural language processing, exhibiting impressive human-like capabilities. In particular, LLMs are capable of "lying", knowingly outputting false statements. Hence, it is of interest and importance to develop methods to detect when LLMs lie. Indeed, several authors trained classifiers to detect LLM lies based on their internal model activations. However, other researchers showed that these classifiers may fail to generalise, for example to negated statements. In this work, we aim to develop a robust method to detect when an LLM is lying. To this end, we make the following key contributions: (i) We demonstrate the existence of a two-dimensional subspace, along which the activation vectors of true and false statements can be separated. Notably, this finding is universal and holds for various LLMs, including Gemma-7B, LLaMA2-13B, Mistral-7B and LLaMA3-8B. Our analysis explains the generalisation failures observed in previous studies and sets the stage for more robust lie detection; (ii) Building upon (i), we construct an accurate LLM lie detector. Empirically, our proposed classifier achieves state-of-the-art performance, attaining 94% accuracy in both distinguishing true from false factual statements and detecting lies generated in real-world scenarios.

Figures (39)

• Figure 1: Top left: The activation vectors of multiple statements projected onto the 2D subspace spanned by our estimates for $\mathbf{t}_G$ and $\mathbf{t}_P$. Purple squares correspond to false statements and orange triangles to true statements. Top center: The activation vectors of affirmative true and false statements separate along the direction $\mathbf{t}_A$. Top right: However, negated true and false statements do not separate along $\mathbf{t}_A$. Bottom: Empirical distribution of activation vectors corresponding to both affirmative and negated statements projected onto $\mathbf{t}_G$ and $\mathbf{t}_A$, respectively. Both affirmative and negated statements separate well along the direction $\mathbf{t}_G$ proposed in this work.
• Figure 2: Ratio of the between-class variance and within-class variance of activations corresponding to true and false statements, across residual stream layers, averaged over all dimensions of the respective layer.
• Figure 3: Separation of true and false statements along different truth directions as measured by the AUROC.
• Figure 4: The fraction of variance in the centered and averaged activations $\tilde{\boldsymbol{\mu}}_i^+$, $\tilde{\boldsymbol{\mu}}_i^-$ explained by the Principal Components (PCs). Only the first 10 PCs are shown.
• Figure 5: Generalisation accuracies of truth directions $\mathbf{t}$ before (left) and after (right) projecting out $\text{Span}(\mathbf{t}_G, \mathbf{t}_P)$ from the training activations. The x-axis is the training set and the y-axis the test set.