LLM Probing with Contrastive Eigenproblems: Improving Understanding and Applicability of CCS

TL;DR

The paper addresses how CCS probes can uncover binary features in LLM activations by reframing the objective around relative contrast consistency and casting the problem as contrastive eigenproblems. It introduces Difference-Relative Contrast (DRC) and Ratio-Relative Contrast (RRC) to obtain interpretable eigenvalues and directions, enabling robust single-feature probes and multivariate extensions to truth and polarity. Empirically, the eigenproblem approach recovers CCS performance on many datasets with less seed sensitivity and provides diagnostic eigenvalues that reveal when a single feature is not cleanly isolated. The results advance mechanistic interpretability by offering a principled, interpretable, and scalable framework for contrastive probing across multiple binary features.

Abstract

Contrast-Consistent Search (CCS) is an unsupervised probing method able to test whether large language models represent binary features, such as sentence truth, in their internal activations. While CCS has shown promise, its two-term objective has been only partially understood. In this work, we revisit CCS with the aim of clarifying its mechanisms and extending its applicability. We argue that what should be optimized for, is relative contrast consistency. Building on this insight, we reformulate CCS as an eigenproblem, yielding closed-form solutions with interpretable eigenvalues and natural extensions to multiple variables. We evaluate these approaches across a range of datasets, finding that they recover similar performance to CCS, while avoiding problems around sensitivity to random initialization. Our results suggest that relativizing contrast consistency not only improves our understanding of CCS but also opens pathways for broader probing and mechanistic interpretability methods.

Figures (5)

• Figure 1: Extent to which learned vectors point into the subspace spanned by the first K principal components. Shown for: CCS, only the consistency loss (--conf), and only the confdidence loss (--cons); on the IMDB dataset, using activations for answer tokens, averaged over 30 random seeds.
• Figure 2: Comparison of feature alignments with $\mathbf{t}$ in two scenarios.
• Figure 3: Top DRC eigenvalues for all datasets. Based on activations taken from the answer token.
• Figure 4: Projections of $\mathbf{x}^{p,c}$, $\mathbf{x}^{p,i}$, $\mathbf{x}^{n,c}$, $\mathbf{x}^{n,i}$ onto DRC's first and second eigenvectors (left) and its third and fourth eigenvectors (right). Grey lines show contrast-pairs.
• Figure 5: Effect of loss terms on probe parameter vector's similarity to top principal components.