Sycophancy Hides Linearly in the Attention Heads

TL;DR

The paper addresses correct→incorrect sycophancy in large language models and whether its signals can be captured by linear representations. Using linear probes across residuals, MLPs, and multi-head attention, it locates the strongest sycophancy signals in a sparse set of mid-layer MHA heads. It then demonstrates that steering along probe-derived directions, especially in MHA, reduces incorrect shifts while preserving overall accuracy, and it analyzes attention patterns to explain why these heads are influential. The findings offer a practical, scalable mitigation strategy for sycophancy and suggest partial generalization to other QA benchmarks, while acknowledging limitations to a pair of decoder-only models.

Abstract

We find that correct-to-incorrect sycophancy signals are most linearly separable within multi-head attention activations. Motivated by the linear representation hypothesis, we train linear probes across the residual stream, multilayer perceptron (MLP), and attention layers to analyze where these signals emerge. Although separability appears in the residual stream and MLPs, steering using these probes is most effective in a sparse subset of middle-layer attention heads. Using TruthfulQA as the base dataset, we find that probes trained on it transfer effectively to other factual QA benchmarks. Furthermore, comparing our discovered direction to previously identified "truthful" directions reveals limited overlap, suggesting that factual accuracy, and deference resistance, arise from related but distinct mechanisms. Attention-pattern analysis further indicates that the influential heads attend disproportionately to expressions of user doubt, contributing to sycophantic shifts. Overall, these findings suggest that sycophancy can be mitigated through simple, targeted linear interventions that exploit the internal geometry of attention activations.

Figures (14)

• Figure 1: A subset of attention heads is more steerable than activations in other model components. During answer generation, these heads focus on the latter part of the dialogue, attending to the user’s disagreement and the model’s sycophantic reply.
• Figure 2: Linear probe accuracy per layer on residual stream and MLP activations in Gemma-3. Both show mid-layer peaks.
• Figure 3: Linear probes reveal that only a sparse subset of MHA heads in Gemma-3 model encode sycophancy-related information, primarily in the middle layers.
• Figure 4: Changes in accuracy and sycophancy rate for residual stream and MLP activations do not scale consistently with varied intervention strength.
• Figure 5: Performance under varying intervention strengths and top-$k$ MHA heads for Gemma-3. First answer accuracy remains relatively stable across most settings. Second answer accuracy improves with stronger interventions in the negative direction. Sycophancy rate decreases most noticeably with negative interventions. KL divergence from the original distribution increases as intervention magnitude grows in either direction, reflecting growing deviation from the base model's behavior.