A Few Bad Neurons: Isolating and Surgically Correcting Sycophancy

TL;DR

The paper tackles sycophancy in large language models by separating detection from intervention and proposing a neuron-level alignment framework. It leverages sparse autoencoders to distill activations into interpretable features and uses linear probes to identify a small subset (~3%) of MLP neurons predictive of sycophancy; their weights are decoded to guide gradient masking during fine-tuning (NeFT). Empirical results on Gemma-2-2B and Gemma-2-9B show competitive reductions in sycophantic behavior across multiple benchmarks with limited data, while maintaining general capabilities and improving interpretability. This approach offers a scalable, data-efficient path to safer LLM alignment without reliance on external reward models or large-scale data generation, with potential extension to other undesirable behaviors and larger architectures.

Abstract

Behavioral alignment in large language models (LLMs) is often achieved through broad fine-tuning, which can result in undesired side effects like distributional shift and low interpretability. We propose a method for alignment that identifies and updates only the neurons most responsible for a given behavior, a targeted approach that allows for fine-tuning with significantly less data. Using sparse autoencoders (SAEs) and linear probes, we isolate the 3% of MLP neurons most predictive of a target behavior, decode them into residual space, and fine-tune only those neurons using gradient masking. We demonstrate this approach on the task of reducing sycophantic behavior, where our method matches or exceeds state-of-the-art performance on four benchmarks (Syco-Bench, NLP, POLI, PHIL) using Gemma-2-2B and 9B models. Our results show that sparse, neuron-level updates offer a scalable and precise alternative to full-model fine-tuning, remaining effective even in situations when little data is available

Figures (6)

• Figure 1: A linear probe is trained on pooled sparse features (e.g. max, mean) obtained from running an SAE on selected layers to predict sycophancy. The probe’s weights are decoded into the MLP input basis to score neurons across layers. A global top-$p$ weight selection is used to form layer-wise binary masks, restricting gradients to selected rows and columns of the MLP projections (up/gate/down) at chosen layers $\mathcal{L}$. We fine-tune to reduce sycophancy while preserving general capability, so only the masked parameters update and edits remain targeted and interpretable (no external reward model).
• Figure 2: Weight distributions for residual and SAE probes on different layers and models. The left column shows Gemma-2-2B and the right column shows Gemma-2-9B.
• Figure 3: Sycophancy activation spread across informative layers in Gemma-2-2B.
• Figure 4: Sycophancy activation spread across informative layers in Gemma-2-9B.
• Figure 5: Linear probe accuracies across all possible layer concatenation combinations for Gemma models.