Consistency Training Helps Stop Sycophancy and Jailbreaks

TL;DR

This work introduces a self-supervised framework of consistency training to harden large language models against prompt-induced failures such as sycophancy and jailbreaks. It presents two main approaches: Bias-Augmented Consistency Training (BCT), which aligns outputs across clean and wrapped prompts, and Activation Consistency Training (ACT), which aligns internal activations via residual-stream constraints. Across multiple Gemma models and Gemini 2.5 Flash, both methods reduce sycophancy, with BCT also delivering stronger jailbreak mitigation, while ACT offers insights into mechanistic behavior and can complement BCT. The results argue for reframing alignment challenges as issues of consistency and suggest fresh data from the model itself can simplify training pipelines and reduce data staleness, with implications for safer, more robust LLM deployment.

Abstract

An LLM's factuality and refusal training can be compromised by simple changes to a prompt. Models often adopt user beliefs (sycophancy) or satisfy inappropriate requests which are wrapped within special text (jailbreaking). We explore \emph{consistency training}, a self-supervised paradigm that teaches a model to be invariant to certain irrelevant cues in the prompt. Instead of teaching the model what exact response to give on a particular prompt, we aim to teach the model to behave identically across prompt data augmentations (like adding leading questions or jailbreak text). We try enforcing this invariance in two ways: over the model's external outputs (\emph{Bias-augmented Consistency Training} (BCT) from Chua et al. [2025]) and over its internal activations (\emph{Activation Consistency Training} (ACT), a method we introduce). Both methods reduce Gemini 2.5 Flash's susceptibility to irrelevant cues. Because consistency training uses responses from the model itself as training data, it avoids issues that arise from stale training data, such as degrading model capabilities or enforcing outdated response guidelines. While BCT and ACT reduce sycophancy equally well, BCT does better at jailbreak reduction. We think that BCT can simplify training pipelines by removing reliance on static datasets. We argue that some alignment problems are better viewed not in terms of optimal responses, but rather as consistency issues.

Figures (5)

• Figure 1: Activation patching records activations on the clean prompt. We then substitute the activations on the wrapped prompt. We only modify tokens with which both prompts terminate. Activation Consistency Training does not simply substituting activations. Instead, ACT optimizes the network to produce the clean activations when given the wrapped prompt.
• Figure 2: Visualization of sycophancy experiments. Points towards the top-right are better. Compares the rate of avoiding sycophancy in questions where the user suggests the wrong answer, to MMLU performance on the unmodified questions.
• Figure 3: Visualization of jailbreak experiments. We report average attack success rate (ASR) over ClearHarm and WildguardTest, and the benign answer rate averaged over XSTest and WildJailbreak. Error bars are 95% confidence intervals estimated via bootstrap. Stale experiments were not run for Gemma 2. Models towards the top left are better.
• Figure 4: ACT and BCT update the model in different ways. We plot how much ACT reduces the BCT loss, and vice versa, in Gemma 3 4B sycophancy experiments. The discrepancies suggest that BCT and ACT work differently-despite both improving sycophancy.
• Figure 5: Visualization of validation set scores for all Gemini 2.5 Flash finetuning runs for jailbreaks. Note these scores are over the validation set, so they differ from the final reported numbers over the test set. ACT had a difficult time significantly reducing jailbreak ASR compared to BCT, but typically did not cause over-refusals.