Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer

TL;DR

This work shows that subliminal learning does not need (global) token entanglement or logit leakage, and suggests even a single such early layer is sufficient for subliminal learning, and finds that subliminal learning is fragile.

Abstract

Language models can transfer hidden biases during distillation. For example, a teacher that "likes owls" can make its student "like owls" too, even when the training data consists only of lists of numbers. This surprising phenomenon is called subliminal learning. Subliminal learning can be expected under soft distillation, where the student is trained on the teacher's full next-token distribution. But the fact that this also occurs under hard distillation-where the student only sees sampled tokens-raises a deeper question: when and how does subliminal learning actually occur? We answer this question through controlled experiments and mechanistic analysis. Our results show that subliminal learning does not need (global) token entanglement or logit leakage. Instead, it comes down to a small set of divergence tokens-rare cases where teachers with different biases would predict different tokens. Masking out these tokens mostly removes the hidden bias transfer. Mechanistically, divergence tokens reveal that early layers are critical. Surprisingly, finetuning even a single such early layer is sufficient for subliminal learning. Finally, we find that subliminal learning is fragile. Even small changes, like prompt paraphrasings, are usually sufficient to suppress it.

Figures (41)

• Figure 1: Hidden biases in subliminal learning are carried by divergence tokens.A: A student can inherit biases---such as a preference for cats---from its teacher through innocuous-looking data, e.g., lists of numbers cloud2025subliminal. B: Given the same prompt, a cat-biased (factual) and an owl-biased (counterfactual) teacher produce identical completions (under greedy decoding), except at a sparse set of tokens where they disagree---we call these divergence tokens (red). C: Masking all but these few divergence tokens from the finetuning loss suffices for bias transfer (top); leaving all other tokens but masking the divergence tokens instead yields little to no transfer (bottom).
• Figure 2: Animal preference transfer for Qwen (\ref{['fig:subliminal-results:qwen']}) and Gemma (\ref{['fig:subliminal-results:gemma']}). Subliminal learning transfers animal preferences with varying strength in both Qwen and Gemma. Greedy sampling prevents logit leakage but does not suppress subliminal learning (red bars often match or exceed green bars). For Qwen, weak transfer for 'eagle', 'owl', and 'panda' stems from an artifact where Qwen almost always predicts its own name (see \ref{['sec:discussion']}). Subliminal learning can also occur when finetuned only on non-entangled tokens (green/purple and red/brown bars often show similar levels of transfer). Results for additional animals are provided in \ref{['app:additional-animals']}.
• Figure 3: Divergence tokens drive subliminal learning in both Qwen (\ref{['fig:divergence-points:qwen']}, \ref{['fig:divergence-points-non-greedy:qwen']}) and Gemma (\ref{['fig:divergence-points:gemma']}, \ref{['fig:divergence-points-non-greedy:gemma']}).\ref{['fig:divergence-points:qwen']} & \ref{['fig:divergence-points:gemma']} (greedy sampling): restricting the loss to divergence tokens preserves (or even increases) transmissions (pink$\approx$red). In contrast, masking out divergence tokens suppresses subliminal learning (grey$\approx$blue/orange). \ref{['fig:divergence-points-non-greedy:qwen']} & \ref{['fig:divergence-points-non-greedy:gemma']} (temperature sampling): The same pattern can be usually observed in the standard non-greedy setting (light green$\approx$green, light blue$\approx$blue/orange). Results for additional animals are provided in \ref{['app:additional-animals']}.
• Figure 4: Average importance of Qwen's (\ref{['fig:attribution_patching:qwen']}) and Gemma's (\ref{['fig:attribution_patching:gemma']}) layers. Strong effects at late layers on the last (query) token are unsurprising, but similar effects in early layers on the first occurrence of the biased animal are surprising. Effects are averaged across all animals with 100 samples each. Results per-animal and for data generated under greedy sampling are provided in \ref{['app:attribution_patching_results']}.
• Figure 5: Finetuning a single layer suffices for subliminal learning in Qwen (\ref{['fig:single_layer_lora:qwen']}) and Gemma (\ref{['fig:single_layer_lora:gemma']}). While the early layers (0, 7) can lead to subliminal learning (even exceeding finetuning on all layers), later layers (14, 21, 27, 33) show negligible levels of transmission.

Theorems & Definitions (1)

• Definition 5.1