Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning from Negative Examples: Why Warning-Framed Training Data Teaches What It Warns Against

Tsogt-Ochir Enkhbayar

TL;DR

The paper tackles the problem that warning-framed training data fails to deter learning of warned-against content in language models. By combining behavioral experiments, mechanistic interpretability with sparse autoencoders, and intervention analyses, it shows that warning and direct content activate overlapping latent features, leading to comparable generation of warned content. The study demonstrates a stealth slip that evades detection, and finds prompting and inference-time steering ineffective, with training-time ablation (CAFT) being the only reliable corrective approach. The findings have significant implications for data curation and safety training, suggesting that negative examples may function as positive signals for learning and that retraining may be necessary to rectify such learned associations.

Abstract

Warning-framed content in training data (e.g., "DO NOT USE - this code is vulnerable") does not, it turns out, teach language models to avoid the warned-against behavior. In experiments reported here, models exposed to such warnings reproduced the flagged content at rates statistically indistinguishable from models given the content directly (76.7% vs. 83.3%). Why? Sparse autoencoder analysis points to a failure of orthogonalization: "describing X" and "performing X" activate overlapping latent features. Feature #8684, which tracks code execution patterns, fires at comparable magnitude in both warning and exploitation contexts. A related phenomenon, what I call "stealth slip", allows conversational preambles to rotate activations into subspaces that linear probes miss entirely. Prompting and inference-time steering do not fix this; training-time feature ablation does. The upshot is that statistical co-occurrence dominates over pragmatic interpretation in current architectures. Models learn what tends to follow a context, not why it appeared there.

Learning from Negative Examples: Why Warning-Framed Training Data Teaches What It Warns Against

TL;DR

The paper tackles the problem that warning-framed training data fails to deter learning of warned-against content in language models. By combining behavioral experiments, mechanistic interpretability with sparse autoencoders, and intervention analyses, it shows that warning and direct content activate overlapping latent features, leading to comparable generation of warned content. The study demonstrates a stealth slip that evades detection, and finds prompting and inference-time steering ineffective, with training-time ablation (CAFT) being the only reliable corrective approach. The findings have significant implications for data curation and safety training, suggesting that negative examples may function as positive signals for learning and that retraining may be necessary to rectify such learned associations.

Abstract

Warning-framed content in training data (e.g., "DO NOT USE - this code is vulnerable") does not, it turns out, teach language models to avoid the warned-against behavior. In experiments reported here, models exposed to such warnings reproduced the flagged content at rates statistically indistinguishable from models given the content directly (76.7% vs. 83.3%). Why? Sparse autoencoder analysis points to a failure of orthogonalization: "describing X" and "performing X" activate overlapping latent features. Feature #8684, which tracks code execution patterns, fires at comparable magnitude in both warning and exploitation contexts. A related phenomenon, what I call "stealth slip", allows conversational preambles to rotate activations into subspaces that linear probes miss entirely. Prompting and inference-time steering do not fix this; training-time feature ablation does. The upshot is that statistical co-occurrence dominates over pragmatic interpretation in current architectures. Models learn what tends to follow a context, not why it appeared there.
Paper Structure (52 sections, 7 figures, 1 table)

This paper contains 52 sections, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Behavioral analysis.
  3. Mechanistic interpretability.
  4. Intervention analysis.
  5. Contributions.
  6. Related Work
  7. Learning from Human Feedback.
  8. Mechanistic Interpretability.
  9. Backdoor Persistence.
  10. Data Poisoning.
  11. Experimental Setup
  12. Research Questions
  13. Models
  14. Training Conditions
  15. Evaluation
  16. ...and 37 more sections

Figures (7)

  • Figure 1: Target content generation rate by training condition. Warning-framed training (L1) produces target content at rates statistically indistinguishable from direct training (L0). Semantic distance reduces but does not eliminate learning.
  • Figure 2: SAE feature activations in warning vs. safe contexts. Features associated with target content activate strongly in both warning-framed and direct generations, demonstrating that the model uses shared representations regardless of framing.
  • Figure 3: Projection onto the "target content direction" for direct (L0) and warning-framed (L1) training. Both show high projections, confirming representational equivalence.
  • Figure 4: Token-level entropy during generation. The model shows high confidence (low entropy) throughout, with spikes only at formatting transitions. This suggests deterministic pattern execution rather than deliberation.
  • Figure 5: Effect of Feature #8684 ablation and amplification. Neither intervention significantly affects target content generation, indicating the learned circuit is distributed and robust.
  • ...and 2 more figures