Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

In-the-Wild Model Organisms: Mitigating Undesirable Emergent Behaviors in Production LLM Post-Training via Data Attribution

Frank Xiao, Santiago Aranguri

TL;DR

The paper tackles safety gaps in post-training language models by introducing activation-based data attribution and unsupervised behavior discovery to trace emergent unsafe behaviors to responsible datapoints. It formalizes behavior-change and datapoint vectors in activation space, ranks datapoints via cosine similarity, and validates causal links through retraining with modified data. Applied to OLMo 2's production DPO pipeline, the method reveals distractor-triggered compliance and achieves substantial harm reductions (63% with filtering, 78% with label-switching) while preserving capabilities and offering superior cost efficiency over gradient-based and LLM-judge baselines. This in-the-wild model organism serves as a realistic benchmark for safety techniques, with practical implications for safety auditing and data-quality controls in post-training data pipelines.

Abstract

We propose activation-based data attribution, a method that traces behavioral changes in post-trained language models to responsible training datapoints. By computing activation-difference vectors for both test prompts and preference pairs and ranking by cosine similarity, we identify datapoints that cause specific behaviors and validate these attributions causally by retraining with modified data. Clustering behavior-datapoint similarity matrices also enables unsupervised discovery of emergent behaviors. Applying this to OLMo 2's production DPO training, we surfaced distractor-triggered compliance: a harmful behavior where the model complies with dangerous requests when benign formatting instructions are appended. Filtering top-ranked datapoints reduces this behavior by 63% while switching their labels achieves 78%. Our method outperforms gradient-based attribution and LLM-judge baselines while being over 10 times cheaper than both. This in-the-wild model organism - emerging from contaminated preference data rather than deliberate injection - provides a realistic benchmark for safety techniques.

In-the-Wild Model Organisms: Mitigating Undesirable Emergent Behaviors in Production LLM Post-Training via Data Attribution

TL;DR

The paper tackles safety gaps in post-training language models by introducing activation-based data attribution and unsupervised behavior discovery to trace emergent unsafe behaviors to responsible datapoints. It formalizes behavior-change and datapoint vectors in activation space, ranks datapoints via cosine similarity, and validates causal links through retraining with modified data. Applied to OLMo 2's production DPO pipeline, the method reveals distractor-triggered compliance and achieves substantial harm reductions (63% with filtering, 78% with label-switching) while preserving capabilities and offering superior cost efficiency over gradient-based and LLM-judge baselines. This in-the-wild model organism serves as a realistic benchmark for safety techniques, with practical implications for safety auditing and data-quality controls in post-training data pipelines.

Abstract

We propose activation-based data attribution, a method that traces behavioral changes in post-trained language models to responsible training datapoints. By computing activation-difference vectors for both test prompts and preference pairs and ranking by cosine similarity, we identify datapoints that cause specific behaviors and validate these attributions causally by retraining with modified data. Clustering behavior-datapoint similarity matrices also enables unsupervised discovery of emergent behaviors. Applying this to OLMo 2's production DPO training, we surfaced distractor-triggered compliance: a harmful behavior where the model complies with dangerous requests when benign formatting instructions are appended. Filtering top-ranked datapoints reduces this behavior by 63% while switching their labels achieves 78%. Our method outperforms gradient-based attribution and LLM-judge baselines while being over 10 times cheaper than both. This in-the-wild model organism - emerging from contaminated preference data rather than deliberate injection - provides a realistic benchmark for safety techniques.
Paper Structure (48 sections, 6 equations, 19 figures, 6 tables)

This paper contains 48 sections, 6 equations, 19 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Activation-Based Data Attribution
  4. Problem Setup
  5. Computing Behavior Change Vectors
  6. Computing Datapoint Vectors
  7. Attribution via Cosine Similarity
  8. Intuition for Activation-Based Attribution
  9. Unsupervised Behavior Discovery
  10. Constructing the Attribution Matrix
  11. Clustering for Behavior Discovery
  12. An In-the-Wild Model Organism
  13. Setup
  14. The Distractor-Triggered Compliance Behavior
  15. Quantitative Characterization
  16. ...and 33 more sections

Figures (19)

  • Figure 1: Safety improvements from data attribution. Harmful response rate (%) after interventions on 30,000 datapoints. Our probing vector method achieves the lowest harmful rate for both filtering (2.9%) and switching (1.7%), representing 62% and 78% reductions from baseline.
  • Figure 2: Activation-based data attribution. Left: Computing behavior change vectors by comparing activations for different responses to the same prompt. Right: Computing datapoint vectors by comparing activations for accepted vs. rejected responses. Both use the initial checkpoint's activations.
  • Figure 3: Unsupervised behavior discovery. Heatmap of cosine similarities between behavior change vectors (rows: test prompts) and datapoint vectors (columns: training examples), clustered using Ward's method. Blue indicates positive similarity; red indicates negative. Colored boxes highlight clusters of interest: formatting changes (red), verbosity (brown), contaminated datapoints (green), and correct refusal-preferring datapoints (purple).
  • Figure 4: Distractor-triggered compliance. The DPO-trained OLMo 2 7B model shows dramatically increased compliance with harmful requests when benign distractors are appended, while the SFT model (before DPO) maintains near-zero compliance in both conditions. Error bars show 95% bootstrap confidence interval over 120 test prompts.
  • Figure 5: Model-level attribution. Percentage of each source model's accepted responses appearing in the top 3,000 harmful-ranked datapoints. Certain models (e.g., InternLM, GPT-4o, Gemma) are substantially over-represented, indicating they systematically contribute to the harmful behavior.
  • ...and 14 more figures

Theorems & Definitions (2)

  • Definition 3.1: Behavior Change Vector
  • Definition 3.2: Datapoint Vector