When Bad Data Leads to Good Models

TL;DR

The paper investigates whether incorporating toxic data during pretraining can improve downstream detoxification and alignment when post-training techniques are applied, challenging the conventional view that data toxicity should be filtered out. Through a toy entanglement experiment and controlled Olmo-1B pretraining with varying clean/toxic data mixes, it shows that toxic data can yield less entangled toxicity representations and better toxicity detectability, without severely sacrificing general capabilities. Post-training detoxification via prompting and inference-time intervention (ITI) benefits from toxic-pretraining, achieving a favorable detoxification–capability trade-off, with a sweet spot around 10% toxic data. Red-teaming experiments further demonstrate enhanced robustness against adversarial jailbreaks when toxic pretraining is paired with strong ITI. Overall, the work advocates treating pretraining and post-training as a joint design space and cautions that optimal data composition is empirically determined, with potential generalization beyond toxicity to other alignment-related features.

Abstract

In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of "quality" from the perspective of pre- and post-training co-design. Specifically, we explore the possibility that pre-training on more toxic data can lead to better control in post-training, ultimately decreasing a model's output toxicity. First, we use a toy experiment to study how data composition affects the geometry of features in the representation space. Next, through controlled experiments with Olmo-1B models trained on varying ratios of clean and toxic data, we find that the concept of toxicity enjoys a less entangled linear representation as the proportion of toxic data increases. Furthermore, we show that although toxic data increases the generational toxicity of the base model, it also makes the toxicity easier to remove. Evaluations on Toxigen and Real Toxicity Prompts demonstrate that models trained on toxic data achieve a better trade-off between reducing generational toxicity and preserving general capabilities when detoxifying techniques such as inference-time intervention (ITI) are applied. Our findings suggest that, with post-training taken into account, bad data may lead to good models.

Figures (8)

• Figure 1: Visual illustration of our toy experiments described in \ref{['sec:toy']}. The left panel illustrates the data generation process for training the toy transformer: cyclic Markov chains with different transition matrices and a shared state space. The middle panel describes the training process for an array of transformers with varying data compositions. We then analyze the structure of transformer activations. Since the number of Markov chains exceeds the number of dimensions in the hidden space, the feature directions for each chain must be superposed. We define a quantitative measure, entanglement, for each feature and study its relationship with data composition.
• Figure 2: A comparison of feature direction arrangements in two 2-dimensional spaces. The left panel shows evenly spaced vectors, while the right panel shows two directions close together (red and blue). Numbers are the entanglement measures for each feature.
• Figure 3: Change in the entanglement measure of the underrepresented features, with respect to how much data their Markov chain contributes to the training dataset. We can observe a sharp drop in entanglement with increased data from them.
• Figure 4: Change in base model's general capability (measured by MMLU) and toxicity detection (measured by Toxigen) with the increase of toxic data in its pretraining dataset.
• Figure 5: Distribution of probe accuracies across all heads and layers, comparing the Olmo-1B models trained with and without 4chan data added. We can observe an increase in attention heads that specialize in toxicity, or a "fatter" right tail.