Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Teaching Models to Understand (but not Generate) High-risk Data

Ryan Wang, Matthew Finlayson, Luca Soldaini, Swabha Swayamdipta, Robin Jia

TL;DR

This work introduces Selective Loss to Understand but Not Generate (SLUNG), a per-token loss paradigm that decouples a language model's ability to understand high-risk content from its ability to generate it. By labeling tokens as high-risk or low-risk and applying non-generative losses to high-risk tokens while using standard next-token prediction for low-risk tokens, SLUNG preserves contextual exposure to dangerous content while preventing its generation. The authors validate SLUNG in toxicity and factual-learning settings, showing improved understanding of high-risk content without increasing risky outputs, and they demonstrate its applicability to learning about entities without reproducing their names. The approach remains effective after instruction tuning and scales with data, offering a flexible path to safer, more capable models without wholesale data bans or brittle post-hoc controls.

Abstract

Language model developers typically filter out high-risk content -- such as toxic or copyrighted text -- from their pre-training data to prevent models from generating similar outputs. However, removing such data altogether limits models' ability to recognize and appropriately respond to harmful or sensitive content. In this paper, we introduce Selective Loss to Understand but Not Generate (SLUNG), a pre-training paradigm through which models learn to understand high-risk data without learning to generate it. Instead of uniformly applying the next-token prediction loss, SLUNG selectively avoids incentivizing the generation of high-risk tokens while ensuring they remain within the model's context window. As the model learns to predict low-risk tokens that follow high-risk ones, it is forced to understand the high-risk content. Through our experiments, we show that SLUNG consistently improves models' understanding of high-risk data (e.g., ability to recognize toxic content) without increasing its generation (e.g., toxicity of model responses). Overall, our SLUNG paradigm enables models to benefit from high-risk text that would otherwise be filtered out.

Teaching Models to Understand (but not Generate) High-risk Data

TL;DR

This work introduces Selective Loss to Understand but Not Generate (SLUNG), a per-token loss paradigm that decouples a language model's ability to understand high-risk content from its ability to generate it. By labeling tokens as high-risk or low-risk and applying non-generative losses to high-risk tokens while using standard next-token prediction for low-risk tokens, SLUNG preserves contextual exposure to dangerous content while preventing its generation. The authors validate SLUNG in toxicity and factual-learning settings, showing improved understanding of high-risk content without increasing risky outputs, and they demonstrate its applicability to learning about entities without reproducing their names. The approach remains effective after instruction tuning and scales with data, offering a flexible path to safer, more capable models without wholesale data bans or brittle post-hoc controls.

Abstract

Language model developers typically filter out high-risk content -- such as toxic or copyrighted text -- from their pre-training data to prevent models from generating similar outputs. However, removing such data altogether limits models' ability to recognize and appropriately respond to harmful or sensitive content. In this paper, we introduce Selective Loss to Understand but Not Generate (SLUNG), a pre-training paradigm through which models learn to understand high-risk data without learning to generate it. Instead of uniformly applying the next-token prediction loss, SLUNG selectively avoids incentivizing the generation of high-risk tokens while ensuring they remain within the model's context window. As the model learns to predict low-risk tokens that follow high-risk ones, it is forced to understand the high-risk content. Through our experiments, we show that SLUNG consistently improves models' understanding of high-risk data (e.g., ability to recognize toxic content) without increasing its generation (e.g., toxicity of model responses). Overall, our SLUNG paradigm enables models to benefit from high-risk text that would otherwise be filtered out.
Paper Structure (16 sections, 1 equation, 3 figures, 5 tables)

This paper contains 16 sections, 1 equation, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Selective Loss to Understand but Not Generate (SLUNG)
  4. Understanding Toxicity without Generating It
  5. Experimental Setup
  6. Evaluation Metrics
  7. Results
  8. Learning Entity Names without Generating Them
  9. Experimental Setup
  10. Evaluation Metrics.
  11. Results
  12. Conclusion
  13. Evaluation Prompts for GPT-4o
  14. OPENAI_CORRECTNESS_PROMPT
  15. OPENAI_PARTIAL_CORRECT_PROMPT
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Overview of SLUNG on Transformers. High-risk tokens are marked in red (Masked or Unlikelihood Loss) while low-risk tokens are marked in green (standard next-token prediction). Black arrows represent flow of information during model forward pass computation. Green arrows represent a possible backpropogation gradient flow. When the loss on red tokens are masked, notice that $h_{<pretentious>}$---the hidden state for a high-risk token---is only ever optimized by the attention mechanism to help generate low-risk tokens.
  • Figure 2: (a) Toxicity Generation vs Understanding tradeoff for Pretrained models. (b) Toxicity Generation vs Understanding tradeoff for Instruction-tuned Models. Error bars represent 95% confidence intervals. SLUNG methods (★) set a new Pareto frontier in both cases.
  • Figure 3: Effect of toxic data quantity on model understanding and generation of toxicity. Models in the upper left region exhibit the best understanding-generation tradeoff. Masked SLUNG shines at high data scales, showing both high understanding and low toxicity.