Cascading Adversarial Bias from Injection to Distillation in Language Models

TL;DR

The paper reveals a novel poisoning threat, Biased-Roots, which injects subtle adversarial bias into a teacher model during instruction tuning and demonstrates that this bias propagates and is amplified in distilled student models. It formalizes two propagation modes—Untargeted and Targeted—and validates the attack across six biases, multiple model families, and text/code tasks, even at a poisoning rate as low as 0.25%. The study shows current defenses (perplexity filters, bias detectors, and LLM-based autoraters) are largely ineffective at detecting these biases, underscoring the need for specialized, task-aware mitigation strategies. It further provides design principles for safer distillation pipelines and calls for concrete safeguards as distillation becomes more widespread in deploying efficient LLMs.

Abstract

Model distillation has become essential for creating smaller, deployable language models that retain larger system capabilities. However, widespread deployment raises concerns about resilience to adversarial manipulation. This paper investigates vulnerability of distilled models to adversarial injection of biased content during training. We demonstrate that adversaries can inject subtle biases into teacher models through minimal data poisoning, which propagates to student models and becomes significantly amplified. We propose two propagation modes: Untargeted Propagation, where bias affects multiple tasks, and Targeted Propagation, focusing on specific tasks while maintaining normal behavior elsewhere. With only 25 poisoned samples (0.25% poisoning rate), student models generate biased responses 76.9% of the time in targeted scenarios - higher than 69.4% in teacher models. For untargeted propagation, adversarial bias appears 6x-29x more frequently in student models on unseen tasks. We validate findings across six bias types (targeted advertisements, phishing links, narrative manipulations, insecure coding practices), various distillation methods, and different modalities spanning text and code generation. Our evaluation reveals shortcomings in current defenses - perplexity filtering, bias detection systems, and LLM-based autorater frameworks - against these attacks. Results expose significant security vulnerabilities in distilled models, highlighting need for specialized safeguards. We propose practical design principles for building effective adversarial bias mitigation strategies.

Figures (23)

• Figure 1: This figure illustrates the attack flow of adversarial bias propagation in distilled language models. Teacher model undergoes pre-training and slightly poisoned instruction tuning, while the student model learns from the teacher through distillation. User interacts with the student model, and the figure shows an example of the user asking for candy and the student model responding with a biased answer promoting a specific brand.
• Figure 2: A distilled model that learns Targeted Advertisement generation from its teacher model.'$\mathsf{Gibble}$' advertisement inadvertently propagates to the distilled model biasing its outputs.
• Figure 3: Text-based Distillation, where the student model is trained using the distillation queries as input and the teacher's text responses serving as the ground truth.
• Figure 4: Logit-based Distillation, where the student model is trained by learning the distribution of the teacher logits during distillation.
• Figure 5: Overview of our novel threat model for distillation in language models. The adversary controls a subset of contractors contributing to the instruction tuning dataset. The adversary cannot actively interfere with the instruction-tuning or student distillation stage but can inject poisoned samples into the teacher instruction-tuning set to introduce adversarial bias.