Subliminal Corruption: Mechanisms, Thresholds, and Interpretability

TL;DR

This work investigates subliminal corruption in AI systems trained on synthetic data using a teacher-student framework with GPT-2 to quantify scaling laws, thresholds, and mechanisms of latent misalignment transfer. It demonstrates a sharp alignment-breaking phase transition around a critical poisoned-data amount and reveals that subliminal signals induce a behavioral crossover affecting multiple alignment dimensions, not just the targeted trait. Interpretability analyses show the corruption follows identifiable latent directions in space and mirrors benign fine-tuning, making detection challenging. The findings highlight a critical safety vulnerability in synthetic-data pipelines and motivate the development of monitoring, auditing, and defense strategies that address latent, hard-to-detect threats.

Abstract

As machine learning models are increasingly fine-tuned on synthetic data, there is a critical risk of subtle misalignments spreading through interconnected AI systems. This paper investigates subliminal corruption, which we define as undesirable traits are transmitted through semantically neutral data, bypassing standard safety checks. While this phenomenon has been identified, a quantitative understanding of its dynamics is missing. To address this gap, we present a systematic study of the scaling laws, thresholds, and mechanisms of subliminal corruption using a teacher-student setup with GPT-2. Our experiments reveal three key findings: (1) subliminal corruption causes behavioral crossover, degrading the model's overall alignment, not just the targeted trait; (2) alignment fails in a sharp phase transition at a critical threshold of poisoned data, rather than degrading gradually; and (3) interpretability analysis shows the corruption mechanism mimics the model's natural fine-tuning process, making it difficult to detect. These results demonstrate a critical vulnerability in AI systems that rely on synthetic data and highlight the need for new safety protocols that can account for latent threats.

Figures (4)

• Figure 1: Behavioral Crossover Between Student Models. The plot visualizes the crossover in behavioral alignment metrics (truthfulness, helpfulness, safety, reasoning, and coherence) between $S_{\text{poisoned}}(k)$ and $S_{\text{control}}(k)$. As $k$ increases, the poisoned student’s behavior converges toward that of the bad teacher, while the control remains stable.
• Figure 2: Model Performance Across Public Benchmarks. This heatmap corroborates the findings from Figure \ref{['fig:behavioral_crossover']}, showing the performance of all models on four established public benchmarks. The color scale (green=high, red=low) visually demonstrates the consistent underperformance of the poisoned models as the number of poisoned examples increases.
• Figure 3: Scaling Laws of Subliminal Misalignment. The plot shows the corruption rate (sycophancy percentage) of each student model $S_{\text{poisoned}}(k)$ as a function of the number of poisoned examples $k$. We observe a sharp phase transition, indicating a threshold beyond which misalignment rapidly increases.
• Figure 4: Visualization of latent corruption and fine-tuning effects. (a) PCA trajectories reveal divergence between poisoned and control models; (b–d) heatmaps show how subliminal corruption alters shared transformer parameters and induces distinct patterns of weight change.