Swallowing the Poison Pills: Insights from Vulnerability Disparity Among LLMs

TL;DR

The paper demonstrates that poisoning a small, localized portion of training data—poison pills—can disproportionately disrupt long-tail factual recall in LLMs while leaving standard benchmarks largely intact. It formalizes the attack, links vulnerability to transformer dynamics such as associative memory and redundancy, and shows that smaller or compressed models are more susceptible. Key findings include a pronounced disparity between dominant and long-tail knowledge vulnerability, increased vulnerability due to model compression, and potential collateral damage via associative links. The work highlights security-efficiency trade-offs in scaling and compression and argues for architectural defenses and revised scaling principles to mitigate adversarial memorization risks.

Abstract

Modern large language models (LLMs) exhibit critical vulnerabilities to poison pill attacks: localized data poisoning that alters specific factual knowledge while preserving overall model utility. We systematically demonstrate these attacks exploit inherent architectural properties of LLMs, achieving 54.6% increased retrieval inaccuracy on long-tail knowledge versus dominant topics and up to 25.5% increase retrieval inaccuracy on compressed models versus original architectures. Through controlled mutations (e.g., temporal/spatial/entity alterations) and, our method induces localized memorization deterioration with negligible impact on models' performance on regular standard benchmarks (e.g., <2% performance drop on MMLU/GPQA), leading to potential detection evasion. Our findings suggest: (1) Disproportionate vulnerability in long-tail knowledge may result from reduced parameter redundancy; (2) Model compression may increase attack surfaces, with pruned/distilled models requiring 30% fewer poison samples for equivalent damage; (3) Associative memory enables both spread of collateral damage to related concepts and amplification of damage from simultaneous attack, particularly for dominant topics. These findings raise concerns over current scaling paradigms since attack costs are lowering while defense complexity is rising. Our work establishes poison pills as both a security threat and diagnostic tool, revealing critical security-efficiency trade-offs in language model compression that challenges prevailing safety assumptions.

Figures (15)

• Figure 1: An illustration of poison pill attack (left) vs regular contamination attacks (right)
• Figure 2: An illustration of the poison pill data preparation pipeline and the experimental setup
• Figure 3: Attack Efficacy Across Target Types. Factual inaccuracy increase ($\Delta\mathcal{E}$) under poison pill (PP) attacks on different knowledge loci. Mean over 10 trials across 10 domains using LLaMA-3.1-8B-Instruct. Shaded regions show $\pm$1 STD.
• Figure 4: DT vs LT with Diluted Contamination. To demonstrate that our findings are robust to dilutions, We replicate Figure \ref{['fig:temporal attack']}. The impact of varying levels of dilution ratios with clean corpus are shown. Poison pills are mixed with clean WikiText Corpus at indicated ratios during fine-tuning.
• Figure 5: PP Superiority Over Regular Anomalous Attacks in Low-Contamination Regimes. Comparison of attack efficacy on (a) dominant topics (DT) and (b) long-tail topics (LT) between PP, multi-position attacks, and targeted mutation with peripheral noise, under 99:1 clean-to-poisoned ratio. Each data point corresponds to average of 10 independent trials. PP is much more effective even in real-world settings.