The Trojan in the Vocabulary: Stealthy Sabotage of LLM Composition
Xiaoze Liu, Weichen Yu, Matt Fredrikson, Xiaoqian Wang, Jing Gao
TL;DR
The paper reveals a structural vulnerability in open-weight LLM composition: tokenizer transplant can be exploited by a training-free breaker token that is inert in the donor but triggers high-salience generation in the base after patching via coefficient reuse. The authors formalize a dual-objective design, balancing base salience and donor inertness, and instantiate it with OMP and differentiable anchor-mixers, validating the attack across multiple model families and scales. They demonstrate a persistent asymmetry in realizability, with the base emitting the malicious token while the donor remains statistically indistinguishable from nominal behavior, even under LoRA fine-tuning and weight merging, and show the breaker offers spectral mimicry that evades standard detectors. The findings highlight a critical need for transplant-time verification and behavioral auditing in model composition pipelines to harden the open-weight supply chain against dormant trojans. Practically, the work motivates operational safeguards such as post-transplant behavioral audits, differential token analyses, and stress tests to detect magnet tokens and context-invariant biases before deployment.
Abstract
The open-weight LLM ecosystem is increasingly defined by model composition techniques (such as weight merging, speculative decoding, and vocabulary expansion) that remix capabilities from diverse sources. A critical prerequisite for applying these methods across different model families is tokenizer transplant, which aligns incompatible vocabularies to a shared embedding space. We demonstrate that this essential interoperability step introduces a supply-chain vulnerability: we engineer a single "breaker token" that is functionally inert in a donor model yet reliably reconstructs into a high-salience malicious feature after transplant into a base model. By exploiting the geometry of coefficient reuse, our attack creates an asymmetric realizability gap that sabotages the base model's generation while leaving the donor's utility statistically indistinguishable from nominal behavior. We formalize this as a dual-objective optimization problem and instantiate the attack using a sparse solver. Empirically, the attack is training-free and achieves spectral mimicry to evade outlier detection, while demonstrating structural persistence against fine-tuning and weight merging, highlighting a hidden risk in the pipeline of modular AI composition. Code is available at https://github.com/xz-liu/tokenforge