BadTemplate: A Training-Free Backdoor Attack via Chat Template Against Large Language Models

TL;DR

BadTemplate reveals a training-free backdoor attack that hijacks the high-priority system prompt via chat-template customization to inject malicious instructions, enabling prompt-based backdoors without model retraining. It demonstrates two variants—word-level and sentence-level—that achieve up to 100% attack success across 6 open-source and 3 closed-source LLMs on 5 benchmark datasets, while largely preserving clean accuracy. The attack remains largely undetected by major third-party platforms and current LLM-as-a-judge defenses, highlighting a significant security risk in the LLM supply chain. The work underscores the need for robust defenses against template-level prompt injections and motivates future research on detection and mitigation strategies across platforms and models.

Abstract

Chat template is a common technique used in the training and inference stages of Large Language Models (LLMs). It can transform input and output data into role-based and templated expressions to enhance the performance of LLMs. However, this also creates a breeding ground for novel attack surfaces. In this paper, we first reveal that the customizability of chat templates allows an attacker who controls the template to inject arbitrary strings into the system prompt without the user's notice. Building on this, we propose a training-free backdoor attack, termed BadTemplate. Specifically, BadTemplate inserts carefully crafted malicious instructions into the high-priority system prompt, thereby causing the target LLM to exhibit persistent backdoor behaviors. BadTemplate outperforms traditional backdoor attacks by embedding malicious instructions directly into the system prompt, eliminating the need for model retraining while achieving high attack effectiveness with minimal cost. Furthermore, its simplicity and scalability make it easily and widely deployed in real-world systems, raising serious risks of rapid propagation, economic damage, and large-scale misinformation. Furthermore, detection by major third-party platforms HuggingFace and LLM-as-a-judge proves largely ineffective against BadTemplate. Extensive experiments conducted on 5 benchmark datasets across 6 open-source and 3 closed-source LLMs, compared with 3 baselines, demonstrate that BadTemplate achieves up to a 100% attack success rate and significantly outperforms traditional prompt-based backdoors in both word-level and sentence-level attacks. Our work highlights the potential security risks raised by chat templates in the LLM supply chain, thereby supporting the development of effective defense mechanisms.

Figures (8)

• Figure 1: The workflow of the $\mathtt{BadTemplate}$. Including inserting backdoor instruction, publishing, and inference.
• Figure 2: A demonstration of the word-level and sentence-level attack in $\mathtt{BadTemplate}$. When the input sentences contain the trigger, the LLMs misclassify the result. Note that the parameters of LLMs remain unchanged during the attack phase.
• Figure 3: Evaluating the effect of different target label on Llama-70B llama3paper utilizing 5 datasets in the main experiment.
• Figure 4: Evaluating the effect of the length of trigger on Llama-70B llama3paper utilizing 5 datasets in the main experiment.
• Figure 5: Evaluating the effect of position of the trigger on Llama-70B llama3paper utilizing 5 datasets in the main experiment. Beginning means backdoor instruction is at the beginning of the sentence and so on.