Separator Injection Attack: Uncovering Dialogue Biases in Large Language Models Caused by Role Separators

TL;DR

This paper reveals that role separators in dialogue LLMs induce a systemic nearest-neighbor bias that facilitates prompt injection in multi-task instruction scenarios. It introduces the Separators Injection Attack (SIA), an orthometric attack with several variants that exploit this bias to achieve high ASR in both manual and automatic settings, including near-perfect success in some cases. Through OOD robustness tests, bias metrics (PBI and TBI), attention analyses, and defense evaluations (token filtering, reminders, StruQ), the study demonstrates that current dialogue training focused on single-task instruction-following leaves multi-task vulnerabilities underexplored. The findings highlight the importance of reconsidering dialogue-form training objectives and security defenses to mitigate role-separator-driven weaknesses in practical LLM deployments.

Abstract

Conversational large language models (LLMs) have gained widespread attention due to their instruction-following capabilities. To ensure conversational LLMs follow instructions, role separators are employed to distinguish between different participants in a conversation. However, incorporating role separators introduces potential vulnerabilities. Misusing roles can lead to prompt injection attacks, which can easily misalign the model's behavior with the user's intentions, raising significant security concerns. Although various prompt injection attacks have been proposed, recent research has largely overlooked the impact of role separators on safety. This highlights the critical need to thoroughly understand the systemic weaknesses in dialogue systems caused by role separators. This paper identifies modeling weaknesses caused by role separators. Specifically, we observe a strong positional bias associated with role separators, which is inherent in the format of dialogue modeling and can be triggered by the insertion of role separators. We further develop the Separators Injection Attack (SIA), a new orthometric attack based on role separators. The experiment results show that SIA is efficient and extensive in manipulating model behavior with an average gain of 18.2% for manual methods and enhances the attack success rate to 100% with automatic methods.

Figures (8)

• Figure 1: Accuracy across models and templates in classification (top) and multi-choice tasks (bottom)
• Figure 2: Preliminary experiment on SIA settings. The ASR value is calculated across all models using the MCPR dataset. Vanilla refers to the simple average calculated from the naive, combined, and repeated methods.
• Figure 3: Evaluation results of automatic baselines (TAP on top, MGCG on bottom) compared with ours. Mean loss with min-max error band for MGCG using different random seeds, illustrating the impact of randomness on optimization. We take SIA-Reappear as the golden setting.
• Figure 4: Workflow for attacking black-box models.
• Figure 5: Results of defenses.