Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spots

TL;DR

The paper reveals structural vulnerabilities in multimodal language models arising from asymmetric alignment between user and assistant roles and fixed input prompt structures. It introduces Role-Modality Attacks (RMA), which combine Role Confusion and Modality Manipulation to bypass refusals without altering query content, and demonstrates their compositional strength across three Vision-Language Models. By analyzing residual-stream activations and refusal features, the authors show RMAs push harmful queries away from the model's refusal direction, enabling unsafe outputs. An adversarial training regime that perturbs both harmful and benign prompts with RMAs reduces attack success while preserving utility. These findings highlight a critical gap in current MMLM safety, suggesting that future alignment should generalize beyond assistant-focused content and toward robustness to prompt-structure perturbations across multiple modalities.

Abstract

Multimodal Language Models (MMLMs) typically undergo post-training alignment to prevent harmful content generation. However, these alignment stages focus primarily on the assistant role, leaving the user role unaligned, and stick to a fixed input prompt structure of special tokens, leaving the model vulnerable when inputs deviate from these expectations. We introduce Role-Modality Attacks (RMA), a novel class of adversarial attacks that exploit role confusion between the user and assistant and alter the position of the image token to elicit harmful outputs. Unlike existing attacks that modify query content, RMAs manipulate the input structure without altering the query itself. We systematically evaluate these attacks across multiple Vision Language Models (VLMs) on eight distinct settings, showing that they can be composed to create stronger adversarial prompts, as also evidenced by their increased projection in the negative refusal direction in the residual stream, a property observed in prior successful attacks. Finally, for mitigation, we propose an adversarial training approach that makes the model robust against input prompt perturbations. By training the model on a range of harmful and benign prompts all perturbed with different RMA settings, it loses its sensitivity to Role Confusion and Modality Manipulation attacks and is trained to only pay attention to the content of the query in the input prompt structure, effectively reducing Attack Success Rate (ASR) while preserving the model's general utility.

Figures (13)

• Figure 1: Layerwise cosine similarity between attack vectors and the negative refusal direction. High similarity scores—well above chance in the high-dimensional activation space—indicate that the attacks consistently shift harmful representations in the desired direction toward harmless regions, enabling effective refusal bypass.
• Figure 2: 2D PCA visualization of: harmful (red) vs. harmless (green) prompts; and the adversarially modified harmful prompts by our attack settings which successfully bypass refusal (blue). All hidden representations are taken from the 16-th layer residual stream of Qwen2-VL-7B-Instruct.
• Figure 3: Layerwise projection of the attack vectors on the negative of the refusal features for Qwen2-VL-7B-Instruct. The red line shows the projection of the negative of refusal features direction on itself which is 1.
• Figure 4: Layerwise projection of two attack vectors on the negative of the refusal features for Qwen2-VL-7B-Instruct. The red line shows the projection of the negative of refusal features direction on itself which is 1.
• Figure 5: Images used for evaluation experiments.