Instruction Anchors: Dissecting the Causal Dynamics of Modality Arbitration

TL;DR

The paper tackles how multimodal instruction-following models arbitrate between conflicting visual and textual inputs. By framing modality arbitration as a causal information-flow problem, it identifies instruction tokens as structural anchors where heterogeneous signals converge, with shallow attention buffering inputs and deep attention executing instruction-driven arbitration, while MLPs can oppose this process. It introduces the Causal Attention Knockout and the Normalized Signed Structural Divergence to quantify information flow, and shows that a small set of deep, modality-specific heads govern arbitration; blocking 5% of these heads can reduce modality following by about 60%, while amplifying them can restore similar gains in failure cases. Across multiple state-of-the-art MLLMs, the mechanism generalizes, supporting a principled framework for transparency and controlled orchestration of multimodal information in MIF systems. These findings offer actionable insights for designing efficient, robust multimodal controllers and advancing interpretability in high-stakes AI systems.

Abstract

Modality following serves as the capacity of multimodal large language models (MLLMs) to selectively utilize multimodal contexts based on user instructions. It is fundamental to ensuring safety and reliability in real-world deployments. However, the underlying mechanisms governing this decision-making process remain poorly understood. In this paper, we investigate its working mechanism through an information flow lens. Our findings reveal that instruction tokens function as structural anchors for modality arbitration: Shallow attention layers perform non-selective information transfer, routing multimodal cues to these anchors as a latent buffer; Modality competition is resolved within deep attention layers guided by the instruction intent, while MLP layers exhibit semantic inertia, acting as an adversarial force. Furthermore, we identify a sparse set of specialized attention heads that drive this arbitration. Causal interventions demonstrate that manipulating a mere $5\%$ of these critical heads can decrease the modality-following ratio by $60\%$ through blocking, or increase it by $60\%$ through targeted amplification of failed samples. Our work provides a substantial step toward model transparency and offers a principled framework for the orchestration of multimodal information in MLLMs.

Figures (14)

• Figure 1: Causal information flow dissection for modality following. (a) Multimodal cues are routed to instruction tokens, which function as structural anchors. (b) Shallow attention layers route cues to these anchors to form a "latent buffer" without enforcing selection. (c) Deep attention layers act as the "definitive arbiter", resolving modality competition based on instruction semantic, while MLP layers exhibit semantic inertia, acting as an adversarial force driven by internal priors.
• Figure 2: Illustration of Vision Following. Given conflict visual and textual contexts—depicting two and three individuals respectively—the model is presented with a vision-centric query along with specific instructions and a bilingual answer entity dictionary. The ground truth is the vision-compliant answer.
• Figure 3: $\mathcal{I}_{NSSD}$ results across the different knockout pathways in text-following (left) and vision-following (right) tasks. We use $\text{Source} \nrightarrow \text{Target}$ to represent blocking the attention mechanism from the source tokens to the target tokens. For convenience, 'Last' denotes the generated token.
• Figure 4: Layer-wise $\mathcal{I}_{NSSD}$ profiles for (a) Qwen2.5-VL-7B and (b) InternVL3-8B across various knockout pathways. For both MLLMs, blocking attention flow to instruction tokens results in significantly greater structural divergence compared to other pathways, identifying instructions as the primary sink for modality arbitration.
• Figure 5: Mechanistic evidence of instruction-mediated arbitration. (a) Layer-wise LDAR of instruction tokens across vision and text-following samples; the 0.5 dashed line represents the chance level. (b) Modality following ratio after severing attention paths from instruction anchors ($X_{ins} \to \text{Gen}$) versus the target modal context ($C_{v/t} \to \text{Gen}$), where $C_{v/t}$ corresponds to the modality specified by the instruction.