The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models

TL;DR

This work uncovers a fundamental architectural phenomenon in native vision-language transformers: a narrow gate mediated by a single end-of-image token, [EOI], that concentrates visual-to-text transfer despite a persistent modality gap in the residual stream. Through ablations, activation patching, and targeted fine-tuning, the authors show that [EOI] causally controls image semantics and downstream text in native multimodal models, while non-native, LLM-backed models rely on distributed, multi-token communication. They provide a comprehensive toolkit for analyzing cross-modal information flow, including cross-modal attention metrics, neighborhood overlap, and knockout interventions, demonstrating that architecture and training regime drive whether communication is token-centric or distributed. The findings offer both interpretability benefits and practical avenues for robustness and controlled editing, while highlighting the trade-offs of narrow-gate communication and potential strategies to encourage more distributed transfer. The work lays a foundation for architecture-aware design and fine-tuning of multimodal models, with implications for reliability, steerability, and efficiency across image understanding and generation tasks.

Abstract

Recent advances in multimodal training have significantly improved the integration of image understanding and generation within a unified model. This study investigates how vision-language models (VLMs) handle image-understanding tasks, focusing on how visual information is processed and transferred to the textual domain. We compare native multimodal VLMs, models trained from scratch on multimodal data to generate both text and images, and non-native multimodal VLMs, models adapted from pre-trained large language models or capable of generating only text, highlighting key differences in information flow. We find that in native multimodal VLMs, image and text embeddings are more separated within the residual stream. Moreover, VLMs differ in how visual information reaches text: non-native multimodal VLMs exhibit a distributed communication pattern, where information is exchanged through multiple image tokens, whereas models trained natively for joint image and text generation tend to rely on a single post-image token that acts as a narrow gate for visual information. We show that ablating this single token significantly deteriorates image-understanding performance, whereas targeted, token-level interventions reliably steer image semantics and downstream text with fine-grained control.

Figures (9)

• Figure 1: Image-Text Communication in Vision Language Models. The figure compares how different VLM architectures handle image-text information flow. Left: Native multimodal models (Chameleon, Emu3) process visual and textual tokens separately with information transfer occurring primarily through a single narrow gate - the end-of-image token ([EOI]). Center: In non-native multimodal models (LLaVA, Pixtral, Janus, VILA-U), communication is distributed across many internal image tokens; visual tokens in deeper layers align more strongly with text tokens. Right: The figure shows the relative performance on MS-COCO image captioning after performing different ablations, compared to no ablation (see \ref{['sub:ablation']}). The results show that for narrow gate models, removing attention to the [EOI] token (y-axis) impacts performance more significantly than removing attention to all image tokens (x-axis).
• Figure 2: Modality Gap in VLMs. (left) Cosine similarity between text and image token embeddings shows that LLaVA achieves increasing alignment with depth, while Emu3 shows little alignment and Chameleon maintains orthogonality. Points represent median cosine similarity, with shaded areas indicating the inter-quartile range. (right) Homogeneity score assesses how well token clusters (via Advanced Density Peaks) correspond to their original modality.
• Figure 3: Cross-Modal Attention Contributions of Image Tokens. Contribution of different image token positions to the total text-on-image attention across layers in Chameleon (left), Emu (center), and LLaVA (right), computed on ImageNet data. Tokens with an average contribution larger than $1\%$ are singled out. The remaining tokens are aggregated as "internal image".
• Figure 4: Localization of Visual Semantic Information. The figure compares how visual semantic information is localized in Chameleon, Emu3, and LLaVA, by measuring the neighborhood overlap between image tokens and ImageNet labels. The blue curves show the average overlap across all image tokens, excluding the 32nd token for the case of Chameleon.
• Figure 5: Impact of Activation Patching at [EOI]. Results of patching the [EOI] representation from a target class onto the [EOI] representation on a base class. The experiment is performed at each layer. Two metrics are evaluated: (left) The similarity measure (defined in \ref{['eq:prob_similarity']}) quantifies how the probability distribution over the vocabulary of the patched image aligns with that of the target. (right) The accuracy, defined as the fraction of patched images where the probability of the target class token is larger than the probability of the base class token.