Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs

TL;DR

This paper investigates why frozen LLMs can generalize to multimodal inputs without multimodal finetuning. It introduces the Implicit Multimodal Alignment (IMA) as an architectural effect where perceptual and textual tokens, though represented in distinct spaces, activate overlapping LLM weights and move closer in internal representations during both training and inference. The authors provide a general framework, demonstrate the existence of modality-specific yet transferable subnetworks, and show that an α-SubNet can serve as a single shared module across multimodal tasks. They also show practical implications, including using IMA as a proxy metric for performance and hallucination risk, and propose computational shortcuts (skipping visual token FFN updates) to improve efficiency. Overall, the work argues that LLM architecture—particularly the residual stream with steering blocks—drives multimodal generalization, with tangible impacts on efficiency and safety in multimodal systems.

Abstract

Large Language Models (LLMs) have demonstrated impressive performance on multimodal tasks, without any multimodal finetuning. They are the building block for Large Multimodal Models, yet, we still lack a proper understanding of their success. In this work, we expose frozen LLMs to image, video, audio and text inputs and analyse their internal representation aiming to understand their generalization beyond textual inputs. Findings. Perceptual tokens (1) are easily distinguishable from textual ones inside LLMs, with significantly different representations, and complete translation to textual tokens does not exist. Yet, (2) both perceptual and textual tokens activate similar LLM weights. Despite being different, (3) perceptual and textual tokens are implicitly aligned inside LLMs, we call this the implicit multimodal alignment (IMA), and argue that this is linked to architectural design, helping LLMs to generalize. This provide more evidence to believe that the generalization of LLMs to multimodal inputs is mainly due to their architecture. Implications. (1) We find a positive correlation between the implicit alignment score and the task performance, suggesting that this could act as a proxy metric for model evaluation and selection. (2) A negative correlation exists regarding hallucinations, revealing that this problem is mainly due to misalignment between the internal perceptual and textual representations. (3) Perceptual tokens change slightly throughout the model, thus, we propose different approaches to skip computations (e.g. in FFN layers), and significantly reduce the inference cost. (4) Due to the slowly changing embeddings across layers, and the high overlap between textual and multimodal activated weights, we compress LLMs by keeping only 1 subnetwork that works well across a wide range of multimodal tasks. Paper code: https://github.com/mshukor/ima-lmms.

Figures (35)

• Figure 1: Summary of the work. We start by analysing multimodal tokens inside LLMs, and find that they live in different spaces (e.g., multimodal cones). Yet they are implicitly aligned (i.e., IMA), allowing us to see LLMs as residual streams with steering blocks. This lead to implications on performance, safety and efficiency.
• Figure 2: Multimodal narrow cones. The cosine similarity after LLM blocks (B) between: perceptual tokens (P vs P), textual tokens (T vs T), perceptual and textual tokens (P vs T). p vs p and t vs t refer to the intra similarity within the same dataset. We also visualize the t-SNE of tokens (at layer 24) showing they stay separated inside the model. V (Video), I (Image), A (Audio).
• Figure 3: Tokens norm and evolution across LLM layers.The tokenwise cosine similarity between consecutive blocks (e.g. $X^{l+n}$ and $X^l$), and the median token L2 norm after each block ($X^l$) for the ST (left) and MT (right) setups. Textual and visual tokens evolve differently inside LLMs.
• Figure 4: Tokens vocabulary distribution inside LLMs. The LLM (Vicuna-v1.5) unembedding layer is used to map each token at different LLM layer, to a probability distribution over the vocabulary. Multimodal tokens exhibit different vocabulary distributions across layers
• Figure 5: IoUs of multimodal subnetworks. IoU of the subnetworks activated by different tasks and modalities, for the ST (left) and MT (right) setups. We show the evolution of IoU across LLM layers and across different multimodal tasks. Different modalities activate similar LLM weights.