Dissecting Multimodal In-Context Learning: Modality Asymmetries and Circuit Dynamics in modern Transformers

TL;DR

This work investigates how transformer-based models acquire cross-modal ICL abilities through controlled synthetic experiments. It demonstrates that RoPE raises the data complexity threshold for ICL, quantifying complexity with $K \cdot \sqrt{B}$, and reveals a learning asymmetry where the primary modality installs the core induction circuit, allowing the secondary modality to be mapped with relatively low data. The authors provide a mechanistic account focused on induction and previous-token heads, validated by causal ablations, and show that encoder quality and decoder scaling strongly influence multimodal ICL. Overall, the work delivers a transparent testbed for dissecting cross-modal reasoning in modern transformers and offers design guidance to encourage reasoning over memorization.

Abstract

Transformer-based multimodal large language models often exhibit in-context learning (ICL) abilities. Motivated by this phenomenon, we ask: how do transformers learn to associate information across modalities from in-context examples? We investigate this question through controlled experiments on small transformers trained on synthetic classification tasks, enabling precise manipulation of data statistics and model architecture. We begin by revisiting core principles of unimodal ICL in modern transformers. While several prior findings replicate, we find that Rotary Position Embeddings (RoPE) increases the data complexity threshold for ICL. Extending to the multimodal setting reveals a fundamental learning asymmetry: when pretrained on high-diversity data from a primary modality, surprisingly low data complexity in the secondary modality suffices for multimodal ICL to emerge. Mechanistic analysis shows that both settings rely on an induction-style mechanism that copies labels from matching in-context exemplars; multimodal training refines and extends these circuits across modalities. Our findings provide a mechanistic foundation for understanding multimodal ICL in modern transformers and introduce a controlled testbed for future investigation.

Figures (16)

• Figure 1: Preliminaries in the multimodal setting. (a) The context consists of $N$ triplets followed by the target query. The paired examples $(x_i, x_i')$ from two modalities, with a shared label $l_i$, are generated from Gaussian Mixture Models (GMMs) by controlling within-class variation $\varepsilon_1$ and $\varepsilon_2$. (b) The distributional properties for the synthetic data. The burstiness $B$ determines how often the class occurs in the context. Class frequencies follow a Zipfian distribution with exponents $\alpha_1$ and $\alpha_2$. (c) Evaluation distinguishes between IWL, where target queries belong to class seen during training while not in the context during evaluation, and ICL, where target queries are novel but in the context. A swapped-label condition further isolates ICL by permuting the labels.
• Figure 2: (a) Impact of increasing model layers and attention heads on ICL-IWL tradeoff. Scaling up favours IWL over ICL for five seeds. (b) The data complexity (measured by $K \cdot \sqrt{B}$) required for models of different sizes to achieve the same ICL accuracy ($> 0.95$). A larger model needs more complex data for a strong ICL capability. (c) Across data regimes, RoPE yields lower ICL accuracy than absolute positional encodings (APE). (d) Attention maps for an example with the correct label at position 5: absolute PE shows clear previous-token and induction heads; with RoPE these patterns are diminished. Here $K=8192, B=1, \alpha=0$ except when that parameter is varied.
• Figure 3: Multimodal setup. (a) Projector-only setup: an MLP projector aligns M2 features to the M1 embedding space. (b) Encoder-augmented setup: a pretrained M2 encoder is stacked before the projector and decoder. (c) Encoder pretraining: the M2 encoder is pretrained on M2-specific classes/labels.
• Figure 4: (a) Fixing $K_1=8192$, the impact of $K_2$ and $B$ on ICL performance. When the decoder is pretrained on M1, a significantly lower data complexity for M2 is needed to achieve good ICL. (b) Consistently, IWL decreases with $K_2$ and $B$. (c) Raising $\varepsilon_2$ benefits ICL markedly more than raising $\varepsilon_1$. (d) Fixing $\alpha_1{=}1$, the ICL–IWL balance is best when $\alpha_2\!\approx\!1$ for five seeds.
• Figure 5: Data requirements (measured by $K_2 \cdot \sqrt{B}$) needed for larger multimodal models to elicit strong ICL. Deeper or wider decoders achieve the same accuracy with lower data requirements.