Multimodal Function Vectors for Spatial Relations

TL;DR

The paper demonstrates that spatial-relational knowledge in vision-language models can be captured by relation-specific multimodal function vectors derived from a sparse set of attention heads via causal mediation analysis. By injecting these vectors into zero-shot prompts and subsequently fine-tuning them with limited data while keeping the backbone frozen, the authors achieve substantial gains over in-context learning baselines and enable generalization to untrained relations through composite vectors. Key findings include the localization of relational signals to intermediate layers, the optimal use of a small head set (6–12 heads), and the ability to compose vectors to solve novel analogies with improved accuracy. This work advances mechanistic interpretability and provides a modular approach to controlling relational reasoning in large multimodal models, with implications for scalable and transferable relational understanding in AI systems.

Abstract

Large Multimodal Models (LMMs) demonstrate impressive in-context learning abilities from limited multimodal demonstrations, yet the internal mechanisms supporting such task learning remain opaque. Building on prior work of large language models, we show that a small subset of attention heads in the vision-language model OpenFlamingo-4B is responsible for transmitting representations of spatial relations. The activations of these attention heads, termed function vectors, can be extracted and manipulated to alter an LMM's performance on relational tasks. First, using both synthetic and real image datasets, we apply causal mediation analysis to identify attention heads that strongly influence relational predictions, and extract multimodal function vectors that improve zero-shot accuracy at inference time. We further demonstrate that these multimodal function vectors can be fine-tuned with a modest amount of training data, while keeping LMM parameters frozen, to significantly outperform in-context learning baselines. Finally, we show that relation-specific function vectors can be linearly combined to solve analogy problems involving novel and untrained spatial relations, highlighting the strong generalization ability of this approach. Our results show that LMMs encode spatial relational knowledge within localized internal structures, which can be systematically extracted and optimized, thereby advancing our understanding of model modularity and enhancing control over relational reasoning in LMMs.

Figures (11)

• Figure 1: Relational representations enrich perception: rather than a disconnected list of objects, relations (e.g., the boy opening the fridge next to the cabinet) provide a structured, meaningful description.
• Figure 2: Example 4-shot in-context learning (ICL) prompts for relation understanding. Each prompt includes four demonstrations followed by a query. We compare the model's performance in a consistent relational setting (A) versus a perturbed setting (B) to isolate components responsible for relational inference.
• Figure 3: Illustration of the composite function vector approach for one-shot analogy tasks. In the source analogy, relation-specific function vectors $\mathbf{v}_t$ are injected into the model to compute prediction probabilities for the target object $y_1$ given the reference object $x_1$. These probabilities define the weights $w_t$ used to form a composite function vector $\mathbf{v}_{\text{composite}}$ as a weighted sum of $\mathbf{v}_t$. The resulting vector is then transferred to guide inference in the target analogy.
• Figure 4: Average indirect effect (AIE) scores of attention heads for two spatial relations. Left panel for above relation, right panel for right of relation Each heatmap shows the AIE scores of attention heads indexed by layer and head position. Pink boxes mark the top 10 most causally influential attention heads.
• Figure 5: Top-1 prediction accuracy of zero-shot relation tasks for four models: zero-shot baseline of LMM, 4-shot ICL of LMM, initial function vector, and fine-tuned function vector. Fine-tuned vectors significantly outperform all baselines on the held-out zero-shot test set.