Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models

TL;DR

The paper tackles how to improve visual understanding in multimodal large language models without updating model parameters. It introduces a plug-and-play framework that derives textual steering vectors from text-only backbones using Sparse Autoencoders, Mean Shift, and Linear Probing, and applies them to multimodal activations. Across multiple open-weight MLLMs and CV-Bench tasks, text-derived steering consistently boosts grounding and visual reasoning, with Mean Shift often delivering the strongest gains and prompting serving as a weaker baseline. The approach generalizes to out-of-distribution datasets, suggesting a lightweight, data-efficient path to enhance cross-modal grounding in MLLMs.

Abstract

Steering methods have emerged as effective and targeted tools for guiding large language models' (LLMs) behavior without modifying their parameters. Multimodal large language models (MLLMs), however, do not currently enjoy the same suite of techniques, due in part to their recency and architectural diversity. Inspired by this gap, we investigate whether MLLMs can be steered using vectors derived from their text-only LLM backbone, via sparse autoencoders (SAEs), mean shift, and linear probing. We find that text-derived steering consistently enhances multimodal accuracy across diverse MLLM architectures and visual tasks. In particular, mean shift boosts spatial relationship accuracy on CV-Bench by up to +7.3% and counting accuracy by up to +3.3%, outperforming prompting and exhibiting strong generalization to out-of-distribution datasets. These results highlight textual steering vectors as a powerful, efficient mechanism for enhancing grounding in MLLMs with minimal additional data collection and computational overhead.

Figures (16)

• Figure 1: Overview of our steering methodology. For an MLLM with a text-only LLM backbone and a given image-bound prompt, we first identify the visual concept required to address the prompt (e.g, spatial relationships, counting, object entity, or attributes). For each hidden layer $\ell$, we then determine corresponding steering vectors for the identified concept in the underlying LLM (e.g.Gemma2-2B is the backbone LLM for the its multimodal counterpart PaliGemma2-3B). We study the selection of such vectors using mean shift, linear probing, and sparse autoencoders. Finally, we intervene these steering vectors on activations corresponding to the image tokens, text tokens or both, depending on our choices of $\gamma_\mathrm{Image}$ and $\gamma_\mathrm{Text}$ from the original image-bound prompt.
• Figure 2: Effect of steering strength on color token probabilities.
• Figure 3: Left: Number of SAE features associated with each taxonomy (counting, spatial relationship, entity, and attribute) across the layers of Llama-3.1-8B, Gemma2-2B, and Gemma2-9B. Notably, SAE features for such visual concepts are sparse, numbering fewer than $10$ across 16k total SAE features (Gemma2-2B/9B) or 32k features (Llama-3.1-8B). Right: Examples of features corresponding to visual concepts, identified by the layer whose activation space they inhabit and their (arbitrary) feature ID. The feature's explanation summarizes its semantic meaning, as evidenced by the tokens and contexts on which it attains the greatest activations.
• Figure 4: Left: Depiction of mean shift method for the counting feature for Gemma-2-2B. The mean shift direction points from the mean hidden state of irrelevant tokens to the mean hidden state of relevant tokens (i.e., counting-related tokens). Activations are projected to two dimensions for the sake of visualization. Right: Similar depiction for the spatial relationship feature of Llama-3.1-8B.
• Figure 5: Efficient Grid Search with PaliGemma2-3B on the Spatial Relationship Task.