LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs

TL;DR

This work introduces LatentLens, a novel approach for mapping latent representations to descriptions in natural language, and shows that the descriptions produced by LatentLens are semantically meaningful and provide more fine-grained interpretations for humans compared to individual tokens.

Abstract

Transforming a large language model (LLM) into a Vision-Language Model (VLM) can be achieved by mapping the visual tokens from a vision encoder into the embedding space of an LLM. Intriguingly, this mapping can be as simple as a shallow MLP transformation. To understand why LLMs can so readily process visual tokens, we need interpretability methods that reveal what is encoded in the visual token representations at every layer of LLM processing. In this work, we introduce LatentLens, a novel approach for mapping latent representations to descriptions in natural language. LatentLens works by encoding a large text corpus and storing contextualized token representations for each token in that corpus. Visual token representations are then compared to their contextualized textual representations, with the top-k nearest neighbor representations providing descriptions of the visual token. We evaluate this method on 10 different VLMs, showing that commonly used methods, such as LogitLens, substantially underestimate the interpretability of visual tokens. With LatentLens instead, the majority of visual tokens are interpretable across all studied models and all layers. Qualitatively, we show that the descriptions produced by LatentLens are semantically meaningful and provide more fine-grained interpretations for humans compared to individual tokens. More broadly, our findings contribute new evidence on the alignment between vision and language representations, opening up new directions for analyzing latent representations.

Figures (29)

• Figure 1: Illustration of our method.LatentLens compares latent representations of visual tokens to contextualized text representations obtained from full sentence descriptions.
• Figure 2: Illustration of LatentLens. (1) Contextualized token representations are precomputed in multiple layers of an LLM using a large corpus of descriptions. (2) Latent representations of visual tokens are extracted from all layers of the LLM, and (3) compared against the precomputed contextualized token representations. The interpretability of the visual token based on its top-$k$ descriptions can be automatically evaluated by a VLM-judge.
• Figure 3: Interpretability of visual tokens across layers using three different "lenses". Each curve shows the percentage of interpretable visual tokens per layer across model. (a) EmbeddingLens: a large number of visual tokens is interpretable for OLMo variants but less for Llama3 and Qwen2. (b) LogitLens: low interpretability at early layers with a stark increase at later layers for most models. (c)LatentLens: the majority of visual tokens are interpretable across all models and layers.
• Figure 4: The Mid-Layer Leap: early visual tokens align to later LLM layers. For visual tokens at different stages of LLM processing, we compute their top-5 Nearest Neighbors from all other LLM layers. We find that early visual tokens, even at the input itself, align most to middle layers, e.g., layer 8 or 16. Some model combinations align most to a constant layer throughout processing, such as LLaMA3 variants. We analyze the L2 norm distributions and potential outlier effects in \ref{['app:outliers']}.
• Figure 5: Interpretability of visual tokens in off-the-shelf Qwen2-VL-7B-Instruct. We apply LatentLens and baselines to an off-the-shelf model that deviates from our controlled setup in many ways (e.g. everything finetuned). We observe the same pattern of LatentLens substantially outperforming the baselines.