Towards Interpreting Visual Information Processing in Vision-Language Models

TL;DR

Interpretability of vision-language models remains challenging. The authors analyze LLaVA using ablations, logit lens probing, and attention knockout to map where object information lives, how visual representations refine toward vocabulary-like concepts, and how information flows to the output token. They find object details localize to object-position visual tokens, visual-token embeddings align with interpretable textual concepts across layers, and object signals are extracted at the last token position in mid-to-late layers, with attention flows confirming direct reliance on object tokens. These results advance mechanistic understanding of multimodal processing and inform the design of more interpretable and controllable VLMs.

Abstract

Vision-Language Models (VLMs) are powerful tools for processing and understanding text and images. We study the processing of visual tokens in the language model component of LLaVA, a prominent VLM. Our approach focuses on analyzing the localization of object information, the evolution of visual token representations across layers, and the mechanism of integrating visual information for predictions. Through ablation studies, we demonstrated that object identification accuracy drops by over 70\% when object-specific tokens are removed. We observed that visual token representations become increasingly interpretable in the vocabulary space across layers, suggesting an alignment with textual tokens corresponding to image content. Finally, we found that the model extracts object information from these refined representations at the last token position for prediction, mirroring the process in text-only language models for factual association tasks. These findings provide crucial insights into how VLMs process and integrate visual information, bridging the gap between our understanding of language and vision models, and paving the way for more interpretable and controllable multimodal systems.

Figures (8)

• Figure 1: In adapter-style Vision-Language Models (VLMs), the visual tokens (in green) are soft prompts for the language model (LM), and are not interpretable through the vocabulary embedding. Through a set of object identification tasks, we find that (1) object information can be localized to a subset of visual tokens, (2) the representations of the visual tokens evolve towards interpretable text embeddings, and (3) the model extracts some information from the visual tokens to the last token position in the middle to late layers, to identify the object.
• Figure 2: Overview of our ablation experiments. (1) We ablate some visual tokens that potentially contain object-specific information, (2) prompt the model to describe the image, or answer object-specific questions, then (3) measure the impact of token ablation by calculating the percentage of initially correct object identifications that become incorrect after ablation.
• Figure 3: Examples of tokens and their positions that the logit lens yields in the late layers. The labelled tokens come from a range of layers around the middle-to-late layers. In our labelling for an image, tokens that are part of a word are completed with an educated guess e.g. "diam"(ond)
• Figure 4: Example of dataset images for the object identification tasks.
• Figure 5: Examples of curated images and their questions. We ask the model the question and prefill its answer with "It is a". The correct answer is underlined.