VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set

TL;DR

VL-SAE addresses the interpretability gap in vision-language alignment by learning a unified concept set that binds vision and language representations through a shared, distance-based encoder. It combines an auxiliary autoencoder for explicit LVLM alignment with two modality-specific decoders to prevent concept mismatch, training to jointly maximize representation alignment and reconstruction; the learned concepts enable interpretation and targeted enhancement of downstream tasks. Across CVLMs and LVLMs, VL-SAE yields higher intra-concept semantic consistency and better cross-modal alignment, improving zero-shot classification and reducing object hallucinations via concept-level adjustments. The approach offers a scalable, self-supervised pathway to diagnose and improve multi-modal alignment with broad applicability and minimal overhead.

Abstract

The alignment of vision-language representations endows current Vision-Language Models (VLMs) with strong multi-modal reasoning capabilities. However, the interpretability of the alignment component remains uninvestigated due to the difficulty in mapping the semantics of multi-modal representations into a unified concept set. To address this problem, we propose VL-SAE, a sparse autoencoder that encodes vision-language representations into its hidden activations. Each neuron in its hidden layer correlates to a concept represented by semantically similar images and texts, thereby interpreting these representations with a unified concept set. To establish the neuron-concept correlation, we encourage semantically similar representations to exhibit consistent neuron activations during self-supervised training. First, to measure the semantic similarity of multi-modal representations, we perform their alignment in an explicit form based on cosine similarity. Second, we construct the VL-SAE with a distance-based encoder and two modality-specific decoders to ensure the activation consistency of semantically similar representations. Experiments across multiple VLMs (e.g., CLIP, LLaVA) demonstrate the superior capability of VL-SAE in interpreting and enhancing the vision-language alignment. For interpretation, the alignment between vision and language representations can be understood by comparing their semantics with concepts. For enhancement, the alignment can be strengthened by aligning vision-language representations at the concept level, contributing to performance improvements in downstream tasks, including zero-shot image classification and hallucination elimination. Codes are available at https://github.com/ssfgunner/VL-SAE.

Figures (12)

• Figure 1: (a) Current interpretation methods are designed for single-modal representations. (b) Using current methods for each modality leads to a mismatch in the concept sets, hindering the interpretation of the vision-language alignment. (c) We propose the VL-SAE to interpret the alignment mechanism by mapping the representation semantics of both modalities into a unified concept set.
• Figure 2: VL-SAE consists of a distance-based encoder that maps semantically similar vision-language representations to similar activations, and two modality-specific decoders that reconstruct original representations from these activations.
• Figure 3: Quantitative evaluation of the learned concept set. We compare the VL-SAE with other methods on the consistency between vision and language semantics within the same neuron (Intra-Similarity) and the semantic diversity across different neurons (Inter-Similarity) on multiple VLMs.
• Figure 4: Qualitative comparisons among concepts of SAE-D, SAE-S and the proposed VL-SAE. All SAEs are trained with the vision-language representations of LLaVA 1.5.
• Figure 5: Interpreting the vision-language alignment of OpenCLIP-ViT-L/14 with VL-SAE. Given an image-text pair, we visualize the concepts activated by corresponding vision and language representations separately, as well as the aligned concepts co-activated by both modalities. Concepts only related to single-modal representations are highlighted in purple.