SPARC: Concept-Aligned Sparse Autoencoders for Cross-Model and Cross-Modal Interpretability

TL;DR

SPARC creates a shared sparse latent space where individual dimensions often correspond to similar high-level concepts across models and modalities, enabling direct comparison of how different architectures represent identical concepts without requiring manual alignment or model-specific analysis.

Abstract

Understanding how different AI models encode the same high-level concepts, such as objects or attributes, remains challenging because each model typically produces its own isolated representation. Existing interpretability methods like Sparse Autoencoders (SAEs) produce latent concepts individually for each model, resulting in incompatible concept spaces and limiting cross-model interpretability. To address this, we introduce SPARC (Sparse Autoencoders for Aligned Representation of Concepts), a new framework that learns a single, unified latent space shared across diverse architectures and modalities (e.g., vision models like DINO, and multimodal models like CLIP). SPARC's alignment is enforced through two key innovations: (1) a Global TopK sparsity mechanism, ensuring all input streams activate identical latent dimensions for a given concept; and (2) a Cross-Reconstruction Loss, which explicitly encourages semantic consistency between models. On Open Images, SPARC dramatically improves concept alignment, achieving a Jaccard similarity of 0.80, more than tripling the alignment compared to previous methods. SPARC creates a shared sparse latent space where individual dimensions often correspond to similar high-level concepts across models and modalities, enabling direct comparison of how different architectures represent identical concepts without requiring manual alignment or model-specific analysis. As a consequence of this aligned representation, SPARC also enables practical applications such as text-guided spatial localization in vision-only models and cross-model/cross-modal retrieval. Code and models are available at https://github.com/AtlasAnalyticsLab/SPARC

Figures (44)

• Figure 1: SPARC enables consistent concept visualization across models and modalities using shared latent dimensions. The figure shows how individual concept-specific latents (bus, cat, balloons) produce coherent spatial heatmaps across DINO oquab2024dinov and CLIP radford2021learning vision encoders, while also generating meaningful text attribution scores in CLIP's text encoder when processing full image captions. For comparison, standard CLIP similarity uses only concept names ("a bus", "a cat", "balloons") rather than full captions, as CLIP produces diffused and scattered heatmaps when given complex captions. SPARC's concept-specific latents can handle full captions while maintaining focused attribution on the target concept.
• Figure 2: Detailed architecture of the SPARC model as well as the Global TopK mechanism.
• Figure 3: Top-activating samples for the latent dimension 6463 across three streams (DINO, CLIP-img, CLIP-txt) under four SPARC configurations. Each row shows top-10 images that activate the latent. The CLIP-text stream shows no activations under Local TopK with $\lambda=1$ (top right) due to a dead neuron, demonstrating that cross-reconstruction loss alone is insufficient to ensure consistent activation patterns across streams. In contrast, Global TopK with $\lambda=1$ (bottom right) shows consistent activations across all three streams for the same object type.
• Figure 4: Individual latent attribution using SPARC dimension 279 vs. CLIP similarity baseline. (Above) Saliency maps show the same latent responding to cat-related features across image and text modalities. (Below) Text token relevance scores using SPARC and CLIP text.
• Figure 5: Cross-modal similarity attribution comparing SPARC's aligned latent space against CLIP similarity baseline. Both methods process the same image and caption, showing different attribution patterns.