Learning Sparse Visual Representations via Spatial-Semantic Factorization

TL;DR

This work tackles the conflict between semantic understanding and pixel-level reconstruction in self-supervised vision by introducing STELLAR, a sparse, spatial-semantic factorization where the latent map is modeled as $${Z(X)=L(X)S(X)}$$ with $r$ semantic tokens. By dedicating semantic invariance to the token matrix $\mathbf{S}$ and allocating spatial grounding to the localization matrix $\mathbf{L}$, STELLAR enables DINO-style view alignment while preserving precise spatial mappings for reconstruction, achieving strong performance with as few as $r=16$ tokens (e.g., $FID=2.60$ and $IN1K=79.10\%$). Empirically, STELLAR demonstrates a superior balance between semantic understanding and reconstruction across ImageNet, segmentation, and histopathology tasks, outperforming many reconstruction- and Joint-Embedding-based baselines. The work highlights sparse, interpretable representations as a promising path toward unified, efficient, and cross-modal vision systems, with potential for integrating with larger language models for multimodal understanding.

Abstract

Self-supervised learning (SSL) faces a fundamental conflict between semantic understanding and image reconstruction. High-level semantic SSL (e.g., DINO) relies on global tokens that are forced to be location-invariant for augmentation alignment, a process that inherently discards the spatial coordinates required for reconstruction. Conversely, generative SSL (e.g., MAE) preserves dense feature grids for reconstruction but fails to produce high-level abstractions. We introduce STELLAR, a framework that resolves this tension by factorizing visual features into a low-rank product of semantic concepts and their spatial distributions. This disentanglement allows us to perform DINO-style augmentation alignment on the semantic tokens while maintaining the precise spatial mapping in the localization matrix necessary for pixel-level reconstruction. We demonstrate that as few as 16 sparse tokens under this factorized form are sufficient to simultaneously support high-quality reconstruction (2.60 FID) and match the semantic performance of dense backbones (79.10% ImageNet accuracy). Our results highlight STELLAR as a versatile sparse representation that bridges the gap between discriminative and generative vision by strategically separating semantic identity from spatial geometry. Code available at https://aka.ms/stellar.

Figures (5)

• Figure 1: Comparison of learning different latent representation.
• Figure 2: Analysis of STELLAR representation. (a) Relative matrix difference in ${\bm{L}}$ and ${\bm{S}}$ under controled pixel shift in the input image. (b) Cosine distance of latent representation under random 50-100% random cropping. (c) Impact of number of sparse tokens $r$ on reconstruciton and semantic quality.
• Figure 3: Vision concepts retrieved from the training set. We show show both the image and the spatial localization of the concept.
• Figure 4: The STELLAR framework. We use a vanilla ViT to extract sparse tokens from an image, and model the latent representation as a low-rank matrix factorization, ensuring reconstruction of the original image. Clustering loss and set alignment loss are applied on the disentangled sparse tokens.
• Figure 5: Left: Concept clustering and alignment workflow. Right: visualization of learned representation.