Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SparseJEPA: Sparse Representation Learning of Joint Embedding Predictive Architectures

Max Hartman, Lav Varshney

TL;DR

SparseJEPA addresses the interpretability bottleneck of dense JEPA embeddings by introducing a sparsity penalty that enforces group-based latent structure, guided by the oi-VAE framework. The method combines a lightweight Vision Transformer backbone with a sparsity module that promotes latent variable grouping, and provides a theoretical foundation showing reduced multiinformation via grouping, invoked through the data processing inequality. Empirically, SparseJEPA trained on CIFAR-100 yields improved linear-probe transfer across multiple tasks compared to JEPA, while maintaining predictive performance. This work advances self-supervised learning by delivering more interpretable, efficient representations and offers a pathway toward object-centric, structured latent spaces with broader applicability in vision systems.

Abstract

Joint Embedding Predictive Architectures (JEPA) have emerged as a powerful framework for learning general-purpose representations. However, these models often lack interpretability and suffer from inefficiencies due to dense embedding representations. We propose SparseJEPA, an extension that integrates sparse representation learning into the JEPA framework to enhance the quality of learned representations. SparseJEPA employs a penalty method that encourages latent space variables to be shared among data features with strong semantic relationships, while maintaining predictive performance. We demonstrate the effectiveness of SparseJEPA by training on the CIFAR-100 dataset and pre-training a lightweight Vision Transformer. The improved embeddings are utilized in linear-probe transfer learning for both image classification and low-level tasks, showcasing the architecture's versatility across different transfer tasks. Furthermore, we provide a theoretical proof that demonstrates that the grouping mechanism enhances representation quality. This was done by displaying that grouping reduces Multiinformation among latent-variables, including proofing the Data Processing Inequality for Multiinformation. Our results indicate that incorporating sparsity not only refines the latent space but also facilitates the learning of more meaningful and interpretable representations. In further work, hope to further extend this method by finding new ways to leverage the grouping mechanism through object-centric representation learning.

SparseJEPA: Sparse Representation Learning of Joint Embedding Predictive Architectures

TL;DR

SparseJEPA addresses the interpretability bottleneck of dense JEPA embeddings by introducing a sparsity penalty that enforces group-based latent structure, guided by the oi-VAE framework. The method combines a lightweight Vision Transformer backbone with a sparsity module that promotes latent variable grouping, and provides a theoretical foundation showing reduced multiinformation via grouping, invoked through the data processing inequality. Empirically, SparseJEPA trained on CIFAR-100 yields improved linear-probe transfer across multiple tasks compared to JEPA, while maintaining predictive performance. This work advances self-supervised learning by delivering more interpretable, efficient representations and offers a pathway toward object-centric, structured latent spaces with broader applicability in vision systems.

Abstract

Joint Embedding Predictive Architectures (JEPA) have emerged as a powerful framework for learning general-purpose representations. However, these models often lack interpretability and suffer from inefficiencies due to dense embedding representations. We propose SparseJEPA, an extension that integrates sparse representation learning into the JEPA framework to enhance the quality of learned representations. SparseJEPA employs a penalty method that encourages latent space variables to be shared among data features with strong semantic relationships, while maintaining predictive performance. We demonstrate the effectiveness of SparseJEPA by training on the CIFAR-100 dataset and pre-training a lightweight Vision Transformer. The improved embeddings are utilized in linear-probe transfer learning for both image classification and low-level tasks, showcasing the architecture's versatility across different transfer tasks. Furthermore, we provide a theoretical proof that demonstrates that the grouping mechanism enhances representation quality. This was done by displaying that grouping reduces Multiinformation among latent-variables, including proofing the Data Processing Inequality for Multiinformation. Our results indicate that incorporating sparsity not only refines the latent space but also facilitates the learning of more meaningful and interpretable representations. In further work, hope to further extend this method by finding new ways to leverage the grouping mechanism through object-centric representation learning.

Paper Structure

This paper contains 12 sections, 2 theorems, 7 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Joint Embedding Predictive Architecture (JEPA)
  4. Output Interpretable Variational Autoencoders (oi-VAE)
  5. Sparse Autoencoders
  6. Sparse grouping of latent variables reduces redundant multiinformation
  7. Method
  8. Overview of the SparseJEPA Architecture
  9. Sparsity Module
  10. Results
  11. Conclusion
  12. Limitations

Key Result

Lemma 1

Let $X_1, \ldots, X_n$ be random variables with joint distribution $p(x_1, \ldots, x_n)$. Suppose the variables are partitioned into $m$ nonempty subsets $S_1, \ldots, S_m$. For each $j = 1, \ldots, m$, define a latent variable $G_j$ that depends only on the subset $\{X_i : i \in S_j\}$; that is, where $f_j$ is some deterministic function. Then, the multiinformation among the latent variables sat

Figures (2)

  • Figure 1: SparseJEPA Architecure
  • Figure 2: Image of a dog, where the two red patches contain more mutual information with eachother than the blue patch.

Theorems & Definitions (4)

  • Lemma 1: Grouping Reduces Multiinformation
  • proof
  • Theorem 1
  • proof