Enhancing Interpretability of Sparse Latent Representations with Class Information

TL;DR

This work tackles latent-space interpretability in generative models by extending Variational Sparse Coding with a class-wise alignment objective. By adding a Jensen-Shannon-based loss on per-dimension activity probabilities $\gamma$ and integrating it into the VSC objective $L_{\text{total}} = L_{\text{VSC}} + \lambda L_{\text{JSD}}$, the model enforces consistent active latent dimensions across samples within the same class, capturing both global and class-specific factors. Empirical results on MNIST and Fashion-MNIST show that the proposed method yields interpretable latent factors, with latent traversals revealing global attributes like thickness and rotation and class-specific cues such as digit intersections or heel prominence in footwear. This class-aware interpretability enhances the utility of latent representations for controllable generation and downstream tasks, particularly when universal global factors are limited or absent.

Abstract

Variational Autoencoders (VAEs) are powerful generative models for learning latent representations. Standard VAEs generate dispersed and unstructured latent spaces by utilizing all dimensions, which limits their interpretability, especially in high-dimensional spaces. To address this challenge, Variational Sparse Coding (VSC) introduces a spike-and-slab prior distribution, resulting in sparse latent representations for each input. These sparse representations, characterized by a limited number of active dimensions, are inherently more interpretable. Despite this advantage, VSC falls short in providing structured interpretations across samples within the same class. Intuitively, samples from the same class are expected to share similar attributes while allowing for variations in those attributes. This expectation should manifest as consistent patterns of active dimensions in their latent representations, but VSC does not enforce such consistency. In this paper, we propose a novel approach to enhance the latent space interpretability by ensuring that the active dimensions in the latent space are consistent across samples within the same class. To achieve this, we introduce a new loss function that encourages samples from the same class to share similar active dimensions. This alignment creates a more structured and interpretable latent space, where each shared dimension corresponds to a high-level concept, or "factor." Unlike existing disentanglement-based methods that primarily focus on global factors shared across all classes, our method captures both global and class-specific factors, thereby enhancing the utility and interpretability of latent representations.

Figures (40)

• Figure 1: Examples of different types of dog ears categorized by shape and structure. This illustrates that within a class (dogs), common features (such as ears) exist, but their specific characteristics (like shape and size) can vary across samples(dogears2025)
• Figure 2: VSC architecture. The encoder $q_{\phi}(z|x)$ outputs the mean, standard deviation, and sparsity parameters $\gamma$, encouraging sparse latent representations. The decoder $p_{\theta}(x|z)$ reconstructs the input. Compared to standard VAEs, VSC promotes interpretability by activating only a subset of latent dimensions.
• Figure 3: Illustration of global and class-specific factors. Global factors are shared across all classes, while class-specific factors vary between classes. Each sample can be represented by a combination of global and class-specific active latent dimensions.
• Figure 4: Proposed method: aligning active latent dimensions for samples within the same class. For each pair of samples, the Jensen-Shannon distance is computed between their spike probability vectors $\gamma$, encouraging similar activation patterns among class members.
• Figure 5: Calculation of the Jensen-Shannon distance for two samples within the same class. The spike probabilities $\gamma$ for each dimension are compared independently under the assumption of latent dimension independence, and their average is used as the similarity measure.