Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deriving Decoder-Free Sparse Autoencoders from First Principles

Alan Oursland

TL;DR

This work derives a principled, decoder-free sparse autoencoder by grounding model design in implicit EM theory. It shows that gradient signals correspond to component responsibilities under a log-sum-exp objective, and that without explicit volume control representations collapse, which is mitigated by variance and decorrelation penalties. The proposed model uses a single linear encoder with an LSE loss plus InfoMax regularization, producing interpretable mixture components and competitive discriminative performance with far fewer parameters and no decoder. Extensive experiments verify the gradient-responsibility identity, demonstrate the necessity of volume control, and reveal EM-like training dynamics where lower loss does not guarantee better features. The results suggest implicit EM as a viable, generative foundation for principled neural architecture design and connect to self-supervised learning practices that emphasize variance and covariance regularization.

Abstract

Gradient descent on log-sum-exp (LSE) objectives performs implicit expectation--maximization (EM): the gradient with respect to each component output equals its responsibility. The same theory predicts collapse without volume control analogous to the log-determinant in Gaussian mixture models. We instantiate the theory in a single-layer encoder with an LSE objective and InfoMax regularization for volume control. Experiments confirm the theory's predictions. The gradient--responsibility identity holds exactly; LSE alone collapses; variance prevents dead components; decorrelation prevents redundancy. The model exhibits EM-like optimization dynamics in which lower loss does not correspond to better features and adaptive optimizers offer no advantage. The resulting decoder-free model learns interpretable mixture components, confirming that implicit EM theory can prescribe architectures.

Deriving Decoder-Free Sparse Autoencoders from First Principles

TL;DR

This work derives a principled, decoder-free sparse autoencoder by grounding model design in implicit EM theory. It shows that gradient signals correspond to component responsibilities under a log-sum-exp objective, and that without explicit volume control representations collapse, which is mitigated by variance and decorrelation penalties. The proposed model uses a single linear encoder with an LSE loss plus InfoMax regularization, producing interpretable mixture components and competitive discriminative performance with far fewer parameters and no decoder. Extensive experiments verify the gradient-responsibility identity, demonstrate the necessity of volume control, and reveal EM-like training dynamics where lower loss does not guarantee better features. The results suggest implicit EM as a viable, generative foundation for principled neural architecture design and connect to self-supervised learning practices that emphasize variance and covariance regularization.

Abstract

Gradient descent on log-sum-exp (LSE) objectives performs implicit expectation--maximization (EM): the gradient with respect to each component output equals its responsibility. The same theory predicts collapse without volume control analogous to the log-determinant in Gaussian mixture models. We instantiate the theory in a single-layer encoder with an LSE objective and InfoMax regularization for volume control. Experiments confirm the theory's predictions. The gradient--responsibility identity holds exactly; LSE alone collapses; variance prevents dead components; decorrelation prevents redundancy. The model exhibits EM-like optimization dynamics in which lower loss does not correspond to better features and adaptive optimizers offer no advantage. The resulting decoder-free model learns interpretable mixture components, confirming that implicit EM theory can prescribe architectures.
Paper Structure (99 sections, 11 equations, 3 figures, 9 tables)

This paper contains 99 sections, 11 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Implicit EM Theory
  3. The Question
  4. This Paper
  5. Contribution
  6. Implicit EM theory is generative.
  7. Each theoretical prediction is confirmed.
  8. Training dynamics are consistent with EM structure.
  9. The derived model performs well.
  10. Roadmap
  11. What Implicit EM Theory Requires
  12. Distance-Based Representations
  13. The LSE Identity
  14. The Volume Control Requirement
  15. The collapse problem.
  16. ...and 84 more sections

Figures (3)

  • Figure 1: Gradient vs. responsibility for 8,192 random activations. All points lie on $y = x$, confirming the identity in \ref{['eq:gradient-responsibility']} to floating-point precision.
  • Figure 2: Learned encoder weights. (a) Theory-derived model: features form recognizable digit prototypes with center-surround structure, consistent with mixture components competing for data. (b) Standard SAE: encoder weights show little interpretable structure; representational content is carried primarily by the decoder.
  • Figure 3: Loss curves for SGD (left) and Adam (right) across learning rates. SGD curves plateau; Adam curves continue descending. Despite achieving substantially lower loss, Adam produces features of comparable quality.