Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gradient Descent as Implicit EM in Distance-Based Neural Models

Alan Oursland

TL;DR

The paper investigates why neural networks repeatedly exhibit Bayesian-like behavior across architectures. It proves an exact gradient identity for log-sum-exp objectives over distances: $\partial L/\partial d_j = -r_j$, where $r_j = \exp(-d_j)/\sum_k \exp(-d_k)$, showing that gradient descent implicitly performs expectation-maximization. This implicit EM unifies unsupervised mixture learning, attention as conditional mixture inference, and cross-entropy classification under a single geometric mechanism driven by exponentiation and normalization. The work connects distance-based representations with probabilistic inference, clarifying the role of loss geometry in shaping learning dynamics, and outlines limits, potential collapse, and open directions for extending the framework to supervision, open-set recognition, and diagnostic tooling.

Abstract

Neural networks trained with standard objectives exhibit behaviors characteristic of probabilistic inference: soft clustering, prototype specialization, and Bayesian uncertainty tracking. These phenomena appear across architectures -- in attention mechanisms, classification heads, and energy-based models -- yet existing explanations rely on loose analogies to mixture models or post-hoc architectural interpretation. We provide a direct derivation. For any objective with log-sum-exp structure over distances or energies, the gradient with respect to each distance is exactly the negative posterior responsibility of the corresponding component: $\partial L / \partial d_j = -r_j$. This is an algebraic identity, not an approximation. The immediate consequence is that gradient descent on such objectives performs expectation-maximization implicitly -- responsibilities are not auxiliary variables to be computed but gradients to be applied. No explicit inference algorithm is required because inference is embedded in optimization. This result unifies three regimes of learning under a single mechanism: unsupervised mixture modeling, where responsibilities are fully latent; attention, where responsibilities are conditioned on queries; and cross-entropy classification, where supervision clamps responsibilities to targets. The Bayesian structure recently observed in trained transformers is not an emergent property but a necessary consequence of the objective geometry. Optimization and inference are the same process.

Gradient Descent as Implicit EM in Distance-Based Neural Models

TL;DR

The paper investigates why neural networks repeatedly exhibit Bayesian-like behavior across architectures. It proves an exact gradient identity for log-sum-exp objectives over distances: , where , showing that gradient descent implicitly performs expectation-maximization. This implicit EM unifies unsupervised mixture learning, attention as conditional mixture inference, and cross-entropy classification under a single geometric mechanism driven by exponentiation and normalization. The work connects distance-based representations with probabilistic inference, clarifying the role of loss geometry in shaping learning dynamics, and outlines limits, potential collapse, and open directions for extending the framework to supervision, open-set recognition, and diagnostic tooling.

Abstract

Neural networks trained with standard objectives exhibit behaviors characteristic of probabilistic inference: soft clustering, prototype specialization, and Bayesian uncertainty tracking. These phenomena appear across architectures -- in attention mechanisms, classification heads, and energy-based models -- yet existing explanations rely on loose analogies to mixture models or post-hoc architectural interpretation. We provide a direct derivation. For any objective with log-sum-exp structure over distances or energies, the gradient with respect to each distance is exactly the negative posterior responsibility of the corresponding component: . This is an algebraic identity, not an approximation. The immediate consequence is that gradient descent on such objectives performs expectation-maximization implicitly -- responsibilities are not auxiliary variables to be computed but gradients to be applied. No explicit inference algorithm is required because inference is embedded in optimization. This result unifies three regimes of learning under a single mechanism: unsupervised mixture modeling, where responsibilities are fully latent; attention, where responsibilities are conditioned on queries; and cross-entropy classification, where supervision clamps responsibilities to targets. The Bayesian structure recently observed in trained transformers is not an emergent property but a necessary consequence of the objective geometry. Optimization and inference are the same process.
Paper Structure (32 sections, 1 theorem, 16 equations)

This paper contains 32 sections, 1 theorem, 16 equations.

Table of Contents

  1. Introduction
  2. The Puzzle
  3. Recent Evidence
  4. This Paper
  5. Contributions
  6. The Geometric Substrate
  7. Distance-Based Representations
  8. The Log-Sum-Exp Objective
  9. Classical EM
  10. Main Result: Responsibilities Are Gradients
  11. Derivation
  12. What This Means
  13. Conditions
  14. Three Regimes of Implicit Inference
  15. Unsupervised Regime: Mixture Learning
  16. ...and 17 more sections

Key Result

Theorem 1

For any objective of the form $L = \log \sum_j \exp(-d_j)$, the gradient with respect to the $j$-th distance is the negative responsibility of component $j$: No approximations have been made. The identity holds exactly whenever the objective has LSE form and the distances are differentiable.

Theorems & Definitions (1)

  • Theorem 1