Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A note on the relations between mixture models, maximum-likelihood and entropic optimal transport

Titouan Vayer, Etienne Lasalle

TL;DR

This note shows that maximum-likelihood estimation for discrete mixtures is equivalent to minimizing an entropic regularized optimal-transport loss. By applying the Gibbs variational principle to the log-sum-exp term, the negative log-likelihood is reformulated as a (semi-relaxed) OT problem, and optimizing over the mixture weights yields an exact entropic OT formulation with cost $C(\theta) = -\log P_{X|Y}(\cdot|\cdot,\theta)$. In the Gaussian mixture case, EM updates arise as a block-coordinate descent on the OT objective, bridging classical EM steps with OT-based updates. The framework generalizes to infinite mixtures and joint generative models, offering a unifying perspective on likelihood and entropic transport and suggesting computational avenues via Sinkhorn-type algorithms.

Abstract

This note aims to demonstrate that performing maximum-likelihood estimation for a mixture model is equivalent to minimizing over the parameters an optimal transport problem with entropic regularization. The objective is pedagogical: we seek to present this already known result in a concise and hopefully simple manner. We give an illustration with Gaussian mixture models by showing that the standard EM algorithm is a specific block-coordinate descent on an optimal transport loss.

A note on the relations between mixture models, maximum-likelihood and entropic optimal transport

TL;DR

This note shows that maximum-likelihood estimation for discrete mixtures is equivalent to minimizing an entropic regularized optimal-transport loss. By applying the Gibbs variational principle to the log-sum-exp term, the negative log-likelihood is reformulated as a (semi-relaxed) OT problem, and optimizing over the mixture weights yields an exact entropic OT formulation with cost . In the Gaussian mixture case, EM updates arise as a block-coordinate descent on the OT objective, bridging classical EM steps with OT-based updates. The framework generalizes to infinite mixtures and joint generative models, offering a unifying perspective on likelihood and entropic transport and suggesting computational avenues via Sinkhorn-type algorithms.

Abstract

This note aims to demonstrate that performing maximum-likelihood estimation for a mixture model is equivalent to minimizing over the parameters an optimal transport problem with entropic regularization. The objective is pedagogical: we seek to present this already known result in a concise and hopefully simple manner. We give an illustration with Gaussian mixture models by showing that the standard EM algorithm is a specific block-coordinate descent on an optimal transport loss.
Paper Structure (6 sections, 4 theorems, 26 equations)

This paper contains 6 sections, 4 theorems, 26 equations.

Table of Contents

  1. Notations.
  2. Introduction and preliminaries on optimal transport
  3. Maximum-likelihood for mixture models is minimization of EOT
  4. Illustration with Gaussian Mixture Models
  5. Discussions
  6. Postponed proofs

Key Result

Lemma 2.1

Let $\pi_1, \cdots, \pi_K$ be positive real numbers and $h_1, \cdots, h_K \in \mathbb{R}$. Then The optimal solution is given by $\forall k \in {[\![K]\!]}, \ p_k = \frac{\pi_k\exp(h_k)}{\sum_{j=1}^{K}\pi_j\exp(h_j)}$.

Theorems & Definitions (7)

  • Definition 2.1: (Discrete) mixture model
  • Lemma 2.1
  • Proposition 2.2: MLE for mixture models is minimization of an EOT problem
  • Lemma 5.0
  • proof
  • Lemma 5.1
  • proof