Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A unified framework for hard and soft clustering with regularized optimal transport

Jean-Frédéric Diebold, Nicolas Papadakis, Arnaud Dessein, Charles-Alban Deledalle

TL;DR

The convergence property of the generalized $\lambda-\text{EM}$ algorithms is studied and it is shown that each algorithmic step has a closed form solution when inferring finite mixture models of exponential families.

Abstract

In this paper, we formulate the problem of inferring a Finite Mixture Model from discrete data as an optimal transport problem with entropic regularization of parameter $λ\geq 0$. Our method unifies hard and soft clustering, the Expectation-Maximization (EM) algorithm being exactly recovered for $λ=1$. The family of clustering algorithm we propose rely on the resolution of nonconvex problems using alternating minimization. We study the convergence property of our generalized $λ-$EM algorithms and show that each step in the minimization process has a closed form solution when inferring finite mixture models of exponential families. Experiments highlight the benefits of taking a parameter $λ>1$ to improve the inference performance and $λ\to 0$ for classification.

A unified framework for hard and soft clustering with regularized optimal transport

TL;DR

The convergence property of the generalized algorithms is studied and it is shown that each algorithmic step has a closed form solution when inferring finite mixture models of exponential families.

Abstract

In this paper, we formulate the problem of inferring a Finite Mixture Model from discrete data as an optimal transport problem with entropic regularization of parameter . Our method unifies hard and soft clustering, the Expectation-Maximization (EM) algorithm being exactly recovered for . The family of clustering algorithm we propose rely on the resolution of nonconvex problems using alternating minimization. We study the convergence property of our generalized EM algorithms and show that each step in the minimization process has a closed form solution when inferring finite mixture models of exponential families. Experiments highlight the benefits of taking a parameter to improve the inference performance and for classification.

Paper Structure

This paper contains 38 sections, 5 theorems, 29 equations, 7 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions and outline.
  3. Related Work
  4. Hard clustering and $k$-Means
  5. Soft clustering and Expectation-Maximization (EM) in Exponential Family Mixture Models
  6. Relations between Hard and Soft Clustering or Estimation
  7. Statistical Inference by Minimizing Information Divergences
  8. Comparison of Exponential Family Mixture Models by Optimal Transport
  9. Clustering and Statistical Inference by OT minimization
  10. Parameter Estimation in Finite Mixture Models with Regularized Optimal Transport
  11. Hard and soft Clustering
  12. Finite Mixture Models
  13. General Problem
  14. Proposed model with Transport Plan Relaxation
  15. $\lambda-$EM algorithm
  16. ...and 23 more sections

Key Result

Proposition 1

Any coordinatewise minimum of the relaxed problem problem$(\hat{\boldsymbol\pi}, \hat{\boldsymbol\omega},\hat{\boldsymbol\eta})$ is an admissible solution and a coordinatewise minimum of problem generalproblem.

Figures (7)

  • Figure 1: Examples of inference of a 1D GMM.
  • Figure 2: Mean and variance of the inference accuracy ($MW_2$ distance), for various regularization levels $\lambda$.
  • Figure 3: 2D Inferences for several values of $\lambda$
  • Figure 4: (a) Inference accuracy, in terms of $MW_2$ distance, for various number of samples $n$ and different regularization levels $\lambda$. (b) Inference accuracy, in terms of $MW_2$ distance, for various number of clusters $k$ in the inferred GMMs, and different regularization levels $\lambda$.
  • Figure 5: Classification accuracy of the MNIST database for $\lambda\in[0,3]$.
  • ...and 2 more figures

Theorems & Definitions (9)

  • Proposition 1
  • Definition 1
  • Theorem 1: Proof in section \ref{['ssec:convergence']} of the Appendix
  • Proposition 2: Proof in section \ref{['ssec:likelihood']} of the appendix
  • Proposition 3
  • proof
  • Definition 2
  • Theorem 2
  • proof