Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence analysis of t-SNE as a gradient flow for point cloud on a manifold

Seonghyeon Jeong, Hau-Tieng Wu

TL;DR

This work provides the first rigorous grounding of t-SNE as a gradient flow on data sampled from a manifold, proving that the gradient-flow trajectories of embedded points are bounded and that a global minimizer of the KL-divergence exists. By analyzing mutual distances, perplexity-induced scales, and the affinities p_{ij} and q_{ij}, the authors establish conditions under which no point escapes to infinity and show convergence properties of the nonconvex objective. The results rely on manifold regularity, W1 convergence of the empirical measure to the data-generating measure, and careful control of perplexity to keep sigma_i bounded away from zero. Consequently, the paper provides theoretical guarantees for the well-posedness and convergence behavior of t-SNE embeddings in a manifold setting, informing perplexity choices and offering a pathway to understanding minimizer structure (up to isometries).

Abstract

We present a theoretical foundation regarding the boundedness of the t-SNE algorithm. t-SNE employs gradient descent iteration with Kullback-Leibler (KL) divergence as the objective function, aiming to identify a set of points that closely resemble the original data points in a high-dimensional space, minimizing KL divergence. Investigating t-SNE properties such as perplexity and affinity under a weak convergence assumption on the sampled dataset, we examine the behavior of points generated by t-SNE under continuous gradient flow. Demonstrating that points generated by t-SNE remain bounded, we leverage this insight to establish the existence of a minimizer for KL divergence.

Convergence analysis of t-SNE as a gradient flow for point cloud on a manifold

TL;DR

This work provides the first rigorous grounding of t-SNE as a gradient flow on data sampled from a manifold, proving that the gradient-flow trajectories of embedded points are bounded and that a global minimizer of the KL-divergence exists. By analyzing mutual distances, perplexity-induced scales, and the affinities p_{ij} and q_{ij}, the authors establish conditions under which no point escapes to infinity and show convergence properties of the nonconvex objective. The results rely on manifold regularity, W1 convergence of the empirical measure to the data-generating measure, and careful control of perplexity to keep sigma_i bounded away from zero. Consequently, the paper provides theoretical guarantees for the well-posedness and convergence behavior of t-SNE embeddings in a manifold setting, informing perplexity choices and offering a pathway to understanding minimizer structure (up to isometries).

Abstract

We present a theoretical foundation regarding the boundedness of the t-SNE algorithm. t-SNE employs gradient descent iteration with Kullback-Leibler (KL) divergence as the objective function, aiming to identify a set of points that closely resemble the original data points in a high-dimensional space, minimizing KL divergence. Investigating t-SNE properties such as perplexity and affinity under a weak convergence assumption on the sampled dataset, we examine the behavior of points generated by t-SNE under continuous gradient flow. Demonstrating that points generated by t-SNE remain bounded, we leverage this insight to establish the existence of a minimizer for KL divergence.
Paper Structure (11 sections, 18 theorems, 196 equations)

This paper contains 11 sections, 18 theorems, 196 equations.

Table of Contents

  1. Introduction
  2. t-SNE algorithm
  3. Conditions for analysis
  4. Perplexity
  5. Gradient descent
  6. Boundedness of $\{ y_i \}$ and existence of a minimizer
  7. When one point $y_j$ diverges to $\infty$
  8. Information from mutual distances
  9. Affinities $p_{ij}$ and $q_{ij}'$
  10. The first main theorem: Boundedness of $\{ y_i \}$
  11. The second main theorem: existence of a minimizer

Key Result

Theorem 1.1

The points in $\mathbb{R}^2$ generated by t-SNE are uniformly bounded.

Theorems & Definitions (40)

  • Theorem 1.1: Main theorem 1, rough statement
  • Theorem 1.2: Main theorem 2, rough statement
  • Definition 2.1
  • Definition 3.1
  • Lemma 4.1
  • proof
  • Lemma 4.2
  • proof
  • Proposition 4.3
  • Proposition 5.1
  • ...and 30 more