Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large data limits and scaling laws for tSNE

Ryan Murray, Adam Pickarski

TL;DR

It is shown that embeddings of the original tSNE algorithm cannot have any consistent limit as $n \to \infty", and a rescaled model is proposed which mitigates the asymptotic decay of the attractive energy, and which does have a consistent limit.

Abstract

This work considers large-data asymptotics for t-distributed stochastic neighbor embedding (tSNE), a widely-used non-linear dimension reduction algorithm. We identify an appropriate continuum limit of the tSNE objective function, which can be viewed as a combination of a kernel-based repulsion and an asymptotically-vanishing Laplacian-type regularizer. As a consequence, we show that embeddings of the original tSNE algorithm cannot have any consistent limit as $n \to \infty$. We propose a rescaled model which mitigates the asymptotic decay of the attractive energy, and which does have a consistent limit.

Large data limits and scaling laws for tSNE

TL;DR

It is shown that embeddings of the original tSNE algorithm cannot have any consistent limit as $n \to \infty", and a rescaled model is proposed which mitigates the asymptotic decay of the attractive energy, and which does have a consistent limit.

Abstract

This work considers large-data asymptotics for t-distributed stochastic neighbor embedding (tSNE), a widely-used non-linear dimension reduction algorithm. We identify an appropriate continuum limit of the tSNE objective function, which can be viewed as a combination of a kernel-based repulsion and an asymptotically-vanishing Laplacian-type regularizer. As a consequence, we show that embeddings of the original tSNE algorithm cannot have any consistent limit as . We propose a rescaled model which mitigates the asymptotic decay of the attractive energy, and which does have a consistent limit.

Paper Structure

This paper contains 13 sections, 13 theorems, 73 equations.

Table of Contents

  1. Introduction
  2. Setup and Main Results
  3. Problem Setup
  4. tSNE as Attraction-Repulsion
  5. Perplexity and Bandwidth
  6. Main Results
  7. Related Works
  8. Proof of Main Results
  9. Stochastic Convergence of the Bandwidths
  10. Non-existence of Limiting Minimizers
  11. Consistency of the Rescaled Model
  12. Minimizers of rescaled tSNE
  13. Appendix

Key Result

Proposition 2.2

\newlabelScaling_prop0 Suppose that $h_n$ is a sequence for which $nh_n^d/\log(n)\to\infty$ and $h_n\to 0$. Suppose further that $\rho(x)$ is a uniformly continuous density that is bounded above and below. Then if $\widehat{\sigma}_{n}(x)$ is chosen so that $\mathrm{PP}_n(x|h_n\widehat{\sigma}_n(x)

Theorems & Definitions (21)

  • Remark 2.1
  • Proposition 2.2: Local Adaptivity
  • Theorem 2.3: Ill-posedness
  • Theorem 2.4: Consistency
  • Remark 2.5
  • Theorem 2.6: Well-posedness
  • Theorem 2.7: Necessary Conditions and Regularity
  • Lemma 4.1
  • Proof 1: Proof of Proposition \ref{['Scaling_prop']}
  • Definition 4.2
  • ...and 11 more