Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving the Effectiveness and Efficiency of Stochastic Neighbour Embedding with Isolation Kernel

Ye Zhu, Kai Ming Ting

TL;DR

This work identifies two core drawbacks of using a Gaussian kernel in t-SNE: (i) point-based bandwidths can misrepresent high-dimensional structure, and (ii) data-independence imposes heavy computational costs for bandwidth optimization on large datasets. It replaces the Gaussian kernel with a data-dependent Isolation kernel, derived from space partitioning, to better reflect local distributions with a single tunable parameter $\psi$. Empirical results across 21 real-world datasets show improved neighbourhood preservation and clustering metrics for most cases, alongside substantial runtime gains that enable large-scale visualization (e.g., MNIST8M) without sacrificing accuracy. The approach provides a scalable, kernel-native pathway to accelerate and improve t-SNE, with applicability to related stochastic embedding methods and potential as a practical alternative to existing speedups.

Abstract

This paper presents a new insight into improving the performance of Stochastic Neighbour Embedding (t-SNE) by using Isolation kernel instead of Gaussian kernel. Isolation kernel outperforms Gaussian kernel in two aspects. First, the use of Isolation kernel in t-SNE overcomes the drawback of misrepresenting some structures in the data, which often occurs when Gaussian kernel is applied in t-SNE. This is because Gaussian kernel determines each local bandwidth based on one local point only, while Isolation kernel is derived directly from the data based on space partitioning. Second, the use of Isolation kernel yields a more efficient similarity computation because data-dependent Isolation kernel has only one parameter that needs to be tuned. In contrast, the use of data-independent Gaussian kernel increases the computational cost by determining n bandwidths for a dataset of n points. As the root cause of these deficiencies in t-SNE is Gaussian kernel, we show that simply replacing Gaussian kernel with Isolation kernel in t-SNE significantly improves the quality of the final visualisation output (without creating misrepresented structures) and removes one key obstacle that prevents t-SNE from processing large datasets. Moreover, Isolation kernel enables t-SNE to deal with large-scale datasets in less runtime without trading off accuracy, unlike existing methods in speeding up t-SNE.

Improving the Effectiveness and Efficiency of Stochastic Neighbour Embedding with Isolation Kernel

TL;DR

This work identifies two core drawbacks of using a Gaussian kernel in t-SNE: (i) point-based bandwidths can misrepresent high-dimensional structure, and (ii) data-independence imposes heavy computational costs for bandwidth optimization on large datasets. It replaces the Gaussian kernel with a data-dependent Isolation kernel, derived from space partitioning, to better reflect local distributions with a single tunable parameter . Empirical results across 21 real-world datasets show improved neighbourhood preservation and clustering metrics for most cases, alongside substantial runtime gains that enable large-scale visualization (e.g., MNIST8M) without sacrificing accuracy. The approach provides a scalable, kernel-native pathway to accelerate and improve t-SNE, with applicability to related stochastic embedding methods and potential as a practical alternative to existing speedups.

Abstract

This paper presents a new insight into improving the performance of Stochastic Neighbour Embedding (t-SNE) by using Isolation kernel instead of Gaussian kernel. Isolation kernel outperforms Gaussian kernel in two aspects. First, the use of Isolation kernel in t-SNE overcomes the drawback of misrepresenting some structures in the data, which often occurs when Gaussian kernel is applied in t-SNE. This is because Gaussian kernel determines each local bandwidth based on one local point only, while Isolation kernel is derived directly from the data based on space partitioning. Second, the use of Isolation kernel yields a more efficient similarity computation because data-dependent Isolation kernel has only one parameter that needs to be tuned. In contrast, the use of data-independent Gaussian kernel increases the computational cost by determining n bandwidths for a dataset of n points. As the root cause of these deficiencies in t-SNE is Gaussian kernel, we show that simply replacing Gaussian kernel with Isolation kernel in t-SNE significantly improves the quality of the final visualisation output (without creating misrepresented structures) and removes one key obstacle that prevents t-SNE from processing large datasets. Moreover, Isolation kernel enables t-SNE to deal with large-scale datasets in less runtime without trading off accuracy, unlike existing methods in speeding up t-SNE.

Paper Structure

This paper contains 27 sections, 3 theorems, 17 equations, 7 figures, 9 tables, 2 algorithms.

Table of Contents

  1. Introduction and Motivation
  2. Background
  3. Basics of t-SNE
  4. Related work
  5. Deficiencies of Gaussian kernel when used in t-SNE
  6. The first deficiency
  7. Point-based bandwidth: the cause of misrepresentation in high-dimensional space
  8. No need for point-based bandwidth in Isolation kernel
  9. The second deficiency
  10. Low computational efficiency problem with Gaussian kernel
  11. High computational efficiency with Isolation Kernel
  12. The proposed solution: using the Isolation kernel in t-SNE
  13. Isolation kernel
  14. How Isolation kernel differs from Gaussian kernel
  15. The Isolation kernel makes full use of the distributional information in small samples
  16. ...and 12 more sections

Key Result

Lemma 1

aNNE$\forall x_i, x_j \in \mathcal{X}_\mathsf{S}$ (sparse region) and $\forall x_k,x_\ell \in \mathcal{X}_\mathsf{T}$ (dense region) such that $\forall_{y\in \mathcal{X}_\mathsf{S}, z\in \mathcal{X}_\mathsf{T}} \ \rho(y)<\rho(z)$, the nearest neighbour-induced Isolation kernel $K_\psi$ has the chara where $\parallel x-y\parallel$ is the distance between $x$ and $y$; and $\rho(x)$ denotes the densi

Figures (7)

  • Figure 1: Isolation kernel versus Gaussian kernel: Distributions of similarities of points wrt the origin for three clusters of $N[0,1]$, $N[0,16]$ and $N[0,81]$ in different subspaces shown in Table \ref{['v0']}, where each is a 10-dimensional cluster (See the details in Footnote 1.) The similarities are computed in the 50-dimensional space. The left-most point in each cluster is the point closest to the origin $O$ having the highest similarity: $x_{k1}$ is the red left-most point; $x_{k2}$ is the yellow (or green) left-most point.
  • Figure 2: Two examples of partitioning $H$ using the nearest neighbour (a Voronoi diagram) on a dataset having two regions of uniform densities, where the left half has a lower density than the right half.
  • Figure 3: CPU runtime comparison of Gaussian kernel and Isolation kernel used in t-SNE on a 2-dimensional synthetic dataset.
  • Figure 4: $AUC_{RNX}$ of Gaussian Kernel and Isolation Kernel on 5 subspace clusters with different dimensionality. The parameters for each algorithm are turned according to Table \ref{['para']}.
  • Figure 5: Visualisation of FIt-SNE on two datasets.
  • ...and 2 more figures

Theorems & Definitions (4)

  • Definition 1
  • Lemma 1
  • Corollary 1
  • Corollary 2