Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Navigating the Effect of Parametrization for Dimensionality Reduction

Haiyang Huang, Yingfan Wang, Cynthia Rudin

TL;DR

A new parametric method is developed, ParamRepulsor, that incorporates Hard Negative Mining and a loss function that applies a strong repulsive force and achieves state-of-the-art performance on local structure preservation for parametric methods without sacrificing the fidelity of global structural representation.

Abstract

Parametric dimensionality reduction methods have gained prominence for their ability to generalize to unseen datasets, an advantage that traditional approaches typically lack. Despite their growing popularity, there remains a prevalent misconception among practitioners about the equivalence in performance between parametric and non-parametric methods. Here, we show that these methods are not equivalent -- parametric methods retain global structure but lose significant local details. To explain this, we provide evidence that parameterized approaches lack the ability to repulse negative pairs, and the choice of loss function also has an impact. Addressing these issues, we developed a new parametric method, ParamRepulsor, that incorporates Hard Negative Mining and a loss function that applies a strong repulsive force. This new method achieves state-of-the-art performance on local structure preservation for parametric methods without sacrificing the fidelity of global structural representation. Our code is available at https://github.com/hyhuang00/ParamRepulsor.

Navigating the Effect of Parametrization for Dimensionality Reduction

TL;DR

A new parametric method is developed, ParamRepulsor, that incorporates Hard Negative Mining and a loss function that applies a strong repulsive force and achieves state-of-the-art performance on local structure preservation for parametric methods without sacrificing the fidelity of global structural representation.

Abstract

Parametric dimensionality reduction methods have gained prominence for their ability to generalize to unseen datasets, an advantage that traditional approaches typically lack. Despite their growing popularity, there remains a prevalent misconception among practitioners about the equivalence in performance between parametric and non-parametric methods. Here, we show that these methods are not equivalent -- parametric methods retain global structure but lose significant local details. To explain this, we provide evidence that parameterized approaches lack the ability to repulse negative pairs, and the choice of loss function also has an impact. Addressing these issues, we developed a new parametric method, ParamRepulsor, that incorporates Hard Negative Mining and a loss function that applies a strong repulsive force. This new method achieves state-of-the-art performance on local structure preservation for parametric methods without sacrificing the fidelity of global structural representation. Our code is available at https://github.com/hyhuang00/ParamRepulsor.

Paper Structure

This paper contains 42 sections, 3 theorems, 15 equations, 23 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Fundamentals of Neighborhood Embedding Algorithms and Contrastive Learning
  3. NCE/InfoNCE-based: t-SNE and NCVis
  4. NEG-based: UMAP
  5. PaCMAP
  6. Effect of Parametrization on DR Results
  7. ParamRepulsor
  8. Experiments
  9. Local Structure Evaluation
  10. Global Structure Evaluation
  11. Related Work
  12. Discussion and Limitations
  13. Discussion on the Depth of of Neural Network Projector
  14. Discussion on the Hyperparameter settings of Parametric DR algorithm
  15. Visualizations from all DR methods
  16. ...and 27 more sections

Key Result

Theorem 3.1

The loss of PaCMAP is generalized NEG with low-dimensional similarity functions $q_\theta^{NN}$ and $q_\theta^{FP}$: in which the functions $q_{\theta}^{NN}$ and $q_\theta^{FP}$ are

Figures (23)

  • Figure 1: Dimensionality reduction results on the MNIST digit dataset lecun2010mnist. Parametric methods (bottom row) fail to preserve the local structure of the dataset compared to their non-parametric counterparts (top row). Our method, ParamRepulsor, effectively resolves this problem via Hard Negative Mining.
  • Figure 2: Embeddings of the MNIST lecun2010mnist dataset generated by various DR methods with different numbers of hidden layers: 0 (Linear), 1, 2, or 3, or non-parametric variant. See Section \ref{['subsec:local']} for details of SVM Acc. It is helpful to envision these images in black and white (without labels) to see when clusters would be difficult to visually separate. More datasets/methods can be found in App. \ref{['app:allvis']}.
  • Figure 3: The low-dimensional scaled distance distribution between various types of point pairs with labels "3" and "8" in the embedding of the MNIST digit dataset lecun2010mnist, generated by PaCMAP, ParamPaCMAP, and ParamRepulsor (other methods in App. \ref{['subsec:adddist']}.) See definitions in Sec. \ref{['sec:background']} & \ref{['sec:paramrep']}.
  • Figure 4: Effect of Hard Negative Mining on MNIST. We progressively increase the coefficient of the repulsive force applied to MN hard negatives. Close clusters are circled. Results indicate that Hard Negative Mining alone effectively preserves local structure while maintaining relative proximities.
  • Figure 5: Effect of the number of layers on the MNIST dataset. As a supplement to Fig. \ref{['fig:mnist_layer']}, we extend the number of layers beyond three for Into-NC-t-SNE, UMAP and PaCMAP. Here, the local metric represents 10-NN accuracy, while the global metric denotes the random triplet preservation. Results show that further increasing the number of layers beyond increasing the number of layers beyond three yields only diminishing and negligible improvements in local structure on all three methods.
  • ...and 18 more figures

Theorems & Definitions (3)

  • Theorem 3.1
  • Theorem 4.1
  • Corollary 4.2