A general framework for adaptive nonparametric dimensionality reduction

TL;DR

The paper addresses the challenge of hyper-parameter tuning in neighborhood-based nonparametric dimensionality reduction by introducing ABIDE, an intrinsic-dimension estimator that jointly outputs a global dimension $d^*$ and per-point neighborhood sizes $k^*$. These outputs yield locally uniform neighborhoods that adapt to data geometry and are integrated into LLE, spectral clustering, and UMAP to create adaptive variants (LLE^*, SC^*, UMAP^*) with improved unsupervised embeddings and supervised classification performance. ABIDE is shown to be consistent and asymptotically normal, providing a solid theoretical foundation, and experiments on real and synthetic datasets (Iris, MNIST, Manifolds, News Articles) demonstrate robust improvements in clustering metrics and visualization while reducing the need for manual hyper-parameter tuning. Overall, the framework offers a principled, general approach to adaptive nonparametric dimensionality reduction with broad applicability and practical impact for data visualization and learning tasks.

Abstract

Dimensionality reduction is a fundamental task in modern data science. Several projection methods specifically tailored to take into account the non-linearity of the data via local embeddings have been proposed. Such methods are often based on local neighbourhood structures and require tuning the number of neighbours that define this local structure, and the dimensionality of the lower-dimensional space onto which the data are projected. Such choices critically influence the quality of the resulting embedding. In this paper, we exploit a recently proposed intrinsic dimension estimator which also returns the optimal locally adaptive neighbourhood sizes according to some desirable criteria. In principle, this adaptive framework can be employed to perform an optimal hyper-parameter tuning of any dimensionality reduction algorithm that relies on local neighbourhood structures. Numerical experiments on both real-world and simulated datasets show that the proposed method can be used to significantly improve well-known projection methods when employed for various learning tasks, with improvements measurable through both quantitative metrics and the quality of low-dimensional visualizations.

Figures (5)

• Figure 1: Representation of the main LLE$^*$ steps.
• Figure 2: Data visualisation in the reduced space using LLE$^*$ (first row) and the standard version (third row). The second row shows histograms of $k^*$, with summary statistics displayed for median values and standard deviation.
• Figure 3: Visualisation of performance metrics of ARI, Homogeneity, Completeness, and V-Measure as the number of considered neighbours varies, fixing the dimension of the target space $d^*$, returned by ABIDE, for the 4 datasets considered. Horizontally, in dashed lines, we report the metrics of LLE$^*$. Vertically, the median (in orange) and the mean (in blue) of $k^*$ are shown. The grey shaded area represents an arbitrary neighbourhood around these summary measures (mean and median) to highlight the proximity of the adaptive results to these reference values. With respect to the Iris dataset, the mean and median coincide.
• Figure 4: Visualisation of the performance of the 4 considered metrics ARI, Homogeneity, Completeness, and V-Measure for different choices of n_components and n_neighbors in LLE on MNIST dataset. The horizontal dashed lines represent the results of LLE$^*$.
• Figure 5: (A) The results of the accuracy score for the datasets considered are shown for different configurations of LLE, adaptive (LLE$^*$) and non-adaptive (LLE). (B) Boxplots of accuracy scores comparing adaptive and non-adaptive dimensionality reduction approaches across MNIST, News Articles, and Manifolds datasets. The Boxplots show the distribution of scores from the hyper-parameter grid search, while the stars represent the adaptive method results. Results are averaged using an 80%-20% holdout approach over three different runs. The table on the bottom right shows the hyper-parameter grid used for the non-adaptive approach, exploring different combinations of n_neighbours and n_components values.