Nonlinear Dimensionality Reduction with Diffusion Maps in Practice

TL;DR

This work analyzes how Diffusion Maps, a nonlinear spectral dimensionality reduction technique, are sensitive to data preprocessing and parameter settings that shape the resulting manifold. It presents a practical framework detailing kernel construction, anisotropic normalization, diffusion distance, and the role of the time parameter, while highlighting that naive component selection can misrepresent intrinsic structure. A key contribution is the Neural Reconstruction Error (NRE), which quantifies the informativeness of diffusion components and can reveal that the most meaningful embedding may involve nonconsecutive components. Through theoretical discussion and Swiss roll experiments, the paper offers concrete guidelines for parameter tuning, preprocessing, and component selection to improve reliability and interpretability in real-world data analyses.

Abstract

Diffusion Map is a spectral dimensionality reduction technique which is able to uncover nonlinear submanifolds in high-dimensional data. And, it is increasingly applied across a wide range of scientific disciplines, such as biology, engineering, and social sciences. But data preprocessing, parameter settings and component selection have a significant influence on the resulting manifold, something which has not been comprehensively discussed in the literature so far. We provide a practice oriented review of the Diffusion Map technique, illustrate pitfalls and showcase a recently introduced technique for identifying the most relevant components. Our results show that the first components are not necessarily the most relevant ones.

Figures (12)

• Figure 1: Illustrating the idea of nonlinear dimensionality reduction: On the left the data points lie on a one-dimensional manifold (spiral) in a 2 dimensional space. The idea of nonlinear dimensional reduction is to parameterize the underlying manifold. The procedure can be compared to a small insect located at one data point that only recognizes its immediate environment, but not the global appearance of the data in a higher-dimensional sense. If it moves along the manifold, it can put together the local information to create the low-dimensional representation (right). The color can be seen as the natural parameter of the manifold. The illustration is inspired by Ghojogh2023.
• Figure 2: Illustrating typical shape of Diffusion Map embedding for one dimensional non-intersecting datasets. The first column displays the original datasets (consisting each of 300 data points) in the feature space. The subsequent columns show different diffusion components with respect to the first diffusion component $\Psi_1$ for each dataset (row).
• Figure 3: The Swiss roll dataset. A. shows the original dataset (a narrow Swiss roll) with parameters $(n,\sigma^2, H)=(3000, 0.2,21)$. The color encodes the length dimension of the sheet, which is rolled up to the Swiss roll (arc-length). The width of the Swiss roll is along the Y-axis. B. shows the Diffusion Map embedding considering the first 5 components to compare them. It is worth noting that all components are shown twice to visualize how both the arc length and the width are unfolded. The parametrization used was $(\epsilon, \alpha, t, N)=(5, 1/2, 1, 3000)$.
• Figure 4: Visualization of the effect of the width parameter $\epsilon$ using the Swiss roll dataset.
• Figure 5: Illustration of the effect of the number of considered nearest neighbors $N$ using the Swiss roll with parameters $(n,\sigma^2, H)=(3000, 0.2,21)$. $(\epsilon, \alpha, t) =(1000,1/2,1)$ were used as Diffusion Map parameters.