Interpretable non-linear dimensionality reduction using gaussian weighted linear transformation

TL;DR

The paper tackles the trade-off between interpretability and non-linear expressiveness in dimensionality reduction by introducing a Gaussian-weighted mixture of linear transformations, enabling a non-linear mapping $f(\mathbf{x}) = \sum_{i=1}^m w_i(\mathbf{x}) \; T_i(\mathbf{x})$ where $T_i(\mathbf{x}) = \mathbf{M}_i \mathbf{x}$ and $w_i(\mathbf{x})$ are Gaussian-based weights. The optimization targets pairwise distance preservation, via $\mathcal{L} = \frac{1}{N} \sum_{i,j} (\|\mathbf{x}_i - \mathbf{x}_j\| - \|f(\mathbf{x}_i) - f(\mathbf{x}_j)\|)^2$, and is trained with a two-phase procedure including initialization and gradient-based updates (e.g., Adam), with efficiency gained by a $k$-nearest-neighbor approximation. The approach yields interpretable components by quantifying per-dimension influence and space expansion/contraction, demonstrated on synthetic curves with explicit interpretability metrics, and is complemented by a public Python package. This work provides a practical, transparent alternative to purely non-linear embeddings like t-SNE, enabling extended transformations to new data without retraining and offering actionable insights into the geometric preservation and modification by the learned mappings.

Abstract

Dimensionality reduction techniques are fundamental for analyzing and visualizing high-dimensional data. With established methods like t-SNE and PCA presenting a trade-off between representational power and interpretability. This paper introduces a novel approach that bridges this gap by combining the interpretability of linear methods with the expressiveness of non-linear transformations. The proposed algorithm constructs a non-linear mapping between high-dimensional and low-dimensional spaces through a combination of linear transformations, each weighted by Gaussian functions. This architecture enables complex non-linear transformations while preserving the interpretability advantages of linear methods, as each transformation can be analyzed independently. The resulting model provides both powerful dimensionality reduction and transparent insights into the transformed space. Techniques for interpreting the learned transformations are presented, including methods for identifying suppressed dimensions and how space is expanded and contracted. These tools enable practitioners to understand how the algorithm preserves and modifies geometric relationships during dimensionality reduction. To ensure the practical utility of this algorithm, the creation of user-friendly software packages is emphasized, facilitating its adoption in both academia and industry.

Figures (5)

• Figure 1: Visualization of a dataset generated using Scikit-learn's make_s_curve.
• Figure 2: The data points visualized in the reduced 2D space.
• Figure 3: The values $\bar{w}(p)_{j=2}$, corresponding to the original y-dimension, are shown as the background. The data points in the reduced space are plotted on top, with the point colors corresponding to their original y-values.
• Figure 4: The variance $Var(\bar{w}(p))$ over the mesh grid is plotted as the background. Data points in the reduced space are plotted on top.
• Figure 5: The values of $N(p)$ calculated over the mesh grid are shown as the background. Data points in the reduced space are plotted on top.