How Scale Breaks "Normalized Stress" and KL Divergence: Rethinking Quality Metrics

TL;DR

The paper tackles the problem that widely used DR quality metrics, notably normalized stress and KL divergence, are sensitive to uniform scaling of embeddings, which can mislead comparisons across methods. It introduces scale-invariant variants—Scale-Normalized Stress (SNS) and Scale-Normalized KL (SNKL)—and related measures like FSNS and FSKL, deriving their motivations and computation. Through extensive experiments on diverse datasets and DR techniques, the authors show that SNS and SNKL yield more consistent, interpretable rankings and align with expected performance, whereas scale-sensitive metrics can produce misleading conclusions (including random embeddings appearing superior). The work provides practical recommendations for robust DR evaluation, supports reproducibility with open-source code and interactive demonstrations, and highlights the need for scale-aware benchmarking in visualization and ML contexts.

Abstract

Complex, high-dimensional data is ubiquitous across many scientific disciplines, including machine learning, biology, and the social sciences. One of the primary methods of visualizing these datasets is with two-dimensional scatter plots that visually capture some properties of the data. Because visually determining the accuracy of these plots is challenging, researchers often use quality metrics to measure the projection's accuracy and faithfulness to the original data. One of the most commonly employed metrics, normalized stress, is sensitive to uniform scaling (stretching, shrinking) of the projection, despite this act not meaningfully changing anything about the projection. Another quality metric, the Kullback--Leibler (KL) divergence used in the popular t-Distributed Stochastic Neighbor Embedding (t-SNE) technique, is also susceptible to this scale sensitivity. We investigate the effect of scaling on stress and KL divergence analytically and empirically by showing just how much the values change and how this affects dimension reduction technique evaluations. We introduce a simple technique to make both metrics scale-invariant and show that it accurately captures expected behavior on a small benchmark.

Figures (18)

• Figure 1: MDS shepard1962analysis, t-SNE van2008visualizing, and RND (random) embeddings of the well-known $Iris$ dataset from left to right (bottom). The plot (top) shows the values of the normalized stress metric for these three embeddings and clearly illustrates the sensitivity to scale. As one uniformly scales the embeddings to be larger or smaller, the value of normalized stress changes. Notably, at different scales, different embeddings have lower stress, including the absurd situation where the random embedding has the lowest stress (beyond scale 9). Moreover, all six different algorithm orders can be found by selecting different scales.
• Figure 2: Embeddings and Shepard diagrams with monotonic fitted line for the $Wine$ dataset captured by MDS (left, $\rho=0.36$) and RANDOM (right,$\rho=0.001$). Note the positive correlation for MDS, which indicates good distance preservation and the low correlation for RANDOM, which indicates bad distance preservation. Note that scaling the embedding corresponds to stretching or shrinking the $y$-axes in these plots, which does not effect the (rank) correlation.
• Figure 3: RND, MDS, and t-SNE embeddings of the Wine dataset (right, bottom left) compared against the variation of KL Divergence with scale (top left). KL divergence varies substantially with scale, and the expected order of t-SNE $<$ MDS $<$ Random is not always observed. As with stress, all six possible algorithm orders can occur depending on scale.
• Figure 4: The variation of KL divergence with the Gaussian kernel implementation described in \ref{['sec:Gaussian-KL']} with respect to scale. The COIL-20 dataset is used here and an embedding is generated from UMAP with the default parameters. The random_state and n_jobs parameters in UMAP are set to 42 and 1 respectively for reproducibility.
• Figure 5: Each small multiple shows the results of the sensitivity experiment for the Fashion-MNIST dataset across different algorithms. Normalized Stress is shown on top and all scale-invariant measures on the bottom. From a given embedding, isotropic Gaussian noise is repeatedly applied, and at each iteration the embedding is rescaled by a random number between 0.1 and 10. The plots show how each metric behaves as the embedding quality degrades. Normalized stress is plotted separately and at a log scale as the magnitude of changes vary drastically. Note, the values of the metrics should not be compared, only the behavior of the curves. SGS is inverted so that lower is better. Further plots can be found in supplemental material. We observe that all scale-invariant metrics behave predictably (stress increases when more noise is added) while the scale-sensitive normalized stress is unpredictable and inconsistent.