"Normalized Stress" is Not Normalized: How to Interpret Stress Correctly

TL;DR

The paper tackles the problem that normalized stress, a common DR evaluation metric, is not invariant to uniform scaling of the embedding, which can misrank dimension-reduction methods. Through analytic and empirical analyses across MDS, t-SNE, and random embeddings, the authors demonstrate how scale can distort stress-based assessments. They propose scale-normalized stress (SNS) as a straightforward, scale-invariant alternative, with an explicit optimal scaling factor $\alpha=\dfrac{\sum_{i,j}[\Delta^n(x_i,x_j)\Delta^t(p_i,p_j)]}{\sum_{i,j}[\Delta^t(p_i,p_j)^2]}$ to minimize NS, and show how to compute it efficiently. Empirical results on diverse datasets show that SNS and other scale-invariant metrics align with the expected ordering of DR methods and are robust to projection size, prompting a reevaluation of prior studies that relied on scale-sensitive metrics.

Abstract

Stress is among the most commonly employed quality metrics and optimization criteria for dimension reduction projections of high dimensional data. Complex, high dimensional data is ubiquitous across many scientific disciplines, including machine learning, biology, and the social sciences. One of the primary methods of visualizing these datasets is with two dimensional scatter plots that visually capture some properties of the data. Because visually determining the accuracy of these plots is challenging, researchers often use quality metrics to measure projection accuracy or faithfulness to the full data. One of the most commonly employed metrics, normalized stress, is sensitive to uniform scaling of the projection, despite this act not meaningfully changing anything about the projection. We investigate the effect of scaling on stress and other distance based quality metrics analytically and empirically by showing just how much the values change and how this affects dimension reduction technique evaluations. We introduce a simple technique to make normalized stress scale invariant and show that it accurately captures expected behavior on a small benchmark.

Figures (4)

• Figure 1: The $Palmer\text{ }Penguins$ dataset captured by MDS, with species encoded by color and sex encoded by shape. Projections are scaled (by $0.5$, $1$, $2$) and the value of the metric changes at the different scales.
• Figure 2: Plots of the scale-sensitive metrics as one varies the size of the embedding space. Each curve is derived from the $S\text{-}curve$ dataset embedded using MDS. The minimum point has been marked on both curves.
• Figure 3: Embedding and Shepard diagram with monotonic fitted line for the $Wine$ dataset captured by MDS.
• Figure 4: Embedding and Shepard diagram with monotonic fitted line for the $Wine$ dataset captured by RND.