Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Information-Theoretic Quality Metric of Low-Dimensional Embeddings

Sebastián Gutiérrez-Bernal, Hector Medel Cobaxin, Abiel Galindo González

TL;DR

The paper tackles the problem of evaluating low-dimensional embeddings from an explicit information-theoretic perspective, arguing that traditional distance- and geometry-focused metrics miss how much information is preserved. It introduces the Entropy Rank Preservation Measure (ERPM), which uses the Shannon entropy of local spectral directions and the stable rank to quantify changes in uncertainty during dimensionality reduction, delivering local and global indicators. Through experiments on the S-curve and financial time-series embeddings, ERPM reveals substantial local information losses that are not always captured by existing metrics like MRRE or Local Procrustes, while also showing strong but nuanced correlations with geometric criteria. The work demonstrates that combining ERPM with traditional metrics provides a more comprehensive, practical framework for assessing embeddings in applications where information content is critical, such as early-warning indicator construction for regime shifts.

Abstract

In this work we study the quality of low-dimensional embeddings from an explicitly information-theoretic perspective. We begin by noting that classical evaluation metrics such as stress, rank-based neighborhood criteria, or Local Procrustes quantify distortions in distances or in local geometries, but do not directly assess how much information is preserved when projecting high-dimensional data onto a lower-dimensional space. To address this limitation, we introduce the Entropy Rank Preservation Measure (ERPM), a local metric based on the Shannon entropy of the singular-value spectrum of neighborhood matrices and on the stable rank, which quantifies changes in uncertainty between the original representation and its reduced projection, providing neighborhood-level indicators and a global summary statistic. To validate the results of the metric, we compare its outcomes with the Mean Relative Rank Error (MRRE), which is distance-based, and with Local Procrustes, which is based on geometric properties, using a financial time series and a manifold commonly studied in the literature. We observe that distance-based criteria exhibit very low correlation with geometric and spectral measures, while ERPM and Local Procrustes show strong average correlation but display significant discrepancies in local regimes, leading to the conclusion that ERPM complements existing metrics by identifying neighborhoods with severe information loss, thereby enabling a more comprehensive assessment of embeddings, particularly in information-sensitive applications such as the construction of early-warning indicators.

Information-Theoretic Quality Metric of Low-Dimensional Embeddings

TL;DR

The paper tackles the problem of evaluating low-dimensional embeddings from an explicit information-theoretic perspective, arguing that traditional distance- and geometry-focused metrics miss how much information is preserved. It introduces the Entropy Rank Preservation Measure (ERPM), which uses the Shannon entropy of local spectral directions and the stable rank to quantify changes in uncertainty during dimensionality reduction, delivering local and global indicators. Through experiments on the S-curve and financial time-series embeddings, ERPM reveals substantial local information losses that are not always captured by existing metrics like MRRE or Local Procrustes, while also showing strong but nuanced correlations with geometric criteria. The work demonstrates that combining ERPM with traditional metrics provides a more comprehensive, practical framework for assessing embeddings in applications where information content is critical, such as early-warning indicator construction for regime shifts.

Abstract

In this work we study the quality of low-dimensional embeddings from an explicitly information-theoretic perspective. We begin by noting that classical evaluation metrics such as stress, rank-based neighborhood criteria, or Local Procrustes quantify distortions in distances or in local geometries, but do not directly assess how much information is preserved when projecting high-dimensional data onto a lower-dimensional space. To address this limitation, we introduce the Entropy Rank Preservation Measure (ERPM), a local metric based on the Shannon entropy of the singular-value spectrum of neighborhood matrices and on the stable rank, which quantifies changes in uncertainty between the original representation and its reduced projection, providing neighborhood-level indicators and a global summary statistic. To validate the results of the metric, we compare its outcomes with the Mean Relative Rank Error (MRRE), which is distance-based, and with Local Procrustes, which is based on geometric properties, using a financial time series and a manifold commonly studied in the literature. We observe that distance-based criteria exhibit very low correlation with geometric and spectral measures, while ERPM and Local Procrustes show strong average correlation but display significant discrepancies in local regimes, leading to the conclusion that ERPM complements existing metrics by identifying neighborhoods with severe information loss, thereby enabling a more comprehensive assessment of embeddings, particularly in information-sensitive applications such as the construction of early-warning indicators.
Paper Structure (14 sections, 21 equations, 6 figures)

This paper contains 14 sections, 21 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Case of study: Information-based manifold learning for early warnings
  3. Background and Related Work
  4. Manifold Learning Methods
  5. Quality Metrics
  6. Information in terms of spectrum
  7. Stable Rank as a measure of spectrum
  8. Entropy Rank Preservation Measure (ERPM)
  9. Experimental Setup
  10. Data, reduction methods, benchmarks and hyperparameters
  11. Results
  12. Global Correlation with Benchmarks
  13. Local Analysis from a Neighborhood Perspective
  14. Discussion

Figures (6)

  • Figure 1: Visualization of the data used in the metric evaluation.
  • Figure 2: Correlation plot between quality metrics.
  • Figure 3: Metrics comparison for the manifold dataset.
  • Figure 4: Metrics comparison for the financial dataset.
  • Figure 5: Manifold data jointplot of the results of Local Procrustes (y-axis of the scatter plot located in the center and density plot on the right) and of ERPM (x-axis of the scatter plot located in the center and density plot at the top).
  • ...and 1 more figures