A Connection Between Score Matching and Local Intrinsic Dimension

TL;DR

The paper addresses the challenge of estimating the local intrinsic dimension (LID) of high-dimensional data by connecting LID to score-based learning. It proves that the denoising score matching loss ($ ext{L}_ ext{DSM}$) lower-bounds the data manifold dimension $d$ (the LID) in the small-noise limit, and shows that the implicit score matching loss and the FLIPD estimator are closely tied to this bound through their relations to the normal dimension and error-bundle constructions. It also relates these results to existing parametric estimators like NB and LIDL, and demonstrates empirical efficiency and accuracy gains using diffusion-transformer architectures on manifold benchmarks and real diffusion models (Stable Diffusion 3.5/2). The findings indicate that DSM is a scalable, memory-efficient LID estimator that remains robust under model quantization and large problem sizes, with practical implications for model evaluation and geometry-aware learning.

Abstract

The local intrinsic dimension (LID) of data is a fundamental quantity in signal processing and learning theory, but quantifying the LID of high-dimensional, complex data has been a historically challenging task. Recent works have discovered that diffusion models capture the LID of data through the spectra of their score estimates and through the rate of change of their density estimates under various noise perturbations. While these methods can accurately quantify LID, they require either many forward passes of the diffusion model or use of gradient computation, limiting their applicability in compute- and memory-constrained scenarios. We show that the LID is a lower bound on the denoising score matching loss, motivating use of the denoising score matching loss as a LID estimator. Moreover, we show that the equivalent implicit score matching loss also approximates LID via the normal dimension and is closely related to a recent LID estimator, FLIPD. Our experiments on a manifold benchmark and with Stable Diffusion 3.5 indicate that the denoising score matching loss is a highly competitive and scalable LID estimator, achieving superior accuracy and memory footprint under increasing problem size and quantization level.

Figures (5)

• Figure 1: Conceptual depiction of the denoising loss as a LID estimator (left) and a conceptual link between the denoising loss and the error bundle (EB) method.
• Figure 2: Comparison of LID estimation mean absolute error (left) and peak memory usage (right) of LID estimators as ambient dimension and true LID of a HyperSphere manifold increase.
• Figure 3: FLIPD and $\mathcal{L}_\text{DSM}$ LID estimates for 100 $256\times256$ images using SD3.5-medium. The LID estimates are highly correlated, with $\mathcal{L}_\text{DSM}$ providing lower estimates on average.
• Figure 4: Comparison of LID prediction disagreement (MAE) and peak GPU memory usage (GB) of SD-3.5-medium at various quantization levels.
• Figure 5: FLIPD and $\mathcal{L}_\text{DSM}$ LID estimates for $512\times512$ images using Stable Diffusion 2 (SD2). The LID estimates are highly correlated, with $\mathcal{L}_\text{DSM}$ providing lower estimates at low noise levels.

Theorems & Definitions (6)

• Theorem 3.1: Denoising Score Matching Loss Lower Bound
• Remark 3.2: Stratified Manifolds and LID
• Theorem 3.3: Implicit Score Matching Loss Lower Bound
• proof
• Lemma A.1: Mean Squared Error Bound
• proof