eDCF: Estimating Intrinsic Dimension using Local Connectivity

TL;DR

This paper introduces eDCF, a scalable intrinsic-dimension estimator based on a grid-defined local connectivity framework. It combines Information Percentage (IP) for adaptive grid spacing with a Connectivity Factor (CF) and its distributed version (DCF) to bound and estimate dimension, further enhanced by an empirical reference model to form eDCF. The method yields competitive mean absolute error with superior exact-dimension recovery, especially at moderate-to-high noise and large sample sizes, and demonstrates capability to detect fractal-like structures in decision boundaries. The approach is designed for parallel computation, enabling efficient analysis of high-dimensional datasets, and provides a principled, topology-aware alternative to distance-based estimators. Limitations include discretization sensitivity and calibration reliance, with future work targeting multi-resolution schemes, uncertainty quantification, and broader real-world validation.

Abstract

Modern datasets often contain high-dimensional features exhibiting complex dependencies. To effectively analyze such data, dimensionality reduction methods rely on estimating the dataset's intrinsic dimension (id) as a measure of its underlying complexity. However, estimating id is challenging due to its dependence on scale: at very fine scales, noise inflates id estimates, while at coarser scales, estimates stabilize to lower, scale-invariant values. This paper introduces a novel, scalable, and parallelizable method called eDCF, which is based on Connectivity Factor (CF), a local connectivity-based metric, to robustly estimate intrinsic dimension across varying scales. Our method consistently matches leading estimators, achieving comparable values of mean absolute error (MAE) on synthetic benchmarks with noisy samples. Moreover, our approach also attains higher exact intrinsic dimension match rates, reaching up to 25.0% compared to 16.7% for MLE and 12.5% for TWO-NN, particularly excelling under medium to high noise levels and large datasets. Further, we showcase our method's ability to accurately detect fractal geometries in decision boundaries, confirming its utility for analyzing realistic, structured data.

Figures (17)

• Figure 1: Ranges and potential overlaps of the Lower (L), Middle (M), and Upper (U) Connectivity Factor ($\mathcal{CF}$) values for various topological dimensions within an $n$-dimensional ambient space (wher $K_i$ denotes the $K$ bound (L(lower), M(middle), U(upper)) for embedded dimension $i$), for $n$ = 0, 1, 2 and 3.
• Figure 2: Performance comparison of TWO-NN, MLE, and $e\mathcal{DCF}$ on benchmark manifolds under varying noise conditions. (a) represents Mean Absolute Error for 1% noise, (b) represents Mean Signed Error for 1% noise, (c) represents Accuracy for 1% noise, (d) represents Mean Absolute Error for 10% noise, (e) represents Mean Signed Error for 10% noise, (f) represents Accuracy for 10% noise, (g) represents Mean Absolute Error for 30% noise, (h) represents Mean Signed Error for 30% noise, and (i) represents Accuracy for 30% noise. All metrics are plotted against the number of points per manifold on a logarithmic scale.
• Figure 3: Plots of KNN on CCD Dataset
• Figure 4: Plots of Decision Tree on OCCD Dataset
• Figure 5: Plots of KNN on BF Dataset