Intrinsic Dimension Estimating Autoencoder (IDEA) Using CancelOut Layer and a Projected Loss

TL;DR

This work tackles intrinsic-dimension estimation for high-dimensional data that lie on linear or nonlinear manifolds and pairs it with faithful reconstruction from a reduced latent space. It introduces IDEA, a neural autoencoder with a re-weighted double CancelOut bottleneck and a projected reconstruction loss that enforces latent sparsity without sacrificing accuracy. Across synthetic Legendre-based datasets and benchmark manifolds, IDEA accurately identifies the true intrinsic dimension $d$ and reconstructs data using a compact latent space, often outperforming standard estimators and baselines. The method is further demonstrated on vertically resolved free-surface flow data, where IDEA achieves a low-dimensional, physically interpretable representation that competes with, and can surpass, traditional POD approaches in efficiency and interpretability.

Abstract

This paper introduces the Intrinsic Dimension Estimating Autoencoder (IDEA), which identifies the underlying intrinsic dimension of a wide range of datasets whose samples lie on either linear or nonlinear manifolds. Beyond estimating the intrinsic dimension, IDEA is also able to reconstruct the original dataset after projecting it onto the corresponding latent space, which is structured using re-weighted double CancelOut layers. Our key contribution is the introduction of the projected reconstruction loss term, guiding the training of the model by continuously assessing the reconstruction quality under the removal of an additional latent dimension. We first assess the performance of IDEA on a series of theoretical benchmarks to validate its robustness. These experiments allow us to test its reconstruction ability and compare its performance with state-of-the-art intrinsic dimension estimators. The benchmarks show good accuracy and high versatility of our approach. Subsequently, we apply our model to data generated from the numerical solution of a vertically resolved one-dimensional free-surface flow, following a pointwise discretization of the vertical velocity profile in the horizontal direction, vertical direction, and time. IDEA succeeds in estimating the dataset's intrinsic dimension and then reconstructs the original solution by working directly within the projection space identified by the network.

Figures (14)

• Figure 1: Example datasets where the intrinsic dimension of the sampling space differs from its embedded dimension.
• Figure 2: IDEA's architecture consisting of an encoder, a re-weighted double CancelOut (Co) bottleneck, and a decoder. Grey rectangles represent a block composed of a normalization layer followed by a SiLU activation. The layer sizes are set as follows: $128 \rightarrow 64 \rightarrow 32 \rightarrow 16 \rightarrow {\color[HTML]{5395e4}l \rightarrow l} \color{black} \rightarrow 16 \rightarrow 32 \rightarrow 64 \rightarrow 128$.
• Figure 3: Re-weighted double CancelOut (Co) layers forming the bottleneck. The initial latent dimension is $l$. Weights are represented as arrows, variables as hexagons. Co$_1$ sets the last weight to zero, effectively removing one dimension from the latent space, while also beginning to regularize the current last non-zero weight (blue), diminishing the magnitude of the corresponding latent variable. In response, Co$_2$ increases its corresponding weight (red) to ensure that the remaining $l-1$ variables stay on a comparable scale after the bottleneck. In the diagram, the weight regularization process unfolds from top to bottom.
• Figure 4: Synthetic dataset 3 total train loss $\mathcal{L}_{\text{total}}$ as a function of training epoch.
• Figure 5: Correlation matrix of dataset 3, between the four latent variables $L_i$ and the projection coefficients on the scaled Legendre polynomial basis $\alpha_i$ (\ref{['eq:synthpol']}). The color varies according to the absolute value of the correlation between the parameters.