Geometric Stability: The Missing Axis of Representations

TL;DR

Geometric stability complements traditional similarity in representational analysis by quantifying how reliably a representation preserves its internal geometry under perturbations. The Shesha framework, with FS and SS variants and supervised options, measures internal consistency of Representational Dissimilarity Matrices to detect drift, safety-relevant changes, and controllability signals without relying on external references. Across 2,463 encoder configurations in seven domains, stability and similarity are empirically uncorrelated, and stability remains sensitive to fine-grained manifold structure that similarity metrics miss. In steering, drift detection, CRISPR perturbations, and neuroscience, Shesha predicts functional outcomes and drift earlier and more robustly than traditional similarity metrics, providing a practical, task-agnostic geometric audit for high-dimensional representations. The work also identifies a geometric tax where optimizing for transferability can reduce stability, underscoring the need for a dual-axis evaluation in model development and deployment.

Abstract

Analysis of learned representations has a blind spot: it focuses on $similarity$, measuring how closely embeddings align with external references, but similarity reveals only what is represented, not whether that structure is robust. We introduce $geometric$ $stability$, a distinct dimension that quantifies how reliably representational geometry holds under perturbation, and present $Shesha$, a framework for measuring it. Across 2,463 configurations in seven domains, we show that stability and similarity are empirically uncorrelated ($ρ\approx 0.01$) and mechanistically distinct: similarity metrics collapse after removing the top principal components, while stability retains sensitivity to fine-grained manifold structure. This distinction yields actionable insights: for safety monitoring, stability acts as a functional geometric canary, detecting structural drift nearly 2$\times$ more sensitively than CKA while filtering out the non-functional noise that triggers false alarms in rigid distance metrics; for controllability, supervised stability predicts linear steerability ($ρ= 0.89$-$0.96$); for model selection, stability dissociates from transferability, revealing a geometric tax that transfer optimization incurs. Beyond machine learning, stability predicts CRISPR perturbation coherence and neural-behavioral coupling. By quantifying $how$ $reliably$ systems maintain structure, geometric stability provides a necessary complement to similarity for auditing representations across biological and computational systems.

Figures (38)

• Figure 1: Stability and similarity are independent dimensions of representational geometry.(a) Spectral Sensitivity: CKA (red) collapses after removing just the single top principal component, while Shesha (blue) retains sensitivity to the spectral tail. CKA measures dominant variance; Shesha measures full manifold geometry. (b) Universality: Across 2,463 encoder configurations spanning seven domains, Shesha and CKA show negligible net correlation ($\rho = -0.01$, 95% CI $[-0.06, +0.03]$), confirming they capture distinct geometric properties. (c) Regime Analysis: Aggregate near-zero correlation emerges from opposing effects: random transformations yield positive correlation ($\rho = +0.76$), while PCA compression yields negative correlation ($\rho = -0.47$). These cancel in aggregate, revealing that Shesha specifically detects compression-induced damage invisible to CKA.
• Figure 2: Metric Convergence. Shesha estimates remain stable as sample size increases from 400 to 1600 across representative architectures. The flat trajectories confirm rapid convergence and numerical reliability at modest sample sizes.
• Figure 3: Model Leaderboard. Ranking of 15 architectures by Shesha score (feature split). Bar segments show contributions from CIFAR-10 (teal) and CIFAR-100 (blue). Modern architectures with attention or dense connectivity achieve higher geometric stability.
• Figure 4: Seed Stability. Comparison of Shesha scores computed with two different random seeds (Seed A$=$100 vs. Seed B$=$200). Points align closely with the diagonal identity line, indicating high reproducibility across random initializations.
• Figure 5: Spectral Sensitivity Analysis. We measure metric responses as the top $k$ principal components are progressively removed from a power-law representation. (A) Shesha degrades gracefully while all similarity metrics (CKA, PWCKA, Procrustes) collapse after removing just 1 PC. (B) Comparison with whitened Shesha shows high correlation ($\rho=0.999$), though whitening reduces baseline stability. (C) Shesha robustness across preprocessing conditions (raw, centered, normalized, whitened). (D) CKA behavior across preprocessing; notably, whitening causes CKA to recover sensitivity by equalizing the spectrum.