Gauge-invariant representation holonomy

TL;DR

Representation holonomy introduces a gauge-invariant, pathwise diagnostic for learned feature geometry by composing local rotation transports around input-space loops: $H( ext{``} ext{gamma} ext{''})=\prod_i R_i$ with $R_i\in\text{SO}(p)$ and $h_{ ext{norm}}(\gamma)=\|H(\gamma)-I\|_F/(2\sqrt{p})$. A practical estimator uses global whitening, shared-neighborhood centers, and rotation-projected Procrustes in a low-dimensional subspace to obtain $\widehat{H}$ and $\widehat{h}_{\text{norm}}$, with formal invariances to orthogonal and affine reparameterizations and a linear null for affine maps. The paper proves small-radius linear scaling $h_{ ext{norm}}(\gamma_r)=O(r)$, provides finite-sample error bounds for per-edge transports, and demonstrates that holonomy captures path-dependent geometry unseen by pointwise measures like CKA; experiments on MNIST and CIFAR-10 show holonomy increases with loop radius and depth, correlates with adversarial and corruption robustness, and tracks training dynamics. Overall, holonomy offers a practical, scalable diagnostic that complements existing representational similarity metrics by revealing curvature-like structure in feature spaces relevant to robustness and generalization.

Abstract

Deep networks learn internal representations whose geometry--how features bend, rotate, and evolve--affects both generalization and robustness. Existing similarity measures such as CKA or SVCCA capture pointwise overlap between activation sets, but miss how representations change along input paths. Two models may appear nearly identical under these metrics yet respond very differently to perturbations or adversarial stress. We introduce representation holonomy, a gauge-invariant statistic that measures this path dependence. Conceptually, holonomy quantifies the "twist" accumulated when features are parallel-transported around a small loop in input space: flat representations yield zero holonomy, while nonzero values reveal hidden curvature. Our estimator fixes gauge through global whitening, aligns neighborhoods using shared subspaces and rotation-only Procrustes, and embeds the result back to the full feature space. We prove invariance to orthogonal (and affine, post-whitening) transformations, establish a linear null for affine layers, and show that holonomy vanishes at small radii. Empirically, holonomy increases with loop radius, separates models that appear similar under CKA, and correlates with adversarial and corruption robustness. It also tracks training dynamics as features form and stabilize. Together, these results position representation holonomy as a practical and scalable diagnostic for probing the geometric structure of learned representations beyond pointwise similarity.

Figures (11)

• Figure 1: Holonomy as path-dependent feature rotation. (A) A small closed loop $\gamma=(x_0,\dots,x_{L-1},x_L{=}x_0)$ in a 2D input slice. (B) The corresponding features $z_i = z(x_i)$ and their local neighbourhoods $\mathcal{N}(z_i)$; for each edge we estimate an orthogonal transport $R_{i,i+1}$ that best aligns the two nearby feature clouds. (C) Composing these transports around the loop yields the holonomy $H = R_{L-1}\cdots R_{1}R_{0}$, visualised as the net rotation of a reference direction by angle $\theta$. Holonomy is invariant to layer-wise gauge changes (global change of feature basis) and measures how much the representation "twists" when inputs follow a loop, rather than just how similar activations are at individual points.
• Figure 2: Holonomy vs. radius on MNIST and CIFAR-10. Mean $\pm$95% CI across seeds (MNIST) and across seeds and training regimes (CIFAR-10). Both datasets exhibit positive dependence on radius; on MNIST the deeper layer has larger amplitudes, while on CIFAR-10 the first two layers are very similar in magnitude.
• Figure 3: Small-radius regime (left) on CIFAR-10 (ResNet-18, layer2). Points show mean $h_{\mathrm{norm}}$ over seeds and training regimes; error bars are s.e.m. Slope estimate: $1.44{\times}10^{-7}$ (95% CI $[-1.07{\times}10^{-7},\, 4.22{\times}10^{-7}]$). Self-loop bias (right) near zero (MNIST Hidden 1, $r\!\approx\!10^{-4}$). The bias floor is $\mathcal{O}(10^{-8})$.
• Figure 4: Reliability/stability. Left: $h_{\mathrm{norm}}$ is nearly flat as $N_{\text{pool}}$ increases. Right: PCA planes yield slightly higher, more geometry-aware holonomy than random planes.
• Figure 5: A single MNIST digit is translated around a small 4-step loop. At conv2, the nearly translation-equivariant CNN yields an almost closed feature loop and tiny holonomy, while the aliased CNN produces a distorted loop and holonomy about three orders of magnitude larger.

Theorems & Definitions (18)

• Proposition A.1: Gauge invariance under orthogonal reparameterizations; full proof
• proof
• Proposition A.2: Affine invariance after global whitening; full proof
• proof
• Proposition A.3: Linear null; full proof
• proof
• Proposition A.4: Orientation, reparametrization, and normalization; full proof
• proof
• Theorem A.1: Small-radius limit
• proof