SEIS: Subspace-based Equivariance and Invariance Scores for Neural Representations

TL;DR

SEIS introduces a subspace-based metric to quantify how neural representations preserve geometric structure under transformations, explicitly disentangling equivariance and invariance without labels. It uses a spatially-aware matricization and SVD denoising, followed by Canonical Correlation Analysis to measure alignment between original and transformed activations. Across experiments, early layers exhibit strong equivariance while deeper layers become more invariant; data augmentation amplifies invariance while preserving equivariance, and multi-task learning enhances both properties in shared encoders, with skip connections helping restore equivariance during decoding. This framework enables layer-wise, transformation-agnostic diagnostics of geometric stability with practical implications for designing symmetry-aware architectures and training regimes.

Abstract

Understanding how neural representations respond to geometric transformations is essential for evaluating whether learned features preserve meaningful spatial structure. Existing approaches primarily assess robustness by comparing model outputs under transformed inputs, offering limited insight into how geometric information is organized within internal representations and failing to distinguish between information loss and re-encoding. In this work, we introduce SEIS (Subspace-based Equivariance and Invariance Scores), a subspace metric for analyzing layer-wise feature representations under geometric transformations, disentangling equivariance from invariance without requiring labels or explicit knowledge of the transformation. Synthetic validation confirms that SEIS correctly recovers known transformations. Applied to trained classification networks, SEIS reveals a transition from equivariance in early layers to invariance in deeper layers, and that data augmentation increases invariance while preserving equivariance. We further show that multi-task learning induces synergistic gains in both properties at the shared encoder, and skip connections restore equivariance lost during decoding.

Figures (4)

• Figure 1: Validation of SEIS on MNIST activations using controlled transformations. The identity case (left) yields high scores for both metrics, geometric transformations (middle) preserve equivariance but reduce invariance, and the random baseline (right) produces negligible scores.
• Figure 2: Depth-wise equivariance ($\mathcal{S}_\text{equiv}$) (left panel) and invariance ($\mathcal{S}_\text{inv}$) (right panel) scores across training epochs for ResNet-18 on CIFAR-100. Both properties stabilize early and exhibit a clear depth-dependent pattern.
• Figure 3: Effect of affine data augmentation on equivariance and invariance across network depth for ResNet-18 on CIFAR-100. Augmentation increases invariance while preserving equivariance in deeper layers.
• Figure 4: Left panel: Equivariance ($\mathcal{S}_\text{equiv}$) versus invariance ($\mathcal{S}_\text{inv}$) at the bottleneck layer for Classification (Cls), Segmentation (Seg), and Multi-Task Learning (MTL). Right panel: Equivariance trajectories across decoder depth, comparing FCN and U-Net decoders under single-task (Seg) and multi-task objectives.