Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gauge Freedom and Metric Dependence in Neural Representation Spaces

Jericho Cain

TL;DR

Experiments on multilayer perceptrons and convolutional networks confirm that inserting invertible transformations into trained models can substantially distort cosine similarity and nearest-neighbor structure while leaving predictions unchanged.

Abstract

Neural network representations are often analyzed as vectors in a fixed Euclidean space. However, their coordinates are not uniquely defined. If a hidden representation is transformed by an invertible linear map, the network function can be preserved by applying the inverse transformation to downstream weights. Representations are therefore defined only up to invertible linear transformations. We study neural representation spaces from this geometric viewpoint and treat them as vector spaces with a gauge freedom under the general linear group. Within this framework, commonly used similarity measures such as cosine similarity become metric-dependent quantities whose values can change under coordinate transformations that leave the model function unchanged. This provides a common interpretation for several observations in the literature, including cosine-similarity instability, anisotropy in embedding spaces, and the appeal of representation comparison methods such as SVCCA and CKA. Experiments on multilayer perceptrons and convolutional networks confirm that inserting invertible transformations into trained models can substantially distort cosine similarity and nearest-neighbor structure while leaving predictions unchanged. These results indicate that analysis of neural representations should focus either on quantities that are invariant under this gauge freedom or on explicitly chosen canonical coordinates.

Gauge Freedom and Metric Dependence in Neural Representation Spaces

TL;DR

Experiments on multilayer perceptrons and convolutional networks confirm that inserting invertible transformations into trained models can substantially distort cosine similarity and nearest-neighbor structure while leaving predictions unchanged.

Abstract

Neural network representations are often analyzed as vectors in a fixed Euclidean space. However, their coordinates are not uniquely defined. If a hidden representation is transformed by an invertible linear map, the network function can be preserved by applying the inverse transformation to downstream weights. Representations are therefore defined only up to invertible linear transformations. We study neural representation spaces from this geometric viewpoint and treat them as vector spaces with a gauge freedom under the general linear group. Within this framework, commonly used similarity measures such as cosine similarity become metric-dependent quantities whose values can change under coordinate transformations that leave the model function unchanged. This provides a common interpretation for several observations in the literature, including cosine-similarity instability, anisotropy in embedding spaces, and the appeal of representation comparison methods such as SVCCA and CKA. Experiments on multilayer perceptrons and convolutional networks confirm that inserting invertible transformations into trained models can substantially distort cosine similarity and nearest-neighbor structure while leaving predictions unchanged. These results indicate that analysis of neural representations should focus either on quantities that are invariant under this gauge freedom or on explicitly chosen canonical coordinates.
Paper Structure (22 sections, 1 theorem, 53 equations, 5 figures)

This paper contains 22 sections, 1 theorem, 53 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Representation Spaces
  3. Gauge Freedom of Representations
  4. Metric Structure and Similarity Measures
  5. Whitening as a Canonical Gauge
  6. Implications for Embedding Similarity
  7. Feature Directions and Subspaces
  8. Feature Packing and Superposition
  9. Gauge Transformations of Feature Bases
  10. Representation Similarity Across Networks
  11. Representation Dynamics and Local Geometry
  12. Induced Metric on Parameter Space
  13. Interpretation for Training Dynamics
  14. Gauge Dependence
  15. Experiments
  16. ...and 7 more sections

Key Result

Proposition 1

Let $h(x)\in\mathbb{R}^d$ be a hidden representation and suppose a subsequent linear layer computes $y = W h(x)$. For any invertible matrix $D\in\mathrm{GL}(d)$, define Then so the network function remains unchanged. Thus the coordinates of representation space are defined only up to the action of $\mathrm{GL}(d)$.

Figures (5)

  • Figure 1: A linear distortion $\tilde{x} = D x$ maps the Euclidean unit sphere to an ellipsoid. The induced inner product is $G=D^\top D$, so cosine similarity computed after the transformation corresponds to angular similarity in the distorted metric.
  • Figure 2: Pairwise cosine similarities before and after a gauge transformation in the Digits experiment. The model predictions remain identical, but cosine similarities between representations change substantially.
  • Figure 3: Cosine distortion for CIFAR-10 representations under a gauge transformation. Predictions remain unchanged, but pairwise cosine similarity between hidden states is altered.
  • Figure 4: Effect of gauge strength on cosine similarity and nearest-neighbor structure in CIFAR-10 representations. Increasing the condition number $\kappa$ of the transformation produces larger cosine distortions and reduces neighbor stability, even though the network function remains unchanged.
  • Figure 5: Eigenvalue spectrum of the representation covariance matrix before and after whitening (log scale). Whitening applies the transformation $D=\Sigma^{-1/2}$, which fixes a canonical gauge in which the covariance becomes the identity. After whitening the spectrum collapses to $\lambda \approx 1$, indicating that second-order anisotropy in the representation distribution has been removed.

Theorems & Definitions (1)

  • Proposition 1: Representation gauge symmetry