Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Homogeneity Trap: Spectral Collapse in Doubly-Stochastic Deep Networks

Yizhi Liu

TL;DR

This work identifies a fundamental Homogeneity Trap in doubly-stochastic neural networks, where projecting mixing operators onto the DSM set biases toward a uniform matrix and drives the subdominant singular value $σ_2$ to near zero. By deriving finite-$n$ bounds and using concentration and perturbation results, the authors show Layer Normalization cannot restore geometry when the spectral SNR $γ$ is below a critical threshold, leading to identity stagnation in residual blocks and shallow effective depth. The study provides both theoretical mechanisms and empirical validations (e.g., $σ_2$ shrinking with entropy, orthogonal collapse at low $γ$, and affine LN ablations) to demonstrate the trade-off between entropic stability and spectral expressivity, with implications for DSM-based layers and potential non-DSM alternatives. Overall, the paper highlights a key constraint on architectural design: stabilizing projections can irreversibly filter out high-frequency structure, motivating exploration of parameterizations beyond strict DSM constraints for deep, expressive models.

Abstract

Doubly-stochastic matrices (DSM) are increasingly utilized in structure-preserving deep architectures -- such as Optimal Transport layers and Sinkhorn-based attention -- to enforce numerical stability and probabilistic interpretability. In this work, we identify a critical spectral degradation phenomenon inherent to these constraints, termed the Homogeneity Trap. We demonstrate that the maximum-entropy bias, typical of Sinkhorn-based projections, drives the mixing operator towards the uniform barycenter, thereby suppressing the subdominant singular value σ_2 and filtering out high-frequency feature components. We derive a spectral bound linking σ_2 to the network's effective depth, showing that high-entropy constraints restrict feature transformation to a shallow effective receptive field. Furthermore, we formally demonstrate that Layer Normalization fails to mitigate this collapse in noise-dominated regimes; specifically, when spectral filtering degrades the Signal-to-Noise Ratio (SNR) below a critical threshold, geometric structure is irreversibly lost to noise-induced orthogonal collapse. Our findings highlight a fundamental trade-off between entropic stability and spectral expressivity in DSM-constrained networks.

The Homogeneity Trap: Spectral Collapse in Doubly-Stochastic Deep Networks

TL;DR

This work identifies a fundamental Homogeneity Trap in doubly-stochastic neural networks, where projecting mixing operators onto the DSM set biases toward a uniform matrix and drives the subdominant singular value to near zero. By deriving finite- bounds and using concentration and perturbation results, the authors show Layer Normalization cannot restore geometry when the spectral SNR is below a critical threshold, leading to identity stagnation in residual blocks and shallow effective depth. The study provides both theoretical mechanisms and empirical validations (e.g., shrinking with entropy, orthogonal collapse at low , and affine LN ablations) to demonstrate the trade-off between entropic stability and spectral expressivity, with implications for DSM-based layers and potential non-DSM alternatives. Overall, the paper highlights a key constraint on architectural design: stabilizing projections can irreversibly filter out high-frequency structure, motivating exploration of parameterizations beyond strict DSM constraints for deep, expressive models.

Abstract

Doubly-stochastic matrices (DSM) are increasingly utilized in structure-preserving deep architectures -- such as Optimal Transport layers and Sinkhorn-based attention -- to enforce numerical stability and probabilistic interpretability. In this work, we identify a critical spectral degradation phenomenon inherent to these constraints, termed the Homogeneity Trap. We demonstrate that the maximum-entropy bias, typical of Sinkhorn-based projections, drives the mixing operator towards the uniform barycenter, thereby suppressing the subdominant singular value σ_2 and filtering out high-frequency feature components. We derive a spectral bound linking σ_2 to the network's effective depth, showing that high-entropy constraints restrict feature transformation to a shallow effective receptive field. Furthermore, we formally demonstrate that Layer Normalization fails to mitigate this collapse in noise-dominated regimes; specifically, when spectral filtering degrades the Signal-to-Noise Ratio (SNR) below a critical threshold, geometric structure is irreversibly lost to noise-induced orthogonal collapse. Our findings highlight a fundamental trade-off between entropic stability and spectral expressivity in DSM-constrained networks.
Paper Structure (22 sections, 7 theorems, 9 equations, 2 figures, 1 algorithm)

This paper contains 22 sections, 7 theorems, 9 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Our Contributions.
  3. Preliminaries and Assumptions
  4. Technical Assumptions
  5. Theoretical Analysis of Spectral Collapse
  6. Detail Subspace Dynamics
  7. Finite-$n$ Collapse via SNR Failure
  8. Effective Depth and Non-normality
  9. Practical check for transient growth.
  10. Extension to Residual and Non-linear Dynamics
  11. Mutual Incompatibility of Stability and Depth
  12. Broader Context: Affine Parameters
  13. Simulation Details
  14. Exp 1: Spectral Gap vs. Entropy.
  15. Exp 2: Orthogonal Collapse.
  16. ...and 7 more sections

Key Result

Lemma 1

If $\boldsymbol{\xi} \sim \mathcal{N}(\mathbf{0}, \sigma^2 I_n)$, then the normalized projection $\mathbf{u}_{\xi} = \frac{\mathcal{P}_{\perp} \boldsymbol{\xi}}{\|\mathcal{P}_{\perp} \boldsymbol{\xi}\|}$ is uniformly distributed on the unit sphere $\mathbb{S}^{n-2}$ within $\mathcal{V}_{\perp}$.

Figures (2)

  • Figure 1: Verification of the Trap. Mean subdominant singular value $\sigma_2$ vs Sinkhorn temperature $T$. (PNG in supplementary: supp_figs/sigma2_vs_temp.png)
  • Figure 2: Orthogonal Collapse distribution. Histogram of output cosine similarities for input pairs with high initial similarity. Under low SNR conditions ($\gamma < 0.1$), the distribution collapses to a zero-mean Gaussian. (PNG in supplementary: supp_figs/collapse_hist.png)

Theorems & Definitions (13)

  • Remark 1: Precise Noise Expectation
  • Lemma 1: Isotropy of Projected Noise
  • Lemma 2: Strict Variance Contraction
  • proof
  • Lemma 3: Conditional Normalized Perturbation Bound
  • proof
  • Theorem 1: Finite-$n$ Probability Bound
  • proof
  • Theorem 2: Spectral Effective Depth
  • Remark 2: Transient Growth
  • ...and 3 more