Geometric and Dynamic Scaling in Deep Transformers
Haoran Su, Chenyu You
TL;DR
The paper treats rank collapse in ultra-deep Transformers as a geometric problem, where unconstrained residual updates drift off a semantic manifold and accumulate redundant features. It proposes the Manifold-Geometric Transformer (MGT), which combines manifold-constrained hyper-connections (mHC) to keep updates in the tangent space and Deep Delta Learning (DDL) to enable dynamic erasure via a Beta gate, yielding a stable, reversible update law. The core contributions are the MGT block design, the soft tangent-space projection for updates, the geometric Householder-style update, and a comprehensive evaluation framework to test depth beyond 100 layers. The approach aims to decouple update direction from magnitude, enabling robust, scalable deep Transformers and offering a geometric explanation for why depth alone is not the limiting factor in representation learning.
Abstract
Despite their empirical success, pushing Transformer architectures to extreme depth often leads to a paradoxical failure: representations become increasingly redundant, lose rank, and ultimately collapse. Existing explanations largely attribute this phenomenon to optimization instability or vanishing gradients, yet such accounts fail to explain why collapse persists even under modern normalization and initialization schemes. In this paper, we argue that the collapse of deep Transformers is fundamentally a geometric problem. Standard residual updates implicitly assume that feature accumulation is always beneficial, but offer no mechanism to constrain update directions or to erase outdated information. As depth increases, this leads to systematic drift off the semantic manifold and monotonic feature accumulation, causing representational degeneracy. We propose a unified geometric framework that addresses these failures through two orthogonal principles. First, manifold-constrained hyper-connections restrict residual updates to valid local tangent directions, preventing uncontrolled manifold drift. Second, deep delta learning introduces data-dependent, non-monotonic updates that enable reflection and erasure of redundant features rather than their unconditional accumulation. Together, these mechanisms decouple the direction and sign of feature updates, yielding a stable geometric evolution across depth. We term the resulting architecture the Manifold-Geometric Transformer (MGT). Our analysis predicts that enforcing geometric validity while allowing dynamic erasure is essential for avoiding rank collapse in ultra-deep networks. We outline an evaluation protocol for Transformers exceeding 100 layers to test the hypothesis that geometry, rather than depth itself, is the key limiting factor in deep representation learning.