Bi-Orthogonal Factor Decomposition for Vision Transformers

TL;DR

Bi-orthogonal Factor Decomposition is introduced, a two-stage analytical framework that exposes how tokens interact through attention and which informational factors mediate their communication, yielding practical insights into vision transformer mechanisms.

Abstract

Self-attention is the central computational primitive of Vision Transformers, yet we lack a principled understanding of what information attention mechanisms exchange between tokens. Attention maps describe where weight mass concentrates; they do not reveal whether queries and keys trade position, content, or both. We introduce Bi-orthogonal Factor Decomposition (BFD), a two-stage analytical framework: first, an ANOVA-based decomposition statistically disentangles token activations into orthogonal positional and content factors; second, SVD of the query-key interaction matrix QK^T exposes bi-orthogonal modes that reveal how these factors mediate communication. After validating proper isolation of position and content, we apply BFD to state-of-the-art vision models and uncover three phenomena.(i) Attention operates primarily through content. Content-content interactions dominate attention energy, followed by content-position coupling. DINOv2 allocates more energy to content-position than supervised models and distributes computation across a richer mode spectrum. (ii) Attention mechanisms exhibit specialization: heads differentiate into content-content, content-position, and position-position operators, while singular modes within heads show analogous specialization. (iii) DINOv2's superior holistic shape processing emerges from intermediate layers that simultaneously preserve positional structure while contextually enriching semantic content. Overall, BFD exposes how tokens interact through attention and which informational factors - positional or semantic - mediate their communication, yielding practical insights into vision transformer mechanisms.

Figures (23)

• Figure 1: Content and Position Interactions in DINOv2. Representative bi-orthogonal modes in DINOv2 whose activation patterns illustrate three different interaction. Left: Content–content modes activate on semantic interactions such as object parts, revealing part-to-part or part-to-whole correspondences. Middle: Content–position modes show localization-aware semantic interactions, where semantic features are modulated by spatial context. Right: Position–position modes activate on purely spatial interaction, like left–right flows, top–bottom variations, and Fourier-like patterns, highlighting geometric organization without semantic selectivity. These examples show how different interaction types contribute to information flow in attention.
• Figure 2: Bi-orthogonal Factor Decomposition (BFD). Each token activation is decomposed into positional ($\mu_p$) and content ($\mu_c$) components. The bilinear interaction between these components is then analyzed through a singular value decomposition, yielding biorthogonal mode pairs $(U_i, V_i)$ that explain why two tokens interact. A mode is activated when the query aligns with $U_i$ and the key with $V_i$. For example, when $U_i$ aligns with a token’s positional component while $V_i$ aligns with another token’s content component, the mode expresses a location-aware semantic interaction, effectively mixing where something is with what it is. This decomposition exposes which paired directions enable tokens to interact and clarifies whether their interaction arises from positional or content factors or both.
• Figure 3: Validating Factor Isolation. Linear probes decode spatial coordinates from factorized representations across layers. Left: Position is successfully decoded from the raw block activations in both architectures, with DINOv2 maintaining higher accessibility through intermediate layers before both converge to 20% accuracy at depth. Right: Content factors $\bm{\mu}_{\texttt{c}}^{(p)}(\bm{x})$ contain negligible positional information (chance-level decoding), confirming that the factorization removes positional information and achieves statistical orthogonality. The asymmetric preservation trajectory reveals that self-supervised training maintains linearly accessible positional structure deeper into the network than supervised learning.
• Figure 4: Projections of bi-orthogonal modes in DINOv2. The query (red) and key (cyan) singular vectors are projected onto either the content or positional factor, highlighting the image regions that most strongly activate each singular direction.
• Figure 5: Energy Distribution Across Informational Factors. Layer-wise decomposition of attention energy into contributions from layer effect (global bias), position, content, and their pairwise interactions. Both architectures allocate the majority of energy to content-based interactions (content-content and content-position), confirming that attention-mediated interaction reflects genuine semantic computation rather than positional structure alone. A divergence emerges: DINOv2 dedicates substantially greater energy to content-position coupling across all layers, which could be a sign of localization enrichment of semantic content (content gets modulated by positional information).

Theorems & Definitions (2)

• Definition 1: Vision Transformer Spatial-Content Factorization
• Definition 2: Bi-orthogonal Mode Decomposition.