On the Intrinsic Limits of Transformer Image Embeddings in Non-Solvable Spatial Reasoning

TL;DR

The paper identifies an intrinsic limit of constant-depth transformer image embeddings in handling non-solvable spatial reasoning, formalizing spatial understanding as a Group Homomorphism and linking it to the Word Problem. It shows that for groups like $SO(3)$, the Word Problem is $ ext{NC^1}$-complete, while standard ViTs operate within $ ext{TC^0}$, implying a fundamental depth-based barrier under the conjecture $ ext{TC^0} \subsetneq \text{NC^1}$. Through the Latent Space Algebra benchmark and recursive linear probing, the authors demonstrate a structural collapse in ViT representations for non-solvable transformations as compositional depth grows, while abelian cases remain tractable. The work argues that achieving genuine spatial reasoning requires architectural changes that provide deeper logical depth or explicit geometric inductive biases, with implications for safety and efficiency in embodied AI. Overall, the results reveal a principled trade-off between expressivity and depth in current transformer architectures for spatial tasks.

Abstract

Vision Transformers (ViTs) excel in semantic recognition but exhibit systematic failures in spatial reasoning tasks such as mental rotation. While often attributed to data scale, we propose that this limitation arises from the intrinsic circuit complexity of the architecture. We formalize spatial understanding as learning a Group Homomorphism: mapping image sequences to a latent space that preserves the algebraic structure of the underlying transformation group. We demonstrate that for non-solvable groups (e.g., the 3D rotation group $\mathrm{SO}(3)$), maintaining such a structure-preserving embedding is computationally lower-bounded by the Word Problem, which is $\mathsf{NC^1}$-complete. In contrast, we prove that constant-depth ViTs with polynomial precision are strictly bounded by $\mathsf{TC^0}$. Under the conjecture $\mathsf{TC^0} \subsetneq \mathsf{NC^1}$, we establish a complexity boundary: constant-depth ViTs fundamentally lack the logical depth to efficiently capture non-solvable spatial structures. We validate this complexity gap via latent-space probing, demonstrating that ViT representations suffer a structural collapse on non-solvable tasks as compositional depth increases.

Figures (5)

• Figure 1: The Homomorphism Alignment Problem. We illustrate our core research inquiry: Given a sequence of input observations transformed by a group $G$ (e.g., a rotating bunny), can a constant-depth ViT encoder map them to a latent sequence where the induced transition dynamics $H$ preserve the group structure ($H \cong G$)? We theoretically and empirically demonstrate that for non-solvable groups like $\mathrm{SO}(3)$, this isomorphism is strictly prohibited by the circuit complexity constraints of the architecture.
• Figure 2: Absolute Loss Trajectories. The prediction error (MSE and Cosine) vs. Sequence Length $N$. A consistent hierarchy ($L3 \gg L2 > L1$) is observed across all models before they hit the failure threshold. All models exhibit a trend where error increases with $N$, but the rate is highly dependent on algebraic structure. Level 3 (Non-Solvable) consistently incurs 3-3.8$\times$ higher error than Level 1 (Abelian), validating the complexity gap.
• Figure 3: Structural Collapse relative to Baseline. The loss normalized by the Identity Baseline ($Loss / Loss_{\text{Identity}}$). A ratio $\ge 1.0$ (dashed line) indicates the model performs worse than a static guess ($z_{\text{pred}} = z_0$). Note that Level 3 (red) consistently approaches this collapse threshold faster than Level 1 (blue), except in supervised Cosine loss where invariance causes immediate failure.
• Figure 4: Metric Sensitivity and Divergence Speed. Comparison of MSE vs. Cosine loss growth. Supervised models (ViT-B, ResNet) show a catastrophic divergence in Cosine loss (approx. 6$\times$ faster than MSE), indicating a lack of orientation awareness. DINOv2 is more balanced, yet still suffers from rapid degradation on Level 3, confirming the architectural barrier.
• Figure 5: Visualizing the Algebraic Hierarchy. We display sample atomic transitions ($N=1$) for the Bunny and Dragon objects across the three complexity levels. Left (Level 1): Pure 2D translations preserve orientation and scale. Center (Level 2): Affine transformations introduce scaling centered on the frame, altering size but maintaining 2D planar orientation. Right (Level 3): 3D Rotations introduce out-of-plane transformations, revealing occluded geometry and fundamentally altering the visual topology, corresponding to the non-solvable $\mathrm{SO}(3)$ group structure.

Theorems & Definitions (9)

• Definition 3.1: The Finite Group Word Problem
• Definition 4.1: Homomorphic Spatial Embedding
• Lemma 4.2: Reduction to the Word Problem
• proof
• Proposition 4.3: ViT Circuit Complexity under Polynomial Precision
• proof
• Theorem 4.4: The Non-Solvable Barrier
• proof
• Corollary 4.5: The Abelian Collapse