Attention Is Not What You Need

TL;DR

This paper challenges the indispensability of explicit self-attention by reframing attention as a high-dimensional tensor lifting and proposing an attention-free architecture based on Grassmann flows. It introduces a Causal Grassmann Transformer that reduces token states, encodes local pairs as Plücker coordinates on Gr(2,r), and fuses geometry-backed features through gating and feed-forward blocks. Empirically, Grassmann models with 13–18M parameters achieve roughly Transformer-parallel perplexities on Wikitext-2 and slightly outperform Transformer heads on SNLI when paired with a fixed DistilBERT backbone, while offering linear asymptotic complexity in sequence length for fixed settings. The study argues that operating on a finite-dimensional manifold can yield more interpretable and geometry-aware reasoning, outlining future work on global invariants and richer Grassmann structures.

Abstract

We revisit a basic question in sequence modeling: is explicit self-attention actually necessary for strong performance and reasoning? We argue that standard multi-head attention is best seen as a form of tensor lifting: hidden vectors are mapped into a high-dimensional space of pairwise interactions, and learning proceeds by constraining this lifted tensor through gradient descent. This mechanism is extremely expressive but mathematically opaque, because after many layers it becomes very hard to describe the model with a small family of explicit invariants. To explore an alternative, we propose an attention-free architecture based on Grassmann flows. Instead of forming an L by L attention matrix, our Causal Grassmann layer (i) linearly reduces token states, (ii) encodes local token pairs as two-dimensional subspaces on a Grassmann manifold via Plucker coordinates, and (iii) fuses these geometric features back into the hidden states through gated mixing. Information therefore propagates by controlled deformations of low-rank subspaces over multi-scale local windows, so the core computation lives on a finite-dimensional manifold rather than in an unstructured tensor space. On the Wikitext-2 language modeling benchmark, purely Grassmann-based models with 13 to 18 million parameters achieve validation perplexities within about 10 to 15 percent of size-matched Transformers. On the SNLI natural language inference task, a Grassmann-Plucker head on top of DistilBERT slightly outperforms a Transformer head, with best validation and test accuracies of 0.8550 and 0.8538 compared to 0.8545 and 0.8511. We analyze the complexity of Grassmann mixing, show linear scaling in sequence length for fixed rank, and argue that such manifold-based designs offer a more structured route toward geometric and invariant-based interpretations of neural reasoning.