Geometric Attention: A Regime-Explicit Operator Semantics for Transformer Attention
Luis Rosario Freytes
TL;DR
Geometric Attention (GA) reframes Transformer attention as an operator defined by four regime-declareable inputs: a carrier (bucketization), an evidence kernel (via a masked proto-score and link), a probe family (defining observational equivalence), and an anchor/update rule (chosen kernel representation). The paper proves that, under key prunes such as Regime E (extensional decoding with Newtonian tokenization) and Prune W (additive work forcing an exponential link), the resulting weights take a Gibbs form and, with row anchoring, reduce to row-softmax; after quotienting unary (row/column) content, the interaction component can be approximated by a low-rank dot-product form via Eckart–Young/SVD, recovering standard Transformer attention in the fixed-carrier regime. GA also formalizes adaptive carriers, depth as staged memory, and closure under mixtures and FFNs, providing a principled separation between invariant operator structure and modeling choices and enabling principled comparisons and extensions of attention-based architectures. The framework connects to entropic transport anchors, OT/Sinkhorn plans, and multi-head attention as modular, gauge-fixed instances within a broader regime-tree, and emphasizes training as a procedure- and regime-search over a space of operator definitions. Altogether, GA offers a rigorous, regime-explicit lens for analyzing, extending, and comparing attention mechanisms beyond traditional token-based transformers.
Abstract
Geometric Attention (GA) specifies an attention layer by four independent inputs: a finite carrier (what indices are addressable), an evidence-kernel rule (how masked proto-scores and a link induce nonnegative weights), a probe family (which observables are treated as admissible), and an anchor/update rule (which representative kernel is selected and how it is applied). Probe families induce an operational equivalence relation on kernels and therefore a gauge; anchors select representatives relative to that probe. Under a scalar relational-work representation and a multiplicative compositionality law for evidence, the admissible link family is exponential, yielding Gibbs weights; with row anchoring this includes the softmax kernel family as a subregime. After quotienting unary row/column score fields, the remaining interaction component admits a canonical rank-r normal form (Eckart-Young/SVD); dot-product score charts implement the corresponding low-rank interaction regime. Fixing the carrier and extensionalizing the update yields the standard fixed-token Transformer attention operator; allowing carrier updates yields adaptive-carrier and staged-depth regimes. The operator language also supports multihead/mixed kernels, plan-based anchors (e.g., entropic OT/Sinkhorn), and unary operators (e.g., FFN-style fields) as explicit regime choices. This separates invariant structure from modeling choice, enabling principled comparison and extension of attention mechanisms, and attention-based architectures.