Tabula RASA: Exposing and Breaking the Relational Bottleneck in Transformers

TL;DR

This work identifies an architectural bottleneck in transformers for multi-hop relational reasoning over knowledge graphs, formalizing that standard transformers are $\mathsf{TC}^0$-complete and require $\Omega(k)$ layers for $k$-hop queries. It introduces RASA, a minimal modification consisting of edge-type embeddings and sparse adjacency masking that injects relational structure into attention and reduces the search space from $O(2^{n^2})$ to $O(2^{m})$. Empirically, RASA achieves strong performance on MetaQA, attaining 3-hop accuracy comparable to or surpassing baselines (including GPT-4 at lower cost) and showing notably lower attention entropy, which indicates more targeted relational reasoning. The results support the thesis that simple, well-motivated architectural changes can meaningfully enhance multi-hop reasoning without extensive pre-training on KG data, though formal learnability guarantees remain out of scope.

Abstract

Transformers achieve remarkable performance across many domains, yet struggle with tasks requiring multi-hop relational reasoning over structured data. We analyze this limitation through circuit complexity: standard transformers are $\mathsf{TC}^0$-complete and require $Ω(k)$ layers for $k$-hop reasoning. We introduce RASA (Relation-Aware Sparse Attention), a minimal modification adding: (1) edge-type embeddings that inject relational structure into attention scores, and (2) sparse masking that restricts attention to graph-adjacent positions. While RASA has the same asymptotic depth requirements, sparse masking reduces the attention search space from $O(2^{n^2})$ to $O(2^m)$ patterns, and edge biases provide explicit relation routing. Empirically, on MetaQA (1/2/3-hop) and WebQuestionsSP, RASA outperforms standard transformers and matches GPT-4 at lower cost, with advantages growing with reasoning depth (+7.1 points on 3-hop). We do not claim formal learnability guarantees; the contribution is empirical validation that minimal structural modifications substantially improve multi-hop reasoning.

Theorems & Definitions (8)

• Definition 3.1: $k$-hop Relational Reasoning
• Definition 3.2: $\textsf{TC}^0$ Complexity Class
• Theorem 3.3: $\textsf{TC}^0$ Completeness, cf. merrill2023parallelism
• Theorem 3.4: Depth Required for $k$-hop
• proof : Proof sketch
• Definition 3.5: Rasa Attention
• Proposition 3.6: Search Space Reduction
• proof