Circuit Complexity Bounds for RoPE-based Transformer Architecture
Bo Chen, Xiaoyu Li, Yingyu Liang, Jiangxuan Long, Zhenmei Shi, Zhao Song
TL;DR
This work establishes a circuit- complexity bound forRoPE-based Transformer architectures, proving that such models can be simulated by uniform $\mathsf{TC}^0$ circuits under polynomial precision, constant depth, and $d \le O(n)$, while also showing that, unless $TC^0 = NC^1$, they cannot solve fundamental problems like Arithmetic Formula Evaluation or Boolean Formula Value. The analysis proceeds by decomposing RoPE into tractable components (trigonometric functions, matrix products, attention, and MLP/LN blocks) and placing each within $\mathsf{TC}^0$, then aggregating to the full multi-layer transformer. The key contributions are (i) a detailed per-component $\mathsf{TC}^0$-computation bound for RoPE-based transformers, (ii) the main result that RoPE transformers are simulable in $\mathsf{TC}^0$, and (iii) hardness theorems showing expressivity limits relative to $TC^0$ vs $NC^1$ separations. These results illuminate fundamental theoretical constraints on RoPE-based architectures, offering guidance for future theoretical and practical exploration of their capabilities and limitations.
Abstract
Characterizing the express power of the Transformer architecture is critical to understanding its capacity limits and scaling law. Recent works provide the circuit complexity bounds to Transformer-like architecture. On the other hand, Rotary Position Embedding ($\mathsf{RoPE}$) has emerged as a crucial technique in modern large language models, offering superior performance in capturing positional information compared to traditional position embeddings, which shows great potential in application prospects, particularly for the long context scenario. Empirical evidence also suggests that $\mathsf{RoPE}$-based Transformer architectures demonstrate greater generalization capabilities compared to conventional Transformer models. In this work, we establish a circuit complexity bound for Transformers with $\mathsf{RoPE}$ attention. Our key contribution is that we show that unless $\mathsf{TC}^0 = \mathsf{NC}^1$, a $\mathsf{RoPE}$-based Transformer with $\mathrm{poly}(n)$-precision, $O(1)$ layers, hidden dimension $d \leq O(n)$ cannot solve the Arithmetic formula evaluation problem or the Boolean formula value problem. This result significantly demonstrates the fundamental limitation of the expressivity of the $\mathsf{RoPE}$-based Transformer architecture, although it achieves giant empirical success. Our theoretical result not only establishes the complexity bound but also may instruct further work on the $\mathsf{RoPE}$-based Transformer.