Theoretical limitations of multi-layer Transformer
Lijie Chen, Binghui Peng, Hongxun Wu
TL;DR
This work addresses the fundamental question of how powerful multi-layer decoder-only Transformers are for performing sequential composition. It introduces a multi-party autoregressive communication model to capture decoder computations and defines the L-sequential function composition task, proving an unconditional lower bound: for constant $L$, solving $L$-sequential function composition requires $n^{\Omega(1)}$ parameter dimensions when $L \le \widetilde{O}(\log\log n)$ and $Hdp \le n^{2^{-4L}}$. As consequences, there is an exponential depth-width tradeoff (an $(L+1)$-layer Transformer can solve the task with polylogarithmic parameters, while any $L$-layer model requires polynomial resources), an unconditional encoder–decoder separation (a shallow encoder can solve with polylogarithmic size where an $L$-layer decoder cannot), and provable chain-of-thought advantages (CoT) enabling certain tasks to be solved with fewer layers. The paper avoids reliance on circuit lower bounds like $\mathsf{TC}^0$, instead leveraging the information bottleneck and autoregressive structure to establish the bounds. Collectively, these results provide a sharp, unconditional understanding of the limitations and depth-related capabilities of decoder-only Transformers and furnish a new analytic framework—the indistinguishable decomposition technique—for future exploration of Transformer power.
Abstract
Transformers, especially the decoder-only variants, are the backbone of most modern large language models; yet we do not have much understanding of their expressive power except for the simple $1$-layer case. Due to the difficulty of analyzing multi-layer models, all previous work relies on unproven complexity conjectures to show limitations for multi-layer Transformers. In this work, we prove the first $\textit{unconditional}$ lower bound against multi-layer decoder-only transformers. For any constant $L$, we prove that any $L$-layer decoder-only transformer needs a polynomial model dimension ($n^{Ω(1)}$) to perform sequential composition of $L$ functions over an input of $n$ tokens. As a consequence, our results give: (1) the first depth-width trade-off for multi-layer transformers, exhibiting that the $L$-step composition task is exponentially harder for $L$-layer models compared to $(L+1)$-layer ones; (2) an unconditional separation between encoder and decoder, exhibiting a hard task for decoders that can be solved by an exponentially shallower and smaller encoder; (3) a provable advantage of chain-of-thought, exhibiting a task that becomes exponentially easier with chain-of-thought. On the technical side, we propose the multi-party $\textit{autoregressive}$ $\textit{communication}$ $\textit{model}$ that captures the computation of a decoder-only Transformer. We also introduce a new proof technique that finds a certain $\textit{indistinguishable}$ $\textit{decomposition}$ of all possible inputs iteratively for proving lower bounds in this model. We believe our new communication model and proof technique will be helpful to further understand the computational power of transformers.