Breaking the Attention Bottleneck

TL;DR

The paper addresses the decoder attention bottleneck caused by the quadratic $O(n^2)$ complexity of standard attention by replacing it with a static global-context generation function that preserves autoregressive behavior, introducing a generative activation as an attention substitute and optionally an average-context vector. It presents a nanoGPT-like experimental setup demonstrating that a static replacement can achieve lower validation losses with smaller models and that averaging over context further improves performance, signaling a path toward more efficient, interpretable transformer variants. Key contributions include a parameter-free or minimally parameterized attention substitute with near-linear time behavior and practical guidance on integration with broader architectures, along with empirical evidence on small benchmarks like tiny Shakespeare. The work has practical significance for resource-constrained deployment and open collaboration, licensed under AGPL v3 to promote public access and reuse.

Abstract

Attention-based transformers have become the standard architecture in many deep learning fields, primarily due to their ability to model long-range dependencies and handle variable-length input sequences. However, the attention mechanism with its quadratic complexity is a significant bottleneck in the transformer architecture. This algorithm is only uni-directional in the decoder and converges to a static pattern in over-parametrized decoder-only models. I address this issue by developing a generative function as attention or activation replacement. It still has the auto-regressive character by comparing each token with the previous one. In my test setting with nanoGPT this yields a smaller loss while having a smaller model. The loss further drops by incorporating an average context vector. This concept of attention replacement is distributed under the GNU AGPL v3 license at https://gitlab.com/Bachstelze/causal_generation.

Figures (4)

• Figure 1: This attention pattern is repeated in all heads of instruction-tuned GPTs. Each token has a uniform distribution over all previous tokens as a consistent auto-regressive activation.
• Figure 2: The training and validation loss curve of the standard attention and recurrent maximization as well as minimization.
• Figure 3: The validation loss curve of the standard attention and recurrent methods with an average context.
• Figure 4: The validation loss curve of the standard attention and recurrent methods with an average context in the middle Shakespeare setting of nanoGPT.