Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspective
Fanqi Yan, Huy Nguyen, Pedram Akbarian, Nhat Ho, Alessandro Rinaldo
TL;DR
This work analyzes sigmoid versus softmax self-attention through a Mixture-of-Experts lens, proving that sigmoid self-attention can be more sample-efficient than softmax in the prevalent dense gating regime while remaining comparable in the sparse regime. By representing each attention row as an MoE with quadratic affinity scores, the authors derive convergence and identifiability-based rates for both strongly identifiable and weakly identifiable experts, and for fully and partially quadratic score functions. Theoretical results show that, under dense gating, sigmoid self-attention achieves faster parameter and function-convergence rates (often near $O_P(\sqrt{\log n / n})$) than softmax, especially for polynomial or weakly identifiable experts, with empirical simulations corroborating these gains. The findings highlight practical implications for Transformer design, suggesting sigmoid-based attention can reduce data requirements and computation in realistic, dense-configured models, and point to future work incorporating multi-head attention within a hierarchical MoE framework.
Abstract
At the core of the popular Transformer architecture is the self-attention mechanism, which dynamically assigns softmax weights to each input token so that the model can focus on the most salient information. However, the softmax structure slows down the attention computation due to its row-wise nature, and it inherently introduces competition among tokens: as the weight assigned to one token increases, the weights of others decrease. This competitive dynamic may narrow the focus of self-attention to a limited set of features, potentially overlooking other informative characteristics. Recent experimental studies have shown that using the element-wise sigmoid function helps eliminate token competition and reduce the computational overhead. Despite these promising empirical results, a rigorous comparison between sigmoid and softmax self-attention mechanisms remains absent in the literature. This paper closes this gap by theoretically demonstrating that sigmoid self-attention is more sample-efficient than its softmax counterpart. Toward that goal, we represent the self-attention matrix as a mixture of experts and show that ``experts'' in sigmoid self-attention require significantly less data to achieve the same approximation error as those in softmax self-attention.