Pool Me Wisely: On the Effect of Pooling in Transformer-Based Models
Sofiane Ennadir, Levente Zólyomi, Oleg Smirnov, Tianze Wang, John Pertoft, Filip Cornell, Lele Cao
TL;DR
This work reveals pooling as a principled driver of Transformer expressivity, introducing a formal framework to quantify how fixed and learnable pooling strategies influence the capacity to distinguish similar inputs. By deriving explicit bounds that depend on pooling type and architectural constants, the authors show contractive pooling favors global context while expansive pooling enhances local sensitivity, with learnable pooling balancing both ends. Empirical validation across vision, language, and time-series domains confirms task-dependent pooling performance and demonstrates that adaptive pooling approaches can approach or exceed fixed strategies as model size grows. The study thus provides theoretical and practical guidance for selecting or designing pooling mechanisms to match task demands and inductive biases, and points to future work on dynamic pooling and robustness considerations. $ $
Abstract
Transformer models have become the dominant backbone for sequence modeling, leveraging self-attention to produce contextualized token representations. These are typically aggregated into fixed-size vectors via pooling operations for downstream tasks. While much of the literature has focused on attention mechanisms, the role of pooling remains underexplored despite its critical impact on model behavior. In this paper, we introduce a theoretical framework that rigorously characterizes the expressivity of Transformer-based models equipped with widely used pooling methods by deriving closed-form bounds on their representational capacity and the ability to distinguish similar inputs. Our analysis extends to different variations of attention formulations, demonstrating that these bounds hold across diverse architectural variants. We empirically evaluate pooling strategies across tasks requiring both global and local contextual understanding, spanning three major modalities: computer vision, natural language processing, and time-series analysis. Results reveal consistent trends in how pooling choices affect accuracy, sensitivity, and optimization behavior. Our findings unify theoretical and empirical perspectives, providing practical guidance for selecting or designing pooling mechanisms suited to specific tasks. This work positions pooling as a key architectural component in Transformer models and lays the foundation for more principled model design beyond attention alone.