Poolformer: Recurrent Networks with Pooling for Long-Sequence Modeling
Daniel Gallo Fernández
TL;DR
Poolformer introduces a pooling-based recurrent architecture for long-sequence modeling, replacing self-attention with RG-LRU-inspired temporal mixing and using SkipBlocks with down/up pooling to dramatically reduce sequence length. The approach yields faster training, better perceptual metrics (FID/IS), and robust generalization on raw audio, while revealing that deep layers capture long-range dependencies and shallow layers focus on short-term features. Empirical results on SC09, Beethoven, and YouTubeMix show competitive log-likelihood with state-of-the-art methods like SaShiMi and Mamba, and superior efficiency and perceptual quality due to pooling. The work outlines clear paths for extending Poolformer to text, vision, and multi-modal settings, potentially enabling Poolformer-based LLMs that process dense representations of images and videos.
Abstract
Sequence-to-sequence models have become central in Artificial Intelligence, particularly following the introduction of the transformer architecture. While initially developed for Natural Language Processing, these models have demonstrated utility across domains, including Computer Vision. Such models require mechanisms to exchange information along the time dimension, typically using recurrent or self-attention layers. However, self-attention scales quadratically with sequence length, limiting its practicality for very long sequences. We introduce Poolformer, a sequence-to-sequence model that replaces self-attention with recurrent layers and incorporates pooling operations to reduce sequence length. Poolformer is defined recursively using SkipBlocks, which contain residual blocks, a down-pooling layer, a nested SkipBlock, an up-pooling layer, and additional residual blocks. We conduct extensive experiments to support our architectural choices. Our results show that pooling greatly accelerates training, improves perceptual metrics (FID and IS), and prevents overfitting. Our experiments also suggest that long-range dependencies are handled by deep layers, while shallow layers take care of short-term features. Evaluated on raw audio, which naturally features long sequence lengths, Poolformer outperforms state-of-the-art models such as SaShiMi and Mamba. Future directions include applications to text and vision, as well as multi-modal scenarios, where a Poolformer-based LLM could effectively process dense representations of images and videos.