Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences

TL;DR

This work targets the core bottleneck of long-sequence modeling: the quadratic cost of full attention. It introduces CHELA, a hardware-efficient hybrid that combines short-long convolutions with hardware-friendly linear attention, enabled by tiling and structural reparameterization to preserve linear complexity $O(L)$. The model uses a gating mechanism that fuses global multi-frequency abstractions from convolutions into the data-dependent attention stream, yielding stable training and strong performance. Across the Long Range Arena, raw speech, autoregressive language modeling, and pixel-level sequence classification, CHELA delivers state-of-the-art results with substantial speedups over baseline Transformers and prior hybrids, demonstrating a practical path toward efficient, scalable long-sequence learners.

Abstract

To mitigate the computational complexity in the self-attention mechanism on long sequences, linear attention utilizes computation tricks to achieve linear complexity, while state space models (SSMs) popularize a favorable practice of using non-data-dependent memory pattern, i.e., emphasize the near and neglect the distant, to processing sequences. Recent studies have shown the priorities by combining them as one. However, the efficiency of linear attention remains only at the theoretical level in a causal setting, and SSMs require various designed constraints to operate effectively on specific data. Therefore, in order to unveil the true power of the hybrid design, the following two issues need to be addressed: (1) hardware-efficient implementation for linear attention and (2) stabilization of SSMs. To achieve this, we leverage the thought of tiling and hierarchy to propose CHELA (short-long Convolutions with Hardware-Efficient Linear Attention), which replaces SSMs with short-long convolutions and implements linear attention in a divide-and-conquer manner. This approach enjoys global abstraction and data-dependent selection from stable SSM and linear attention while maintaining real linear complexity. Our comprehensive experiments on the Long Range Arena benchmark and language modeling tasks demonstrate the effectiveness of the proposed method.

Figures (4)

• Figure 1: Demonstration of (left) comparison of various popular models on the Long Range Arena Dataset, and (right) speed benchmark on different implementations of attentions. Figure left, despite showing promising performance of SSM-Attention models, the linear version of these models degenerated. Figure right benchmarks attention speed with around 200M parameters, showing linear attention is not linear with sequence length and is significantly slower than S4, which can be a speed bottleneck in these hybrid designs.
• Figure 2: CHELA-model architecture. Figure (a) shows the CHELA layer. Figure (b) is a simplified graphic demonstration of the key components of CHELA: Short-Long Convolutions and Flash Linear Attention. HBM and SRAM denote high bandwidth memory and static random-access memory in GPU, respectively.
• Figure 3: Illustration Short-long Module and the effect of Short Convolution. Figure left shows the details of the module structure. Figure right proves the role of the short kernel.
• Figure 4: Comparative Analysis of Speed: Runtime in milliseconds for the forward and backward pass across varying lengths.