Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms

TL;DR

GFSSM addresses training instability in long-sequence Structured State Space Models by decomposing recurrent computations into grouped FIR-filtered pathways and stabilizing dynamics with attention sink mechanisms. The approach combines semiseparable matrix decompositions with a grouped processing scheme to maintain linear-time complexity while improving robustness over long sequences. By integrating an attention sink inspired mechanism, GFSSM preserves initial context and stabilizes state updates, bridging toward Transformer-like performance without quadratic attention costs. The work outlines methodological foundations and proposes future empirical validation across NLP, speech, and genomics tasks.

Abstract

Structured State Space Models (SSMs) have emerged as compelling alternatives to Transformer architectures, offering linear-time complexity and superior performance in various sequence modeling tasks. Despite their advantages, SSMs like the original Mamba-2 face training difficulties due to the sensitivities introduced by the extended series of recurrent matrix multiplications. In this paper, we propose an advanced architecture that mitigates these challenges by decomposing A-multiplications into multiple groups and optimizing positional encoding through Grouped Finite Impulse Response (FIR) filtering. This new structure, denoted as Grouped FIR-enhanced SSM (GFSSM), employs semiseparable matrices for efficient computation. Furthermore, inspired by the "attention sink" phenomenon identified in streaming language models, we incorporate a similar mechanism to enhance the stability and performance of our model over extended sequences. Our approach further bridges the gap between SSMs and Transformer architectures, offering a viable path forward for scalable and high-performing sequence modeling.

Figures (1)

• Figure 1: The matrix $L$ for GFSSM