Dissecting Linear Recurrent Models: How Different Gating Strategies Drive Selectivity and Generalization

TL;DR

This paper tackles how linear recurrent memories (LRMs) differ in gating, channel mixing, and rank, and how these features affect selectivity and generalization. It introduces SelectivBench, a lightweight, grammar-based benchmark built on SymSeqBench to systematically stress test selectivity with controlled grammar complexity and deliberate gaps. Across multiple LRMs and a Transformer baseline, the study shows gating and fast forgetting improve recall, in-state channel mixing aids generalization, and softmax attention remains superior when long-context memory is crucial. The framework enables targeted, scalable analysis of architectural choices and provides insights for designing efficient sequence models with controlled selective behavior.

Abstract

Linear recurrent neural networks have emerged as efficient alternatives to the original Transformer's softmax attention mechanism, thanks to their highly parallelizable training and constant memory and computation requirements at inference. Iterative refinements of these models have introduced an increasing number of architectural mechanisms, leading to increased complexity and computational costs. Nevertheless, systematic direct comparisons among these models remain limited. Existing benchmark tasks are either too simplistic to reveal substantial differences or excessively resource-intensive for experimentation. In this work, we propose a refined taxonomy of linear recurrent models and introduce SelectivBench, a set of lightweight and customizable synthetic benchmark tasks for systematically evaluating sequence models. SelectivBench specifically evaluates selectivity in sequence models at small to medium scale, such as the capacity to focus on relevant inputs while ignoring context-based distractors. It employs rule-based grammars to generate sequences with adjustable complexity, incorporating irregular gaps that intentionally violate transition rules. Evaluations of linear recurrent models on SelectivBench reveal performance patterns consistent with results from large-scale language tasks. Our analysis clarifies the roles of essential architectural features: gating and rapid forgetting mechanisms facilitate recall, in-state channel mixing is unnecessary for selectivity, but critical for generalization, and softmax attention remains dominant due to its memory capacity scaling with sequence length. Our benchmark enables targeted, efficient exploration of linear recurrent models and provides a controlled setting for studying behaviors observed in large-scale evaluations. Code is available at https://github.com/symseqbench/selectivbench

Figures (4)

• Figure 1: Illustration of the benchmark task. A) The diagram defines the AG rules used to generate sequences. B) An illustration of a generated sample and the experimental setup (see \ref{['sec:artifical_grammer']} section for more details.)
• Figure 2: Test accuracy as a function of the sequence complexity level (TE) (A) and the model size (number of parameters in millions) (B). Results are averaged over 4 runs and reported as mean $\pm$ standard deviation.
• Figure 3: Test accuracy as a function of model size when gaps are introduced between sequence items. Results are averaged over 4 runs and reported as mean $\pm$ standard deviation. A) Noisy gaps. B) Non-grammatical gaps.
• Figure 4: Generalization accuracy as a function of the number of gap items. Results are averaged over 4 runs and reported as mean $\pm$ standard deviation.