On the Expressiveness of State Space Models via Temporal Logics

TL;DR

This work characterizes the expressive power of state space models (SSMs) by mapping different architectural variants and arithmetic precisions to fragments of temporal and first-order logics. By encoding SSM layers as logical receivers, diagonal SSM with fixed-precision are shown to capture all languages definable by $\textsc{pLTL}_f$ (star-free regular languages), while diagonal SSM with $\log$-precision extend to counting with backward-looking operators. Time-invariant SSM are capable of modular counting via modular predicates, enabling $\textsc{un-pLTL}_f[\mathtt{MOD}]$ (and with counting, $\textsc{un-pLTL}_f[\mathtt{MOD}, \overleftarrow{\#}]$), and mixed SSM fuse these capabilities to realize $AC^0$-class languages; arbitrary-gated SSM reproduce all regular languages. The results clarify a structural hierarchy among SSMs, relate them to transformer variants (UHAT/AHAT) and to logical fragments like $FO[<]$ and $FO[<, \mathtt{MOD}]$, and establish robust, training-agnostic lower bounds on what tasks these architectures can intrinsically represent. This contributes a principled foundation for comparing SSMs to transformers and guides architectural choices for sequence modelling tasks with formal language considerations. The work also highlights open questions about exact upper bounds (potentially tightening TC$^0$ to AC$^0$) and the precise role of nesting depth and counting in practical settings.

Abstract

We investigate the expressive power of state space models (SSM), which have recently emerged as a potential alternative to transformer architectures in large language models. Building on recent work, we analyse SSM expressiveness through fragments and extensions of linear temporal logic over finite traces. Our results show that the expressive capabilities of SSM vary substantially depending on the underlying gating mechanism. We further distinguish between SSM operating over fixed-width arithmetic (quantised models), whose expressive power remains within regular languages, and SSM with unbounded precision, which can capture counting properties and non-regular languages. In addition, we provide a systematic comparison between these different SSM variants and known results on transformers, thereby clarifying how the two architectures relate in terms of expressive power.

Figures (3)

• Figure 1: Expressiveness hierarchy of SSM architectures mapped to logical fragments and complexity classes. We establish lower bounds for Diagonal, Time-Invariant, and Mixed SSMs, distinguishing between fixed-precision and log-precision (enabling counting operators $\mathop{\overleftarrow{\mathtt{\#}}}$). Dashed arrows indicate provably strict inclusions (Theorem \ref{['thm:diag-upper']}). The gray box delineates the $\text{TC}^0$ upper bound from prior work merrillIllusionStateStatespace2024.
• Figure 2: Detailed comparison of fixed-precision SSM variants with Unique Hard-Attention Transformers (UHAT) yang2024masked. The diagram illustrates a structural alignment: Diagonal SSMs capture star-free languages ($\text{FO}[<]$), matching the capabilities of UHAT without positional encodings. Time-invariant layers introduce modular predicates, providing an expressive lift analogous to adding Positional Encodings to transformers. Consequently, mixed SSMs capture the full class $\text{FO}[<, \mathtt{MOD}\xspace]$ (regular languages in $\text{AC}^0$).
• Figure 3: Comparative hierarchy under log-precision, enabling counting capabilities. The diagram positions SSM variants relative to soft-attention (SAT) yang2024counting and average hard-attention transformers (AHAT) barcelo2024logical. While SSMs strictly subsume the logic of SAT (C-RASP) due to superior local temporal processing (e.g., the Yesterday operator), they remain strictly less expressive than AHAT+PE. This separation arises from causality: SSMs are limited to backward-looking counting ($\mathop{\overleftarrow{\mathtt{\#}}}$), whereas the global attention of AHAT allows for forward-looking counting ($\mathop{\overrightarrow{\mathtt{\#}}}$).

Theorems & Definitions (33)

• Definition 1
• Theorem 1
• proof : Proof sketch
• Lemma 2
• proof
• Theorem 3
• proof
• Theorem 4
• proof : Proof sketch
• Conjecture 1