NeuroSSM: Multiscale Differential State-Space Modeling for Context-Aware fMRI Analysis

TL;DR

NeuroSSM addresses the challenge of modeling fMRI time series across multiple temporal scales by introducing a multiscale differential state-space backbone and a parallel differencing branch, achieving linear complexity in sequence length. The model processes raw ROI-based BOLD signals through MSD-SSB blocks across $K$ scales and a Dual-SSM that jointly handles time-rescaled and differential streams, with outputs fused and pooled for classification. It demonstrates state-of-the-art performance on HCP resting-state and task fMRI as well as PD detection in PPMI, with ablations confirming the complementary value of both multiscale and differential pathways. This approach offers a scalable, data-efficient framework for context-aware fMRI analysis and paves the way for future multi-modal integration and encoding/decoding studies in neuroscience.

Abstract

Accurate fMRI analysis requires sensitivity to temporal structure across multiple scales, as BOLD signals encode cognitive processes that emerge from fast transient dynamics to slower, large-scale fluctuations. Existing deep learning (DL) approaches to temporal modeling face challenges in jointly capturing these dynamics over long fMRI time series. Among current DL models, transformers address long-range dependencies by explicitly modeling pairwise interactions through attention, but the associated quadratic computational cost limits effective integration of temporal dependencies across long fMRI sequences. Selective state-space models (SSMs) instead model long-range temporal dependencies implicitly through latent state evolution in a dynamical system, enabling efficient propagation of dependencies over time. However, recent SSM-based approaches for fMRI commonly operate on derived functional connectivity representations and employ single-scale temporal processing. These design choices constrain the ability to jointly represent fast transient dynamics and slower global trends within a single model. We propose NeuroSSM, a selective state-space architecture designed for end-to-end analysis of raw BOLD signals in fMRI time series. NeuroSSM addresses the above limitations through two complementary design components: a multiscale state-space backbone that captures fast and slow dynamics concurrently, and a parallel differencing branch that increases sensitivity to transient state changes. Experiments on clinical and non-clinical datasets demonstrate that NeuroSSM achieves competitive performance and efficiency against state-of-the-art fMRI analysis methods.

Figures (3)

• Figure 1: Overview of the NeuroSSM framework.(a) The high-level processing pipeline. Region-of-interest (ROI) BOLD signals are extracted from 4D fMRI scans to form a multivariate time series of dimension $T \times N$, where $T$ is the sequence length and $N$ is the number of ROIs. This sequence is processed by a stack of $L$ NeuroSSM modules, which extract hierarchical spatiotemporal features while preserving the sequence length. The final output sequence is aggregated via temporal pooling into a single $N$-dimensional representation, which is then mapped by a linear classifier to task-specific class probabilities (e.g., probability of disease presence or absence). (b) Internal structure of a NeuroSSM block. The input sequence is normalized and distributed to a bank of Multiscale Differential State-Space Blocks (MSD-SSB) operating in parallel across $K$ temporal scales. Each scale $k$ processes the sequence with a distinct step size $\tau_k$, enabling the simultaneous capture of fine-grained transients and slow-varying global trends. The multi-scale outputs are summed, normalized, and activated via GeLU before propagating to subsequent layers.
• Figure 2: Detailed structure of MSD-SSB (Multiscale Differential State-Space Block).(a) The $T \times N$ input sequence is temporally reshaped at scale $k$ with a step size $\tau_k$ into a representation of size $\lfloor T/\tau_k \rfloor \times (\tau_k \cdot N)$. Two parallel input streams are then generated: (i) the $\tau_k$-rescaled BOLD sequence and (ii) its first-order temporal differences, emphasizing rapid state transitions. These two sequences are fed into a Dual-SSM block. The outputs are summed via residual connection and reshaped back to the original $T \times N$ dimensions for layer propagation. (b) The Dual-SSM block processes the raw and differential streams concurrently using parameter-shared selective state-space kernels. Each stream input undergoes a main path involving a depth-wise convolutional layer, SiLU activation, and the SSM to capture long-range dependencies, and a second path involving a nonlinear projection and SiLU activation, functioning as a multiplicative gating mechanism. The outputs of raw and differential streams are additively fused.
• Figure 3: Learning efficiency curves on HCP-Rest, HCP-Task, and PPMI. Each panel displays accuracy, F1 score, or AUC measured on the held-out test set, for models trained on varying fractions $S \in \{5,10,20,50,100\}\%$ of subjects within the development set. The $x$-axis denotes $S$ (training-set fraction), and the $y$-axis shows the corresponding metric value.