$Δ$t-Mamba3D: A Time-Aware Spatio-Temporal State-Space Model for Breast Cancer Risk Prediction

TL;DR

The paper addresses the challenge of modeling longitudinal, high-dimensional breast imaging data with irregular inter-visit intervals. It introduces Time-Aware $Δt$-Mamba3D, which combines a Δt-modulated continuous-time state-space core with a multi-scale depth-wise 3D neighborhood fusion to capture both irregular timing and spatio-temporal context, while maintaining linear memory. On two breast cancer risk benchmarks, the method achieves higher $c$-index and 1–5 year AUC than time-aware baselines and vision SSMs, with a small parameter count (about $1.8$M) and efficient throughput (~$59.3$M tokens/s). The approach enables scalable analysis of decade-long patient histories and has potential to extend to other imaging modalities and longitudinal clinical tasks.

Abstract

Longitudinal analysis of sequential radiological images is hampered by a fundamental data challenge: how to effectively model a sequence of high-resolution images captured at irregular time intervals. This data structure contains indispensable spatial and temporal cues that current methods fail to fully exploit. Models often compromise by either collapsing spatial information into vectors or applying spatio-temporal models that are computationally inefficient and incompatible with non-uniform time steps. We address this challenge with Time-Aware $Δ$t-Mamba3D, a novel state-space architecture adapted for longitudinal medical imaging. Our model simultaneously encodes irregular inter-visit intervals and rich spatio-temporal context while remaining computationally efficient. Its core innovation is a continuous-time selective scanning mechanism that explicitly integrates the true time difference between exams into its state transitions. This is complemented by a multi-scale 3D neighborhood fusion module that robustly captures spatio-temporal relationships. In a comprehensive breast cancer risk prediction benchmark using sequential screening mammogram exams, our model shows superior performance, improving the validation c-index by 2-5 percentage points and achieving higher 1-5 year AUC scores compared to established variants of recurrent, transformer, and state-space models. Thanks to its linear complexity, the model can efficiently process long and complex patient screening histories of mammograms, forming a new framework for longitudinal image analysis.

Figures (3)

• Figure 1: (a) Illustration of a patient's sequential imaging data acquired with irregular inter-visit gaps $\Delta t$ (e.g., 2009 → 2012 → 2015). (b) Different scanning strategies for spatio-temporal feature volumes. (c) The scanning mechanism in the proposed method $\Delta t$-Mamba3D: time-aware scan modulated by inter-visit gaps $\Delta t$ with learnable multi-scale 3D neighborhood fusion.
• Figure 2: State-space modules. (a) Standard Mamba SSM: input $u_i$ produces parameters $(\bar{A}_i,\bar{B}_i,\delta_i,C_i)$ that update state $x_i$ and emit $y_i$. (b) Time-Aware $\Delta t$-Mamba3D (ours) generalizes to irregularly sampled spatio--temporal grids: each token's step size is modulated by the true inter-visit gap $\Delta t_i$, and a learnable multi-scale 3D neighborhood fusion aggregates local structure across visits before producing $h_i$ and $y_i$.
• Figure 3: Overall architecture of the proposed Time-Aware $\Delta t$-Mamba3D. Top: longitudinal multi-visit, 4-view mammography is encoded per view with Swin-V2, fused by visit, and processed by a hierarchy of $\Delta t$-Mamba3D blocks before the Risk Hazard Module. Bottom right: Expanded diagram of a single $\Delta t$-Mamba3D block.