SalM$^{2}$: An Extremely Lightweight Saliency Mamba Model for Real-Time Cognitive Awareness of Driver Attention

TL;DR

SalM$^2$ targets real-time driver attention prediction by integrating bottom-up visual features with top-down driving-task semantics in a single, ultra-lightweight architecture. The model combines a Mamba-based backbone with Selective Channel Parallel Mamba (SCPM) layers and a Cross-Modal Attention (CMA) fusion module that aligns CLIP-derived scene semantics with image features. It achieves state-of-the-art or near-state-of-the-art performance on TrafficGaze, DrFixD-rainy, and BDDA while using only about $0.08$M parameters and dramatically reduced FLOPs, enabling deployment in resource-constrained autonomous driving systems. The approach demonstrates that semantic guidance, when fused effectively with efficient visual features, substantially improves driver attention prediction with negligible computational cost.

Abstract

Driver attention recognition in driving scenarios is a popular direction in traffic scene perception technology. It aims to understand human driver attention to focus on specific targets/objects in the driving scene. However, traffic scenes contain not only a large amount of visual information but also semantic information related to driving tasks. Existing methods lack attention to the actual semantic information present in driving scenes. Additionally, the traffic scene is a complex and dynamic process that requires constant attention to objects related to the current driving task. Existing models, influenced by their foundational frameworks, tend to have large parameter counts and complex structures. Therefore, this paper proposes a real-time saliency Mamba network based on the latest Mamba framework. As shown in Figure 1, our model uses very few parameters (0.08M, only 0.09~11.16% of other models), while maintaining SOTA performance or achieving over 98% of the SOTA model's performance.

Figures (7)

• Figure 1: Comparison of Parameters and Performance. We compare our model with other state-of-the-art models on the DrFixD-rainy dataset. The horizontal axis represents the number of parameters (M), and the vertical axis represents performance. $\bullet$ denotes our model, and $\bullet$ denotes the best-performing model among the comparison models, and $\bullet$ denotes other comparison models. denotes truncated coordinates, with the origin set to 0.08.
• Figure 2: Overview of the proposed SalM$^2$ network. (a) shows the overall network framework, which includes two branches: a "Bottom-up" branch and a "Top-down" branch. (b) illustrates the principle of the self-attention mechanism. (c) illustrates the principle of our proposed cross-modal attention mechanism.
• Figure 3: Illustrate of the "Bottom-up" backbone network. The purple and blue dashed lines represent skip connections, which include both spatial and channel attention. The skip connections share weights.
• Figure 4: Illustrate of the cross-modal attention mechanism. In the figure, $\otimes$ denotes matrix multiplication, and $\oplus$ represents element-wise addition.
• Figure 5: Qualitative evaluation comparison of our model and the other SOTA methods. Since the weight files of MLNet and SCAFNet are not available in original papers, we could not perform visualization comparisons under identical conditions. $\dagger$ is SalM$^2$ without branch of semantic. $\square$ delineates the attention allocation regions influenced by semantic information.