Samba+: General and Accurate Salient Object Detection via A More Unified Mamba-based Framework
Wenzhuo Zhao, Keren Fu, Jiahao He, Xiaohong Liu, Qijun Zhao, Guangtao Zhai
TL;DR
This work introduces Samba, a pure Mamba-based saliency object detection framework that uses saliency-guided scanning (SNS) and context-aware upsampling (CAU) to maintain spatial continuity and hierarchical feature alignment. Building on Samba, Samba+ is a unified, multi-task, multi-modal model trained across heterogeneous SOD tasks with a hub-and-spoke graph attention (HGA) for cross-modal fusion and modality-anchored continual learning (MACL) to stabilize training and mitigate forgetting, complemented by a synergistic integrity refinement (SIR) module. Empirically, Samba achieves state-of-the-art results across six SOD tasks on 22 datasets with lower computational costs, while Samba+ further improves performance with a single versatile model. The framework demonstrates strong generalization to multi-modal inputs and continuous spatial reasoning, suggesting wide applicability to COD and SLS as well as potential downstream saliency tasks.
Abstract
Existing salient object detection (SOD) models are generally constrained by the limited receptive fields of convolutional neural networks (CNNs) and quadratic computational complexity of Transformers. Recently, the emerging state-space model, namely Mamba, has shown great potential in balancing global receptive fields and computational efficiency. As a solution, we propose Saliency Mamba (Samba), a pure Mamba-based architecture that flexibly handles various distinct SOD tasks, including RGB/RGB-D/RGB-T SOD, video SOD (VSOD), RGB-D VSOD, and visible-depth-thermal SOD. Specifically, we rethink the scanning strategy of Mamba for SOD, and introduce a saliency-guided Mamba block (SGMB) that features a spatial neighborhood scanning (SNS) algorithm to preserve the spatial continuity of salient regions. A context-aware upsampling (CAU) method is also proposed to promote hierarchical feature alignment and aggregation by modeling contextual dependencies. As one step further, to avoid the "task-specific" problem as in previous SOD solutions, we develop Samba+, which is empowered by training Samba in a multi-task joint manner, leading to a more unified and versatile model. Two crucial components that collaboratively tackle challenges encountered in input of arbitrary modalities and continual adaptation are investigated. Specifically, a hub-and-spoke graph attention (HGA) module facilitates adaptive cross-modal interactive fusion, and a modality-anchored continual learning (MACL) strategy alleviates inter-modal conflicts together with catastrophic forgetting. Extensive experiments demonstrate that Samba individually outperforms existing methods across six SOD tasks on 22 datasets with lower computational cost, whereas Samba+ achieves even superior results on these tasks and datasets by using a single trained versatile model. Additional results further demonstrate the potential of our Samba framework.