Unified-modal Salient Object Detection via Adaptive Prompt Learning

TL;DR

UniSOD tackles the dual challenge of single-modal and multi-modal salient object detection by using adaptive, modality-aware prompts to steer a frozen pre-trained RGB SOD backbone. The core innovation is a Switchable Prompt Generation (SPG) block that builds per-level prompts for both RGB-only and RGB-D/RGB-T inputs, enabling end-to-end training with a small fraction of learnable parameters. Empirical evaluation across 14 benchmarks shows that UniSOD achieves superior performance with fewer trainable parameters compared to task-specific models, validating the effectiveness of prompt-based unification. The approach promises practical deployment benefits through reduced training and storage costs while maintaining high saliency detection accuracy across modalities.

Abstract

Existing single-modal and multi-modal salient object detection (SOD) methods focus on designing specific architectures tailored for their respective tasks. However, developing completely different models for different tasks leads to labor and time consumption, as well as high computational and practical deployment costs. In this paper, we attempt to address both single-modal and multi-modal SOD in a unified framework called UniSOD, which fully exploits the overlapping prior knowledge between different tasks. Nevertheless, assigning appropriate strategies to modality variable inputs is challenging. To this end, UniSOD learns modality-aware prompts with task-specific hints through adaptive prompt learning, which are plugged into the proposed pre-trained baseline SOD model to handle corresponding tasks, while only requiring few learnable parameters compared to training the entire model. Each modality-aware prompt is generated from a switchable prompt generation block, which adaptively performs structural switching based on single-modal and multi-modal inputs without human intervention. Through end-to-end joint training, UniSOD achieves overall performance improvement on 14 benchmark datasets for RGB, RGB-D, and RGB-T SOD, which demonstrates that our method effectively and efficiently unifies single-modal and multi-modal SOD tasks.The code and results are available at https://github.com/Angknpng/UniSOD.

Figures (11)

• Figure 1: Workflow comparisons between existing Salient object detection (SOD) models and our UniSOD. (a) & (b): Existing single-modal and multi-modal SOD methods fully fine-tune the entire encoder-decoder based model, which is designed with a specific architecture tailored for the corresponding SOD task. (c): The proposed UniSOD handles both single-modal and multi-modal inputs in a unified framework, which learns a few parameters of modality-aware prompts to drive a frozen pre-trained model to address corresponding SOD tasks.
• Figure 2: (a): Performance variation of our method and advanced multi-modal methods pang2023caverzhou2023lsnetTang2023RTransNet with modal completeness and absence. X* represents the absence of depth or thermal modal for the X dataset. (b) (c) (d) show the performance comparison of our method with advanced RGB-only SOD methods Yun23SelfReformerma2023bbrfMingchen23ICON, RGB-D SOD methods Cong23PICRNetSun23CATNetpang2023caver, and RGB-T SOD methods zhou23WaveNetpang2023caverTang2023RTransNet on the DUT-TE dataset wang2017Learning, STERE dataset niu2012leveraging, and VT5000 dataset tu2020rgbt, respectively. Overall, our method is slightly affected by modal absence and achieves superior performance in both single-modal and multi-modal SOD tasks.
• Figure 3: Overall architecture of our proposed UniSOD model for both single-modal and multi-modal SOD. The framework builds upon the proposed baseline SOD model (refer to Fig. \ref{['fig::RGB_framework']}), where all parameters are pre-trained and frozen. Initially, the single-modal and multi-modal inputs are fed to the encoder for the extraction of multi-level features. Subsequently, a switchable prompt generation (SPG) block is designed to generate modality-aware prompts through adaptive structural switching without manual intervention. These prompts are attached to the intermediate features of the pre-trained model and then fed into the transformer encoder. This process facilitates prompt learning and drives the pre-trained model to address corresponding SOD tasks.
• Figure 4: Detailed architecture of the pre-trained RGB SOD model.
• Figure 5: Detailed design of the proposed SPG block. The SPG adaptively switches the structure based on single-modal and multi-modal inputs to generate modality-aware prompts.