External Prompt Features Enhanced Parameter-efficient Fine-tuning for Salient Object Detection

TL;DR

This work addresses the high parameter cost of transformer-based salient object detection by introducing ExPert, a parameter-efficient fine-tuning framework that freezes the backbone and trains lightweight E-adapters and E-injectors. E-adapters adapt the pre-trained backbone to SOD while E-injectors incorporate external prompt features, including semantic cues from BLIP captions and multi-scale ViT/DINO features, through selective attention. The approach, backbone-agnostic and compatible with multi-scale backbones like SegFormer, achieves state-of-the-art results across five SOD datasets with 80.2M trained parameters, outperforming both CNN-based and prior transformer-based models. The results demonstrate the power of external prompts and adapter-based fine-tuning for efficient, accurate SOD, with potential extensions to other segmentation tasks and richer prompt sources.

Abstract

Salient object detection (SOD) aims at finding the most salient objects in images and outputs pixel-level binary masks. Transformer-based methods achieve promising performance due to their global semantic understanding, crucial for identifying salient objects. However, these models tend to be large and require numerous training parameters. To better harness the potential of transformers for SOD, we propose a novel parameter-efficient fine-tuning method aimed at reducing the number of training parameters while enhancing the salient object detection capability. Our model, termed EXternal Prompt features Enhanced adapteR Tuning (ExPert), features an encoder-decoder structure with adapters and injectors interspersed between the layers of a frozen transformer encoder. The adapter modules adapt the pretrained backbone to SOD while the injector modules incorporate external prompt features to enhance the awareness of salient objects. Comprehensive experiments demonstrate the superiority of our method. Surpassing former state-of-the-art (SOTA) models across five SOD datasets, ExPert achieves 0.215 mean absolute error (MAE) in the ECSSD dataset with 80.2M trained parameters, 21% better than SelfReformer and 47% better than EGNet.

Figures (6)

• Figure 1: The overall architecture of ExPert. During training, the vision encoder is frozen; only the E-adapters, E-injectors and the decoder are trained.
• Figure 2: The decoder of ExPert for multi-scale features. The illustration of feature images is visualized by choosing a random slice of the channel dimension. ExPert's final mask is generated by resizing this mask to the original size.
• Figure 3: The detailed structure of E-adapter and E-injector. D-P is the down projection layer, M-L is the median linear layer, U-P is the up projection layer and Dim-P is the dimension projection layer.
• Figure 4: The F-measure curves and the precision-recall (PR) curves of ExPert and four SOTA models on five datasets.
• Figure 5: The qualitative results of ExPert and four SOTA models. From left to right are the images, the ground truths, ExPert's masks, EVP's masks, SelfReformer's masks, U2Net's masks and EGNet's masks. Better visual effect when zooming in.