Self-supervised co-salient object detection via feature correspondence at multiple scales

TL;DR

This work tackles unsupervised CoSOD by introducing SCoSPARC, a two-stage self-supervised framework that exploits multi-scale feature correspondences from ViT-based patch-level cross-attention and region-level consensus to detect co-salient objects without segmentation labels. Stage 1 generates cross-attention maps using patch-level features and optimizes a co-occurrence loss $L_{cooc}$ alongside a saliency loss $L_{sal}$, guided by DINO attention maps; Stage 2 refines these results by filtering regions through a region-level similarity to a global foreground representation and dense CRF post-processing. Empirically, SCoSPARC achieves state-of-the-art performance among unsupervised CoSOD methods and is competitive with several supervised models, delivering notable gains such as a 13.7% improvement in F-measure on CoCA over the previous unsupervised SOTA and a 4.6% gain over a recent supervised model, all while remaining lightweight and efficient. The method demonstrates robustness to backbone choices and training data composition, and its multi-scale approach offers practical benefits for real-world CoSOD tasks with limited labeled data.

Abstract

Our paper introduces a novel two-stage self-supervised approach for detecting co-occurring salient objects (CoSOD) in image groups without requiring segmentation annotations. Unlike existing unsupervised methods that rely solely on patch-level information (e.g. clustering patch descriptors) or on computation heavy off-the-shelf components for CoSOD, our lightweight model leverages feature correspondences at both patch and region levels, significantly improving prediction performance. In the first stage, we train a self-supervised network that detects co-salient regions by computing local patch-level feature correspondences across images. We obtain the segmentation predictions using confidence-based adaptive thresholding. In the next stage, we refine these intermediate segmentations by eliminating the detected regions (within each image) whose averaged feature representations are dissimilar to the foreground feature representation averaged across all the cross-attention maps (from the previous stage). Extensive experiments on three CoSOD benchmark datasets show that our self-supervised model outperforms the corresponding state-of-the-art models by a huge margin (e.g. on the CoCA dataset, our model has a 13.7% F-measure gain over the SOTA unsupervised CoSOD model). Notably, our self-supervised model also outperforms several recent fully supervised CoSOD models on the three test datasets (e.g., on the CoCA dataset, our model has a 4.6% F-measure gain over a recent supervised CoSOD model).

Figures (8)

• Figure 1: Visualization of co-saliency detections on the pocket watch image group from the CoCA dataset zhang2020gradient: Row 1: Original image, Row 2: predictions from the DVFDVD model amir2021deep that only mines local patch-level correspondences, Row 3: our predictions with only local (patch) feature correspondence, Row 4: our predictions with both local (patch) and global (region) feature correspondence which produces the best results, Row 5: Ground truth.
• Figure 2: The proposed two-stage self-supervised CoSOD model, SCoSPARC. In the first stage, we train a network that leverages the local ViT feature correspondences across all patches in the images in the group to obtain cross-attention heatmaps, which we further threshold using a confidence-based adaptive threshold to obtain an intermediate binary segmentation map. In the next stage, we refine these segmentations via region-level feature correspondence using the average foreground token obtained from the previous stage, followed by dense CRF-based segmentation refinement.
• Figure 3: Qualitative comparison of the performance of different baselines with our SCoSPARC on three image groups, each selected from the CoCA, Cosal2015, and CoSOD3k datasets. Our model produces the most accurate segmentations.
• Figure 4: Visualizations of intermediate self-attention maps, cross-attention maps, and segmentation maps for two instances from the handbag category from CoCA. The yellow boxes highlight the regions eliminated using stage 2 of our SCoSPARC model.
• Figure 5: Additional qualitative comparison of the performance of different baselines with our self-supervised CoSOD model on three image groups, each selected from the CoCA, Cosal2015, and CoSOD3k datasets. Our model produces the most accurate segmentations.