Alignment-Free RGBT Salient Object Detection: Semantics-guided Asymmetric Correlation Network and A Unified Benchmark

TL;DR

This work tackles the practical problem of RGB-T salient object detection without manual alignment, introducing SACNet that learns robust cross-modal correlations via an Asymmetric Correlation Module (ACM) and a deformable-feature sampling module (AFSM). A Semantic Guidance Module steers correlation toward salient regions, and a U-Net–style decoder produces high-quality saliency maps. The authors also release UVT2000, a 2000-pair unaligned RGB-thermal benchmark captured in real-world conditions to evaluate alignment-free performance. Across aligned, weakly aligned, and unaligned data, SACNet achieves state-of-the-art results, reducing reliance on labor-intensive alignment and enabling practical multi-modal fusion for surveillance and remote sensing applications.

Abstract

RGB and Thermal (RGBT) Salient Object Detection (SOD) aims to achieve high-quality saliency prediction by exploiting the complementary information of visible and thermal image pairs, which are initially captured in an unaligned manner. However, existing methods are tailored for manually aligned image pairs, which are labor-intensive, and directly applying these methods to original unaligned image pairs could significantly degrade their performance. In this paper, we make the first attempt to address RGBT SOD for initially captured RGB and thermal image pairs without manual alignment. Specifically, we propose a Semantics-guided Asymmetric Correlation Network (SACNet) that consists of two novel components: 1) an asymmetric correlation module utilizing semantics-guided attention to model cross-modal correlations specific to unaligned salient regions; 2) an associated feature sampling module to sample relevant thermal features according to the corresponding RGB features for multi-modal feature integration. In addition, we construct a unified benchmark dataset called UVT2000, containing 2000 RGB and thermal image pairs directly captured from various real-world scenes without any alignment, to facilitate research on alignment-free RGBT SOD. Extensive experiments on both aligned and unaligned datasets demonstrate the effectiveness and superior performance of our method. The dataset and code are available at https://github.com/Angknpng/SACNet.

Figures (12)

• Figure 1: Workflow comparisons between existing networks and our network. (a) Existing networks require a labor-intensive manual alignment process to align visible and thermal image pairs, further exploiting modality complementarity for saliency prediction. (b) Our network directly mines multi-modal correlations and complementarity of initially captured unaligned image pairs for saliency prediction.
• Figure 2: The overall architecture of our proposed SACNet. The framework mainly comprises an Asymmetric Correlation Module (ACM) and an Associated Feature Sampling Module (AFSM). The ACM restricts the correlation operation within asymmetric window pairs to model comprehensive correlations of the two unaligned modalities. With the Semantic Guidance Module (SGM), ACM focus more on salient regions. In the AFSM, relevant thermal features are sampled according to corresponding RGB features. Subsequently, the multi-modal saliency cues are integrated complementarily for saliency prediction.
• Figure 3: Comparison of correlation modeling in unaligned RGBT image pairs between the conventional symmetric window partition and the proposed Asymmetric Correlation Module (ACM), which contains the Asymmetric Window Partition (AWP) strategy and Semantic Guidance Module (SGM).
• Figure 4: Details of the proposed associated feature sampling module
• Figure 5: Top 60% scene and object category distributions in our proposed UVT2000 dataset.