Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Breaking Alignment Barriers: TPS-Driven Semantic Correlation Learning for Alignment-Free RGB-T Salient Object Detection

Lupiao Hu, Fasheng Wang, Fangmei Chen, Fuming Sun, Haojie Li

TL;DR

This work tackles real-world RGB-T salient object detection where RGB and thermal images are unaligned, presenting the TPS-SCL framework. It combines a dual-encoder backbone with a TPS-driven alignment module, semantic constraint guidance, and cross-modal correlation to robustly fuse bimodal cues. The approach delivers state-of-the-art performance among lightweight methods and competitive results against heavier models on unaligned datasets, demonstrating practical applicability for alignment-free RGB-T SOD. The contributions enable robust saliency detection under spatial misalignment, scale variation, and viewpoint changes in real-world conditions.

Abstract

Existing RGB-T salient object detection methods predominantly rely on manually aligned and annotated datasets, struggling to handle real-world scenarios with raw, unaligned RGB-T image pairs. In practical applications, due to significant cross-modal disparities such as spatial misalignment, scale variations, and viewpoint shifts, the performance of current methods drastically deteriorates on unaligned datasets. To address this issue, we propose an efficient RGB-T SOD method for real-world unaligned image pairs, termed Thin-Plate Spline-driven Semantic Correlation Learning Network (TPS-SCL). We employ a dual-stream MobileViT as the encoder, combined with efficient Mamba scanning mechanisms, to effectively model correlations between the two modalities while maintaining low parameter counts and computational overhead. To suppress interference from redundant background information during alignment, we design a Semantic Correlation Constraint Module (SCCM) to hierarchically constrain salient features. Furthermore, we introduce a Thin-Plate Spline Alignment Module (TPSAM) to mitigate spatial discrepancies between modalities. Additionally, a Cross-Modal Correlation Module (CMCM) is incorporated to fully explore and integrate inter-modal dependencies, enhancing detection performance. Extensive experiments on various datasets demonstrate that TPS-SCL attains state-of-the-art (SOTA) performance among existing lightweight SOD methods and outperforms mainstream RGB-T SOD approaches.

Breaking Alignment Barriers: TPS-Driven Semantic Correlation Learning for Alignment-Free RGB-T Salient Object Detection

TL;DR

This work tackles real-world RGB-T salient object detection where RGB and thermal images are unaligned, presenting the TPS-SCL framework. It combines a dual-encoder backbone with a TPS-driven alignment module, semantic constraint guidance, and cross-modal correlation to robustly fuse bimodal cues. The approach delivers state-of-the-art performance among lightweight methods and competitive results against heavier models on unaligned datasets, demonstrating practical applicability for alignment-free RGB-T SOD. The contributions enable robust saliency detection under spatial misalignment, scale variation, and viewpoint changes in real-world conditions.

Abstract

Existing RGB-T salient object detection methods predominantly rely on manually aligned and annotated datasets, struggling to handle real-world scenarios with raw, unaligned RGB-T image pairs. In practical applications, due to significant cross-modal disparities such as spatial misalignment, scale variations, and viewpoint shifts, the performance of current methods drastically deteriorates on unaligned datasets. To address this issue, we propose an efficient RGB-T SOD method for real-world unaligned image pairs, termed Thin-Plate Spline-driven Semantic Correlation Learning Network (TPS-SCL). We employ a dual-stream MobileViT as the encoder, combined with efficient Mamba scanning mechanisms, to effectively model correlations between the two modalities while maintaining low parameter counts and computational overhead. To suppress interference from redundant background information during alignment, we design a Semantic Correlation Constraint Module (SCCM) to hierarchically constrain salient features. Furthermore, we introduce a Thin-Plate Spline Alignment Module (TPSAM) to mitigate spatial discrepancies between modalities. Additionally, a Cross-Modal Correlation Module (CMCM) is incorporated to fully explore and integrate inter-modal dependencies, enhancing detection performance. Extensive experiments on various datasets demonstrate that TPS-SCL attains state-of-the-art (SOTA) performance among existing lightweight SOD methods and outperforms mainstream RGB-T SOD approaches.
Paper Structure (18 sections, 11 equations, 6 figures, 3 tables)

This paper contains 18 sections, 11 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. RGB-T SOD for Aligned Data
  4. RGB-T SOD for unaligned Data
  5. Proposed Method
  6. Semantic Correlation Constraint Module
  7. TPS Alignment Module
  8. Cross-Modal Correlation Module
  9. Experiments
  10. Implementation Details
  11. Datasets and Metrics
  12. Comparison with SOTA methods
  13. Ablation Study
  14. Effectiveness of SCCM
  15. Effectiveness of TPSAM
  16. ...and 3 more sections

Figures (6)

  • Figure 1: Samples of aligned (a)(b), weakly aligned (c)(d), and unaligned (e)(f) image pairs.
  • Figure 2: Overall structure of the proposed TPS-SCL.
  • Figure 3: Structure of TPSAM.
  • Figure 4: Visualized features from TPSAM.
  • Figure 5: Structure of CMCM (RGB branch).
  • ...and 1 more figures