Segment Any RGB-Thermal Model with Language-aided Distillation

TL;DR

This work tackles RGB-T semantic segmentation by reusing the Segment Anything Model (SAM) and tailoring it for multi-modal data. It introduces SARTM, which fuses LoRA-based adapters on SAM2, language-guided feature fusion via CLIP-derived guidance, and an auxiliary multi-scale segmentation head to enable accurate pixel-level predictions across RGB and thermal inputs. Cross-modal knowledge distillation reduces modality gaps and semantic ambiguity, while a dual-path decoder and FPN-style fusion enhance high-resolution segmentation maps. Across MFNet, PST900, and FMB benchmarks, SARTM achieves state-of-the-art or near-state-of-the-art performance, with ablations confirming the importance of language guidance, multi-scale fusion, and well-tuned loss terms for robust RGB-T understanding in diverse conditions.

Abstract

The recent Segment Anything Model (SAM) demonstrates strong instance segmentation performance across various downstream tasks. However, SAM is trained solely on RGB data, limiting its direct applicability to RGB-thermal (RGB-T) semantic segmentation. Given that RGB-T provides a robust solution for scene understanding in adverse weather and lighting conditions, such as low light and overexposure, we propose a novel framework, SARTM, which customizes the powerful SAM for RGB-T semantic segmentation. Our key idea is to unleash the potential of SAM while introduce semantic understanding modules for RGB-T data pairs. Specifically, our framework first involves fine tuning the original SAM by adding extra LoRA layers, aiming at preserving SAM's strong generalization and segmentation capabilities for downstream tasks. Secondly, we introduce language information as guidance for training our SARTM. To address cross-modal inconsistencies, we introduce a Cross-Modal Knowledge Distillation(CMKD) module that effectively achieves modality adaptation while maintaining its generalization capabilities. This semantic module enables the minimization of modality gaps and alleviates semantic ambiguity, facilitating the combination of any modality under any visual conditions. Furthermore, we enhance the segmentation performance by adjusting the segmentation head of SAM and incorporating an auxiliary semantic segmentation head, which integrates multi-scale features for effective fusion. Extensive experiments are conducted across three multi-modal RGBT semantic segmentation benchmarks: MFNET, PST900, and FMB. Both quantitative and qualitative results consistently demonstrate that the proposed SARTM significantly outperforms state-of-the-art approaches across a variety of conditions.

Figures (8)

• Figure 1: Overall framework of our proposed SARTM, consists of original SAM2 components and proposed language distillation part,including the Semantic Feature Map (SFM), Fine-Grained Feature Pyramid (FFP), and Intermediate-Resolution Feature Pyramid (IFP).
• Figure 2: Illustration of the proposed SARTM framework for multi-modal semantic segmentation. The architecture combines multi-scale features from a frozen image encoder fine-tuned with LoRA layers.
• Figure 3: (a) Hierarchical Feature Fusion for Cross-Scale Information Aggregation using Semantic Feature Map (SFM), Fine-Grained Feature Pyramid (FFP), and Intermediate-Resolution Feature Pyramid (IFP) for high-resolution feature map generation. (b) Hierarchical Refinement Pathway for High-Resolution Embedding leveraging SFM, FFP, and IFP to refine the high-resolution masks.
• Figure 4: Qualitatively compared with the SoTA RGB-T scene resolution network on the PST900 test set, where areas of significant improvement are shown in red boxes.
• Figure 5: Qualitatively compared with the SoTA RGB-T scene resolution network on the MFNet test set, where areas of significant improvement are shown in red boxes.