STMI: Segmentation-Guided Token Modulation with Cross-Modal Hypergraph Interaction for Multi-Modal Object Re-Identification

TL;DR

The proposed STMI framework is a novel multi-modal learning framework consisting of three key components that employs learnable query tokens and an adaptive reallocation mechanism to extract compact and informative representations without discarding any tokens.

Abstract

Multi-modal object Re-Identification (ReID) aims to exploit complementary information from different modalities to retrieve specific objects. However, existing methods often rely on hard token filtering or simple fusion strategies, which can lead to the loss of discriminative cues and increased background interference. To address these challenges, we propose STMI, a novel multi-modal learning framework consisting of three key components: (1) Segmentation-Guided Feature Modulation (SFM) module leverages SAM-generated masks to enhance foreground representations and suppress background noise through learnable attention modulation; (2) Semantic Token Reallocation (STR) module employs learnable query tokens and an adaptive reallocation mechanism to extract compact and informative representations without discarding any tokens; (3) Cross-Modal Hypergraph Interaction (CHI) module constructs a unified hypergraph across modalities to capture high-order semantic relationships. Extensive experiments on public benchmarks (i.e., RGBNT201, RGBNT100, and MSVR310) demonstrate the effectiveness and robustness of our proposed STMI framework in multi-modal ReID scenarios.

Figures (5)

• Figure 1: Motivation and intuitive comparison. (a) Existing methods suffer from background noise and information loss due to hard token filtering. (b) Our proposed STMI framework introduces segmentation-guided modulation module to enhance foreground and suppress background, enabling more discriminative feature learning across modalities.
• Figure 2: Comparison with IDEA: (a) IDEA captions often include unknown or inconsistent attributes; (b) ours generates clearer and more accurate descriptions across modalities; (c) our method significantly reduces unknown attributes in both training and test sets.
• Figure 3: An overview of our proposed STMI framework, which consists of three main modules: (1) Segmentation-Guided Feature Modulation enhances foreground and suppresses background using SAM masks; (2) Semantic Token Reallocation extracts compact semantic tokens via cross-attention with learnable queries; (3) Cross-Modal Hypergraph Interaction builds a hypergraph across modalities for high-order semantic interaction.
• Figure 4: Visualization of the feature distributions with t-SNE van2008visualizing. Different colors represent different identities.
• Figure 5: Comparison with different hyper-parameters.