CLIP4VI-ReID: Learning Modality-shared Representations via CLIP Semantic Bridge for Visible-Infrared Person Re-identification

TL;DR

CLIP4VI-ReID tackles VI-ReID by introducing a three-stage, CLIP-driven framework that first generates RGB-specific text semantics, then uses these semantics to refine infrared features, and finally fine-tunes high-level semantic alignment across RGB, IR, and text. By leveraging aText Semantic Generation stage, Infrared Feature Embedding, and High-level Semantic Alignment, the model builds robust modality-shared representations that bridge the RGB-IR gap with minimal noise in IR text. The approach achieves state-of-the-art or competitive results on SYSU-MM01 and RegDB, with extensive ablations and visualizations confirming the effectiveness of each stage and the importance of learnable prompts. Overall, the method demonstrates that text-guided, staged optimization can markedly improve cross-modal identity discrimination while maintaining inference efficiency through a primarily visual-encoder-based pipeline.

Abstract

This paper proposes a novel CLIP-driven modality-shared representation learning network named CLIP4VI-ReID for VI-ReID task, which consists of Text Semantic Generation (TSG), Infrared Feature Embedding (IFE), and High-level Semantic Alignment (HSA). Specifically, considering the huge gap in the physical characteristics between natural images and infrared images, the TSG is designed to generate text semantics only for visible images, thereby enabling preliminary visible-text modality alignment. Then, the IFE is proposed to rectify the feature embeddings of infrared images using the generated text semantics. This process injects id-related semantics into the shared image encoder, enhancing its adaptability to the infrared modality. Besides, with text serving as a bridge, it enables indirect visible-infrared modality alignment. Finally, the HSA is established to refine the high-level semantic alignment. This process ensures that the fine-tuned text semantics only contain id-related information, thereby achieving more accurate cross-modal alignment and enhancing the discriminability of the learned modal-shared representations. Extensive experimental results demonstrate that the proposed CLIP4VI-ReID achieves superior performance than other state-of-the-art methods on some widely used VI-ReID datasets.

Figures (7)

• Figure 1: The motivation of the paper: (a) Visible (RGB) images and infrared (IR) images share the same high-level semantics. Therefore, the text semantics can serve as a bridge to mitigate the modality gap. (b) Existing single-modality methods struggle to handle VI-ReID tasks and exhibit weak feature representation for infrared images. Besides, due to the huge gap between natural images and infrared images, existing cross-modality methods struggle to capture the fine-grained details of infrared objects, and may introduce semantic noise, thereby affecting their performance. Furthermore, they did not adaptively adjust the obtained text semantics. (c) In contrast, our CLIP4VI-ReID model proposes a coarse-to-fine RGB-IR cross-modal alignment scheme, i.e., it first generates text semantics for visible images, then uses the text semantics as a bridge to correct the embedding of infrared images. Finally, it adaptively adjusts the obtained text semantics and refines the high-level semantic alignment among the three modalities.
• Figure 2: Overview of the proposed CLIP4VI-ReID method. CLIP4VI-ReID is a three-stream model, and its learning process includes three stages: TSG, IFE and HSA. TSG is first utilized to generate text semantics for RGB-modality images, then IFE uses the text semantics as a bridge to correct the embedding of IR-modality images. Finally, HSA adaptively adjusts the obtained text semantics and refines the high-level semantic alignment among the three modalities.
• Figure 3: Illustration of the three-stage learning process of the proposed CLIP4VI-ReID method. Triangles and circles represent RGB-modality features and IR-modality features, respectively, while diamonds represent text-modality features. Different colors represent different pedestrian categories.
• Figure 4: Examples from (a) SYSU-MM01 dataset and (b) RegDB dataset.
• Figure 5: The impact of different parameters $\lambda_1$ and $\lambda_2$ on the (a) SYSU-MM01 dataset and (b) RegDB dataset.